Maven Wagon
  1. Maven Wagon
  2. WAGON-381

Both Maven 2 and 3 fail to retrieve a <dependency> that is larger than Integer.MAX_VALUE

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.2
    • Fix Version/s: 2.5
    • Component/s: wagon-http
    • Labels:
      None
    • Number of attachments :
      2

      Description

      We have a .tar* file stored in corporate Maven repository, its size is 2.5Gb. Trying to bring it with Maven (both 2 and 3) causes file of 2147483647 size to be downloaded to local maven repo, which is exactly 2<sup>31</sup>-1.

      1. 1.png
        32 kB
      2. 2.png
        19 kB

        Activity

        Hide
        Henri Gomez added a comment -

        Tested with latest Maven 3.1-SNAPSHOT, same problem :

        Apache Maven 3.1-SNAPSHOT (ac64dd6bb6096e8dbd54e3520208e5737fb1c804; 2013-05-29 18:48:30-0700)
        Maven home: /root/apache-maven-3.1-SNAPSHOT
        Java version: 1.6.0_45, vendor: Sun Microsystems Inc.
        Java home: /opt/mycorp/jvm/java-1.6.0-sun-x64/jre
        Default locale: fr_FR, platform encoding: UTF-8
        OS name: "linux", version: "3.4.33-2.24-default", arch: "amd64", family: "unix"
        

        Artifact is 2.5Gb long

        Show
        Henri Gomez added a comment - Tested with latest Maven 3.1-SNAPSHOT, same problem : Apache Maven 3.1-SNAPSHOT (ac64dd6bb6096e8dbd54e3520208e5737fb1c804; 2013-05-29 18:48:30-0700) Maven home: /root/apache-maven-3.1-SNAPSHOT Java version: 1.6.0_45, vendor: Sun Microsystems Inc. Java home: /opt/mycorp/jvm/java-1.6.0-sun-x64/jre Default locale: fr_FR, platform encoding: UTF-8 OS name: "linux" , version: "3.4.33-2.24- default " , arch: "amd64" , family: "unix" Artifact is 2.5Gb long
        Hide
        Olivier Lamy added a comment -

        still weird for me.
        I added a unit test https://git-wip-us.apache.org/repos/asf?p=maven-wagon.git;a=commit;h=05098b3f0f0378a7272f0d9491ef356889bfc67f
        And those huge files (with Content-Length or Chunked) are correctly downloaded.

        Show
        Olivier Lamy added a comment - still weird for me. I added a unit test https://git-wip-us.apache.org/repos/asf?p=maven-wagon.git;a=commit;h=05098b3f0f0378a7272f0d9491ef356889bfc67f And those huge files (with Content-Length or Chunked) are correctly downloaded.
        Hide
        Benson Margulies added a comment -

        We're seeing the same thing here with Nexus as the repo.

        Show
        Benson Margulies added a comment - We're seeing the same thing here with Nexus as the repo.
        Hide
        Tamás Cservenák added a comment - - edited

        I was able to reproduce this with Wagon 2.4 (used in all released Mavens since 3.0.5 and 3.1.x line, even today's master uses Wagon 2.4). Also, I was able to test and verify it's fixed in Wagon 2.5-SNAPSHOT.

        Setup:
        Created 3G file, put it under my local Nexus (built from master). Using curl -IL ... verified that Nexus emits correct headers, especially that Content-Length header is correct:

        [cstamas@Zaphod nexus-perftest]$ curl -IL http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat
        HTTP/1.1 200 OK
        Date: Thu, 12 Sep 2013 09:29:11 GMT
        Set-Cookie: rememberMe=deleteMe; Path=/nexus; Max-Age=0; Expires=Wed, 11-Sep-2013 09:29:11 GMT
        Server: Nexus/2.7.0-SNAPSHOT
        Accept-Ranges: bytes
        ETag: "{SHA1{6e7f6dca8def40df0b21f58e11c1a41c3e000285}}"
        Content-Type: application/octet-stream
        Last-Modified: Thu, 12 Sep 2013 09:15:14 GMT
        Content-Length: 3221225472
        X-Content-Type-Options: nosniff
        
        [cstamas@Zaphod nexus-perftest]$ 
        

        Then, I took "vanilla" Maven 3.0.4 (that should be plagued with this issue, and is the first to use non-lightweight Wagon for HTTP) and built a sample project, noticed following lines in console proving the existence of bug:

        ...
        Downloading: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat
        ...
        [WARNING] Checksum validation failed, no checksums available from the repository for http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat
        Downloaded: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat (2097152 KB at 38750.0 KB/sec)
        ...
        

        Notice the file size Maven 3.0.4 reported, it's not 3G. Verified in local repo, only 2G were downloaded, file was incomplete.

        Then, I locally built Wagon 2.5-SNAPSHOT (from 574561f6d5d1721813c2ec8902aed9e28022854b), then built Maven from master (from 02d12423060361d5444817083c64adf276c3953e) where I bumped Wagon version to 2.5-SNAPSHOT, nuked local repository and rerun the sample project build:

        ...
        Downloading: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat
        [WARNING] Checksum validation failed, no checksums available from the repository for http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat
        Downloaded: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat (3145728 KB at 86418.7 KB/sec)
        ...
        

        Notice, how the reported file size is correct. Also, verified in local storage by comparing size and using openssl sha1 that the file is complete and correctly downloaded (no corruption of any kinds happened), the hashes matched.

        Also, please note that all Maven versions since 3.0.5 release (even 3.1.0 and 3.1.1 and even current master of Maven) all of them use Wagon 2.4, hence testing with latest Maven builds would not help, as all of those use Wagon 2.4, unless you used a branch with upped Wagon version, which in Henry's case is not true, it's a build from master branch
        https://github.com/apache/maven/commit/ac64dd6bb6096e8dbd54e3520208e5737fb1c804

        In case of Ben, it's a Jenkins build, probably another master branch build.

        Hence, I consider the bug fixed, and am closing this issue. Please reopen, if you find some concerns not addressed.

        Show
        Tamás Cservenák added a comment - - edited I was able to reproduce this with Wagon 2.4 (used in all released Mavens since 3.0.5 and 3.1.x line , even today's master uses Wagon 2.4). Also, I was able to test and verify it's fixed in Wagon 2.5-SNAPSHOT. Setup: Created 3G file, put it under my local Nexus (built from master). Using curl -IL ... verified that Nexus emits correct headers, especially that Content-Length header is correct: [cstamas@Zaphod nexus-perftest]$ curl -IL http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat HTTP/1.1 200 OK Date: Thu, 12 Sep 2013 09:29:11 GMT Set-Cookie: rememberMe=deleteMe; Path=/nexus; Max-Age=0; Expires=Wed, 11-Sep-2013 09:29:11 GMT Server: Nexus/2.7.0-SNAPSHOT Accept-Ranges: bytes ETag: "{SHA1{6e7f6dca8def40df0b21f58e11c1a41c3e000285}}" Content-Type: application/octet-stream Last-Modified: Thu, 12 Sep 2013 09:15:14 GMT Content-Length: 3221225472 X-Content-Type-Options: nosniff [cstamas@Zaphod nexus-perftest]$ Then, I took "vanilla" Maven 3.0.4 (that should be plagued with this issue, and is the first to use non-lightweight Wagon for HTTP) and built a sample project, noticed following lines in console proving the existence of bug: ... Downloading: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat ... [WARNING] Checksum validation failed, no checksums available from the repository for http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat Downloaded: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat (2097152 KB at 38750.0 KB/sec) ... Notice the file size Maven 3.0.4 reported, it's not 3G . Verified in local repo, only 2G were downloaded, file was incomplete. Then, I locally built Wagon 2.5-SNAPSHOT (from 574561f6d5d1721813c2ec8902aed9e28022854b), then built Maven from master (from 02d12423060361d5444817083c64adf276c3953e) where I bumped Wagon version to 2.5-SNAPSHOT, nuked local repository and rerun the sample project build: ... Downloading: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat [WARNING] Checksum validation failed, no checksums available from the repository for http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat Downloaded: http://localhost:8081/nexus/content/groups/public/org/test/test/1.0/test-1.0-data.dat (3145728 KB at 86418.7 KB/sec) ... Notice, how the reported file size is correct. Also, verified in local storage by comparing size and using openssl sha1 that the file is complete and correctly downloaded (no corruption of any kinds happened), the hashes matched. Also, please note that all Maven versions since 3.0.5 release (even 3.1.0 and 3.1.1 and even current master of Maven) all of them use Wagon 2.4, hence testing with latest Maven builds would not help, as all of those use Wagon 2.4, unless you used a branch with upped Wagon version, which in Henry's case is not true, it's a build from master branch https://github.com/apache/maven/commit/ac64dd6bb6096e8dbd54e3520208e5737fb1c804 In case of Ben, it's a Jenkins build, probably another master branch build. Hence, I consider the bug fixed, and am closing this issue. Please reopen, if you find some concerns not addressed.
        Hide
        Tamás Cservenák added a comment -

        Closing the issue as the bug reported here is fixed in Wagon 2.5

        Show
        Tamás Cservenák added a comment - Closing the issue as the bug reported here is fixed in Wagon 2.5

          People

          • Assignee:
            Olivier Lamy
            Reporter:
            Evgeny Goldin
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: