Corrupted zip created by assembly: extracting the zip forgets certain folders (or throws permission denied errors) possibly because zip index is corrupted

Should be "cd droolsjbpm-integration-distribution/target" instead of "cd droolsjbpm-integration/target"

Note that the reproduce recipe does not always reproduce it.
It's maybe a race condition in the zip algorithm which the assembly-plugin uses?

During the 5.2.0.CR1 it occurred again, but for another assembly (drools-planner this time). This is a serious problem.
It's strange that both times it occurred on the reference_manual directory. It feels like chmod related troubles.

During 5.2.0 first failed attempt it occurred again, for droolsjbpm-integration again this time.
During 5.2.0.Final succeeded attempt it occurred again, for drools-planner that time.

So hopefully this is a reproduce recipe:
git clone git@github.com:droolsjbpm/droolsjbpm-build-bootstrap.git
droolsjbpm-build-bootstrap/script/git-clone-others.sh
droolsjbpm-build-bootstrap/script/mvn-all.sh clean install -DskipTests -Dfull
cd droolsjbpm-build-distribution/droolsjbpm-uber-distribution/target/droolsjbpm-uber-distribution-*
unzip *.zip
// check the reference_manual directory contents after unzipping (not when zipped)

Similar problems with javadoc directory

I recently encountered a similar problem.
However I could only reproduce it with java7 (windows/linux, oracle/openjdk does not matter).

java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b04)

or

java version "1.7.0_147-icedtea"
OpenJDK Runtime Environment (IcedTea7 2.0) (7~b147-2.0-0ubuntu0.11.10.1)

The zip was fine when using java6.

java version "1.6.0_23"
OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10.2)

The zip also seems to work just fine when unpacked in windows.

So I was able to reproduce it with this:

git clone git://github.com/openengsb/openengsb-framework.git
cd openengsb-framework
git checkout v2.4.2
mvn clean install -DskipTests
cd assembly/target
unzip openengsb-framework-2.4.2.zip
ls -ld openengsb-framework-2.4.2/config

$ ls openengsb-framework-2.4.2/config/ -ld
d--------- 2 christoph christoph 4096 2012-03-30 02:04 openengsb-framework-2.4.2/config/

@Christoph Can you reproduce it consistently? I cannot: one mvn install might have it, while the next doesn't.

I'm having the same problem like Christoph Gr. The only difference is that I'm using a mac and therefore have a different java build.

java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04-414-11M3626)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01-414, mixed mode)

on Darwin hal.lan 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64

I think I hit the same or similar problem yesterday. Same build I've done every day for a year, but yesterday it created a ZIP file containing a corrupt entry (truncated, and maybe corrupted as well). One of the contained files should have been 27MB in size, but was only about 4MB. Another assembly containing the same file was not corrupted during the same build, and the original file brought into the assembly remained intact and 27MB in size.

Version 2.2.1 of the maven-assembly-plugin. Maybe an important fact if there's a race condition or something - my PC has 12 cores, 3.33GHz

Other version info:

Apache Maven 2.2.1 (r801777; 2009-08-06 20:16:01+0100)
Java version: 1.6.0_29
Java home: C:\Program Files\Java\jdk1.6.0_29\jre
Default locale: en_GB, platform encoding: Cp1252
OS name: "windows xp" version: "5.1" arch: "x86" Family: "windows"

(Yes, I know, Windows XP!!)

@Rick That sounds like a different issue. With our zips, all the content is in the zips, but there's something wrong with some of the directory permissions in some builds, which makes them lose file as the are unzip on linux, ok when unzipped on mac and not unzippable on windows.

Hi, sorry for the late answer.

Yes I can reproduce it consistently.

I now figured out that downgrading the assembly-plugin to 2.2.2 seems to fix the problem.

Reproduced with assembly plugin 2.2.2 too (after upgrading from 2.2.1).

@Christoph Your 2.3.0 problem is definitely different then ours. I recommend you open a different jira and since you can reproduce it consistently, you might want to extract a simple project to reproduce it.

Reproduced with assembly plugin 2.3, even with explicitly stating <fileMode>755</fileMode>.
However I did notice something important: It only happens with <dependencySet> with <unpack>true</unpack>.

This code can have it:

    <dependencySet>
      <includes>
        <include>org.drools.planner:drools-planner-core:*:javadoc</include>
      </includes>
      <outputDirectory>javadoc</outputDirectory>
      <unpack>true</unpack>
      <useStrictFiltering>true</useStrictFiltering>
    </dependencySet>
    <dependencySet>
      <includes>
        <include>org.drools.planner:drools-planner-docs:jdocbook</include>
      </includes>
      <outputDirectory>reference_manual</outputDirectory>
      <unpack>true</unpack>
      <unpackOptions>
        <excludes>
          <exclude>META-INF/**</exclude>
        </excludes>
      </unpackOptions>
      <useStrictFiltering>true</useStrictFiltering>
    </dependencySet>

This code does not have it:

    <!-- Javadocs -->
    <fileSet>
      <directory>${project.build.directory}/aggregated-javadocs/apidocs</directory>
      <outputDirectory>javadocs</outputDirectory>
    </fileSet>

@Christoph I 've found a possible workaround, can you check if it works for you too?

    <dependencySet>
      ...
      <!-- Workaround for https://jira.codehaus.org/browse/MASSEMBLY-557 -->
      <fileMode>755</fileMode>
      <directoryMode>755</directoryMode>
      <unpack>true</unpack>
      ...
    </dependencySet>

Indeed that fixed it. Thanks a ton.
I added forced the directoryMode to 755 in the assembly-descriptor-section of the corresponding directory and that was it.

diff --git a/assembly/src/descriptors/bin.xml b/assembly/src/descriptors/bin.xml
index 562c4db..c874142 100644
--- a/assembly/src/descriptors/bin.xml
+++ b/assembly/src/descriptors/bin.xml
@@ -98,6 +98,7 @@
       <outputDirectory>/config/</outputDirectory>
       <lineEnding>dos</lineEnding>
       <fileMode>0644</fileMode>
+      <directoryMode>0755</directoryMode>
     </fileSet>
     <!-- Copy openengsb system repo -->
     <fileSet>

SUMMARY of the issue: the assembly zip is randomly corrupted if <dependenySet> together with <unzip>true</> is used without explicitly setting the <directoryMode>...</>.

I presume the bug is in org.apache.maven.plugin.assembly.archive.task.AddArtifactTask#execute near AddArtifactTask.java line 145:

                if ( directoryMode != -1 )
                {
                    archiver.setDirectoryMode( directoryMode );
                    dirModeSet = true;
                }

I think the directoryMode (or the defaultDirectoryMode) should be set to a default value anyway. Or that the Archiver from plexus-archiver should do that automatically.

I'm not sure that's it.
We do not use either <dependencySet> nor <unzip> in our assembly-descriptor.
The "config" directory in question, is part of the assembly's src-folder (--> it's copied from target/classes).

We basically unpack another assembly (apache-karaf) and among other things add an additional folder named "config".
The unpacking we configured using an execution-configuration in the assembly's pom.xml

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <executions>
          <execution>
            <id>unpack-unix</id>
            <phase>generate-resources</phase>
            <goals>
              <goal>unpack</goal>
            </goals>
            <configuration>
              <artifactItems>
                <artifactItem>
                  <groupId>org.apache.karaf</groupId>
                  <artifactId>apache-karaf</artifactId>
                  <type>tar.gz</type>
                  <outputDirectory>target/dependencies/unix</outputDirectory>
                </artifactItem>
              </artifactItems>
            </configuration>
          </execution>
          ...

The config-directory is added using a <fileSet>

    ...
    <fileSet>
      <directory>target/classes/config</directory>
      <outputDirectory>/config/</outputDirectory>
      <lineEnding>dos</lineEnding>
      <fileMode>0644</fileMode>
    </fileSet>
    ...

You can browse the full source at https://github.com/openengsb/openengsb-framework/tree/v2.4.2/assembly

Currently we downgraded the assembly-plugin to 2.2.2 as a workaround (the <directoryMode> thing is not in the main tree yet).

@Christoph Gritschenberger

We have a similar setup, that can be reduced to a fileSet with a directory but no explicit directoryMode set. Try adding

<directoryMode>0755</directoryMode>

to your fileSet. Worked for us.

That's exactly what I did to make it work.

Could someone test 2.4-SNAPSHOT and see if the problem is still there ?

@Kristian it's random (not always reproducable). However, since no one linked a commit to the assembly plugin that defaults the directoryMode correctly, it's probably still in there.

@Geoffery; the default file modes originate within plexus-archiver. Our linking to commits in plexus is somewhat arbitrary. There is a fair chance this issue is fixed since I vaguely recall fixing something about file attributes in plexus

I just tested again with maven-assembly-plugin-2.4-20121112.092019-526.jar
Behavior is still the same. We still have to stick with the above workaround.

BTW: it's 100% reproducible for me when following the steps I described earlier (https://jira.codehaus.org/browse/MASSEMBLY-557?focusedCommentId=295409&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-295409)

I just upgraded 2.5-SNAPSHOT to the latest plexus-archiver. We've had some pretty significant bug fixes on file modes in archiver, so you may want to test 2.5-SNAPSHOT in a few hours when it hits repository.apache.org