JRuby (please use github issues at http://bugs.jruby.org)
  1. JRuby (please use github issues at http://bugs.jruby.org)
  2. JRUBY-5726

FileUtils#cp (ultimately FileStat.identical? implementations) breaky on Windows

    Details

    • Testcase included:
      yes
    • Number of attachments :
      0

      Description

      FileUtils#cp under Windows, Ruby 1.9 mode, can fail with an erroneous "same file" error. It appears, maybe, that if the contents of the src and dest files are the same, the cp() feels like we're trying to copy the exact same file.

      [INFO] Copying harder
      [WARNING] ArgumentError: same file: target/dirA/thefile and target/dirB/the_other_file
      [WARNING]    fu_each_src_dest at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:1417
      [WARNING]   fu_each_src_dest0 at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:1434
      [WARNING]    fu_each_src_dest at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:1416
      [WARNING]                  cp at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:391
      [WARNING]              (root) at c:/Users/bob/torquebox/fu_cp/fu_cp.rb:12
      [WARNING]                load at org/jruby/RubyKernel.java:1073
      [WARNING]              (root) at -e:1
      

      A simple script, which run under --1.9 on Windows to demonstrate this issue:

      require 'fileutils'
      puts 'Setting up'
      FileUtils.rm_rf( 'target' )
      FileUtils.mkdir_p( 'target/dirA/' )
      FileUtils.mkdir_p( 'target/dirB/' )
      open( 'target/dirA/thefile', 'w' ) {|f|
        f.puts 'Contents of the file'
      }
      puts 'Copying'
      FileUtils.cp( 'target/dirA/thefile', 'target/dirB/the_other_file' )
      puts 'Copying harder'
      FileUtils.cp( 'target/dirA/thefile', 'target/dirB/the_other_file' )
      

      For convenience, a handy maven project and git repo is available:

      https://github.com/torquebox/fu_cp

      Simply clone it, and run `mvn package` to see it work fine on Windows under 1.8 compatibility. Then `mvn package -P1.9` to watch it fail under 1.9 with a stack.

      $ mvn package
      [INFO] Scanning for projects...
      [INFO]
      [INFO] ------------------------------------------------------------------------
      [INFO] Building TorqueBox 1.0.0-SNAPSHOT
      [INFO] ------------------------------------------------------------------------
      [INFO]
      [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ fu_cp ---
      [WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
      [INFO] skip non existing resourceDirectory c:\Users\bob\torquebox\fu_cp\src\main\resources
      [INFO]
      [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ fu_cp ---
      [INFO] No sources to compile
      [INFO]
      [INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ fu_cp ---
      [WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
      [INFO] skip non existing resourceDirectory c:\Users\bob\torquebox\fu_cp\src\test\resources
      [INFO]
      [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ fu_cp ---
      [INFO] No sources to compile
      [INFO]
      [INFO] --- maven-surefire-plugin:2.5:test (default-test) @ fu_cp ---
      [INFO] No tests to run.
      [INFO]
      [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ fu_cp ---
      [WARNING] JAR will be empty - no content was marked for inclusion!
      [INFO] Building jar: c:\Users\bob\torquebox\fu_cp\target\fu_cp-1.0.0-SNAPSHOT.jar
      [INFO]
      [INFO] --- jruby-maven-plugin:0.27.0-SNAPSHOT:jruby (fu, fu#cp) @ fu_cp ---
      [INFO] Setting up
      [INFO] Copying
      [INFO] Copying harder
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 2.859s
      [INFO] Finished at: Tue Apr 19 16:47:12 EDT 2011
      [INFO] Final Memory: 4M/15M
      [INFO] ------------------------------------------------------------------------
      
      bob@WIN-FKV9SEEPI2N ~/torquebox/fu_cp (master)
      $ mvn package -P1.9
      [INFO] Scanning for projects...
      [INFO]
      [INFO] ------------------------------------------------------------------------
      [INFO] Building TorqueBox 1.0.0-SNAPSHOT
      [INFO] ------------------------------------------------------------------------
      [INFO]
      [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ fu_cp ---
      [WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
      [INFO] skip non existing resourceDirectory c:\Users\bob\torquebox\fu_cp\src\main\resources
      [INFO]
      [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ fu_cp ---
      [INFO] No sources to compile
      [INFO]
      [INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ fu_cp ---
      [WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
      [INFO] skip non existing resourceDirectory c:\Users\bob\torquebox\fu_cp\src\test\resources
      [INFO]
      [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ fu_cp ---
      [INFO] No sources to compile
      [INFO]
      [INFO] --- maven-surefire-plugin:2.5:test (default-test) @ fu_cp ---
      [INFO] No tests to run.
      [INFO]
      [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ fu_cp ---
      [WARNING] JAR will be empty - no content was marked for inclusion!
      [INFO] Building jar: c:\Users\bob\torquebox\fu_cp\target\fu_cp-1.0.0-SNAPSHOT.jar
      [INFO]
      [INFO] --- jruby-maven-plugin:0.27.0-SNAPSHOT:jruby (fu, fu#cp) @ fu_cp ---
      [INFO] Setting up
      [INFO] Copying
      [INFO] Copying harder
      [WARNING] ArgumentError: same file: target/dirA/thefile and target/dirB/the_other_file
      [WARNING]    fu_each_src_dest at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:1417
      [WARNING]   fu_each_src_dest0 at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:1434
      [WARNING]    fu_each_src_dest at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:1416
      [WARNING]                  cp at file:/C:/Users/bob/.m2/repository/org/jruby/jruby-complete/1.6.1/jruby-complete-1.6.1.jar!/META-INF/jruby.home/lib/ruby/1.9/fileutils.rb:391
      [WARNING]              (root) at c:/Users/bob/torquebox/fu_cp/fu_cp.rb:12
      [WARNING]                load at org/jruby/RubyKernel.java:1073
      [WARNING]              (root) at -e:1
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 3.766s
      [INFO] Finished at: Tue Apr 19 16:47:21 EDT 2011
      [INFO] Final Memory: 4M/15M
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal de.saumya.mojo:jruby-maven-plugin:0.27.0-SNAPSHOT:jruby (fu, fu#cp) on project fu_cp: Execution fu, fu#cp of goal de.saumya.mojo:jruby-maven-plugin:0.27.0-SNAPSHOT
      1]
      [ERROR]
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR]
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
      
      bob@WIN-FKV9SEEPI2N ~/torquebox/fu_cp (master)
      $
      

        Issue Links

          Activity

          Hide
          bob mcwhirter added a comment -

          fwiw, both cases pass on OSX.

          Also, I tripped across this attempting to do some RubyGems work with 1.9, so it might smack others.

          Show
          bob mcwhirter added a comment - fwiw, both cases pass on OSX. Also, I tripped across this attempting to do some RubyGems work with 1.9, so it might smack others.
          Hide
          Ben Browning added a comment -

          The issue appears to be the implementation of isIdentical() in BaseHeapFileStat - https://github.com/jruby/jnr-posix/blob/master/src/org/jruby/ext/posix/BaseHeapFileStat.java#L118

          According to MSDN documentation at http://msdn.microsoft.com/en-us/library/14h5k7ff%28v=VS.100%29.aspx, st_ino and thus ino() has no meaning on Windows filesystems. This means it can't be used to check if two files are identical. In testing, it appears all files on Windows have the same inode value of 0.

          I'd suggest WindowsFileStat override the isIdentical() method from BaseHeapFileStat and check based on some other criteria, like identical expanded paths.

          Show
          Ben Browning added a comment - The issue appears to be the implementation of isIdentical() in BaseHeapFileStat - https://github.com/jruby/jnr-posix/blob/master/src/org/jruby/ext/posix/BaseHeapFileStat.java#L118 According to MSDN documentation at http://msdn.microsoft.com/en-us/library/14h5k7ff%28v=VS.100%29.aspx , st_ino and thus ino() has no meaning on Windows filesystems. This means it can't be used to check if two files are identical. In testing, it appears all files on Windows have the same inode value of 0. I'd suggest WindowsFileStat override the isIdentical() method from BaseHeapFileStat and check based on some other criteria, like identical expanded paths.
          Hide
          Tobias Crawley added a comment -

          I've submitted a pull request that uses canonical paths instead of inodes on Windows (https://github.com/jruby/jruby/pull/71). I patched this in JRuby instead of jnr-posix, since at the FileStat level, there doesn't appear to be a way to access the path.

          I also have a patch to jnr-posix that throws an UnsupportedOperationException in WindowsFileStat#isIdentical (via POSIXHandler#unimplementedError), but haven't yet issued a pull request, since it opens a can of worms. There are lots of places in the FileStat impls (at least when the platform is windows - I suspect other platforms have the same issue) where operations aren't supported, but none of them throw. From a design standpoint, is that something that should change? Or is just returning null or -1 the appropriate behavior? My vote would be to report to the handler via unimplementedError in these cases, but changing this behavior would impact any code that relies on the current behavior.

          Show
          Tobias Crawley added a comment - I've submitted a pull request that uses canonical paths instead of inodes on Windows ( https://github.com/jruby/jruby/pull/71 ). I patched this in JRuby instead of jnr-posix, since at the FileStat level, there doesn't appear to be a way to access the path. I also have a patch to jnr-posix that throws an UnsupportedOperationException in WindowsFileStat#isIdentical (via POSIXHandler#unimplementedError), but haven't yet issued a pull request, since it opens a can of worms. There are lots of places in the FileStat impls (at least when the platform is windows - I suspect other platforms have the same issue) where operations aren't supported, but none of them throw. From a design standpoint, is that something that should change? Or is just returning null or -1 the appropriate behavior? My vote would be to report to the handler via unimplementedError in these cases, but changing this behavior would impact any code that relies on the current behavior.
          Hide
          Lelon Stoldt added a comment -

          Confirmed broken in Windows 7/jruby 1.6.4 (1.9.2-p136). Very annoying as it break the rails 3.1 asset pipeline.

          Show
          Lelon Stoldt added a comment - Confirmed broken in Windows 7/jruby 1.6.4 (1.9.2-p136). Very annoying as it break the rails 3.1 asset pipeline.
          Show
          Hiro Asari added a comment - I applied Toby's patch to the master and 1.6 branches. https://github.com/jruby/jruby/commit/1458491c133967941ae865877aa818f9126df533 and https://github.com/jruby/jruby/commit/58d9a33a705b374bdc57a2b9743fca99c3a677d7 respectively.
          Hide
          Tobias Crawley added a comment -

          I've confirmed this is fixed for us with 1.6.5. Feel free to close.

          Show
          Tobias Crawley added a comment - I've confirmed this is fixed for us with 1.6.5. Feel free to close.
          Hide
          Hiro Asari added a comment -

          Thanks for the confirmation.

          Show
          Hiro Asari added a comment - Thanks for the confirmation.

            People

            • Assignee:
              Hiro Asari
              Reporter:
              bob mcwhirter
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: