GeoTools
  1. GeoTools
  2. GEOT-4090

Features missing when importing shapefile

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 8.0-M4
    • Fix Version/s: 2.7.5
    • Component/s: shapefile plugin
    • Labels:
      None
    • Testcase included:
      yes

      Description

      We've had reports of shapefiles not being completely imported into our system (no error message but more than 50% of the features missing). Notably, QGis handles these files fine. I've attached some test data and created a small test case. It shows that the FeatureCollection returned does contain the correct number of features, except that many of them are null values! This is the output I'm getting:

      Found 2272 features.
      Iterated through 418 features using FeatureIterator.
      Iterated through 2272 features using Array. Found 1854 nulls.
      Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.64 sec <<< FAILURE!

      Results :

      Failed tests:
      testShapeIO(org.geotools.testcase.TestShapeIO): Nulls found!

      We were using the FeatureIterator and didn't even notice that features we're missing until we had it reported to us.

      The test case should be runnable by typing:
      $ mvn test

      Any idea what's going on? Like I said QGis seems to handle the data fine. The test case has some code commented out for exporting whatever geotools reads from the shapefile. Opening this export alongside the shapefile import.shp in QGis clearly illustrates the missing features.

        Activity

        Hide
        Andrea Aime added a comment -
        I don't have time to check right now, but the most likely thing is that the shapefile is not compliant with the specification and qgis is more lenient and manages to read them anyways.
        Another option is that qgis ignores the deletion marks in a shapefile that has been edited with ArcMap
        Yet another option is that we have a bug.
        Show
        Andrea Aime added a comment - I don't have time to check right now, but the most likely thing is that the shapefile is not compliant with the specification and qgis is more lenient and manages to read them anyways. Another option is that qgis ignores the deletion marks in a shapefile that has been edited with ArcMap Yet another option is that we have a bug.
        Hide
        Simon Sandlund added a comment -
        If the deletion mark is the one checked for in DBaseFileReader.read() there doesn't appear to be any in this file. I haven't found a good place to look for compliancy.
        Show
        Simon Sandlund added a comment - If the deletion mark is the one checked for in DBaseFileReader.read() there doesn't appear to be any in this file. I haven't found a good place to look for compliancy.
        Hide
        Andrea Aime added a comment -
        There is only the spec afaik: http://www.google.it/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCsQFjAA&url=http%3A%2F%2Fwww.esri.com%2Flibrary%2Fwhitepapers%2Fpdfs%2Fshapefile.pdf&ei=fbpxT8OmHOSK4gTE5bysDw&usg=AFQjCNHp9v_VCrbagYu23yPhKgqEO9KX8Q

        GeoTools already handles some invalid shapefiles, but there is always space for making it more tolerant to other misbehaviors
        Show
        Andrea Aime added a comment - There is only the spec afaik: http://www.google.it/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CCsQFjAA&url=http%3A%2F%2Fwww.esri.com%2Flibrary%2Fwhitepapers%2Fpdfs%2Fshapefile.pdf&ei=fbpxT8OmHOSK4gTE5bysDw&usg=AFQjCNHp9v_VCrbagYu23yPhKgqEO9KX8Q GeoTools already handles some invalid shapefiles, but there is always space for making it more tolerant to other misbehaviors
        Hide
        Simon Sandlund added a comment -
        I've made some attempts to "clean" the shapefile. I exported it to geojson using qgis, imported it back in and then exported it to shape again, thinking that this would fix any errors in the shapefile. I'm still seeing the same behaviour from GeoTools, though.
        Show
        Simon Sandlund added a comment - I've made some attempts to "clean" the shapefile. I exported it to geojson using qgis, imported it back in and then exported it to shape again, thinking that this would fix any errors in the shapefile. I'm still seeing the same behaviour from GeoTools, though.
        Hide
        Andrea Aime added a comment -
        Found where the issue is, this file contains a very high number of degenerate lines made of a single point. There is code trying to handle this particular broken case, but it is not working as it should.
        I have a fix but I cannot commit until I have a test to prove the fix works.
        In order to build the test I need some broken data, like your shapefile: is the file redistributable?
        I'll only grab one record out of it, don't need the full file, but the licensing conditions must allow me to redistribute it along with the Geotools sources
        Show
        Andrea Aime added a comment - Found where the issue is, this file contains a very high number of degenerate lines made of a single point. There is code trying to handle this particular broken case, but it is not working as it should. I have a fix but I cannot commit until I have a test to prove the fix works. In order to build the test I need some broken data, like your shapefile: is the file redistributable? I'll only grab one record out of it, don't need the full file, but the licensing conditions must allow me to redistribute it along with the Geotools sources
        Hide
        Simon Sandlund added a comment -
        I'm glad you found a fix! Grabbing one record from the file is not a problem.
        Show
        Simon Sandlund added a comment - I'm glad you found a fix! Grabbing one record from the file is not a problem.
        Hide
        Andrea Aime added a comment -
        Patch applied to trunk, could not backport quicly to 2.7.x for the moment, need to revisit, now I have to catch a plane :-p
        Show
        Andrea Aime added a comment - Patch applied to trunk, could not backport quicly to 2.7.x for the moment, need to revisit, now I have to catch a plane :-p
        Hide
        Andrea Aime added a comment -
        Patch applied to 2.7.x as well
        Show
        Andrea Aime added a comment - Patch applied to 2.7.x as well
        Andrea Aime made changes -
        Field Original Value New Value
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 2.7.5 [ 18236 ]

          People

          • Assignee:
            Andrea Aime
            Reporter:
            Simon Sandlund
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: