GeoServer
  1. GeoServer
  2. GEOS-2611

Adding shapefile datastore causes java.lang.OutOfMemoryError: Java heap space

    Details

    • Type: Bug Bug
    • Status: Closed Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0 , 1.7.1, 1.7.2
    • Fix Version/s: 2.0-RC2
    • Component/s: None
    • Labels:
      None
    • Environment:
      Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
      Kernel 2.6.9-42.ELsmp #1 SMP i686
      Intel Pentium 4 CPU 3.06GHz
      2GB RAM
      Java JDK 1.5.0u12
    • Number of attachments :
      0

      Description

      In GeoServer 1.7.0 and newer, I get a OutOfMemoryError when I try to add a shapefile datastore to the configuration. This same test worked with 1.6.5 and earlier versions. I go to DataStores and click "new", select "shapefile" and specify an ID, and on the next screen I specify the path to the file. I leave the charset at its default ISO-8859-1, and the spatial index and memory mapped buffer options are left off.

      When I click the "submit" button, I wait for 20+ seconds without seeing any response. Then I see this error on the console:

      form connection params

      { url=file:data/shapefile_geography/LOW_Lakes.shp, charset=ISO-8859-1}

      05 Feb 18:24:42 WARN [geoserver.action] - Unable to fetch a list of FeatureType names from datastore.
      java.lang.OutOfMemoryError: Java heap space
      at java.lang.String.<init>(String.java:208)
      at java.lang.StringBuilder.toString(StringBuilder.java:431)
      at org.geotools.data.shapefile.ShpFilesLocker.setTraceException(ShpFilesLocker.java:54)
      at org.geotools.data.shapefile.ShpFilesLocker.<init>(ShpFilesLocker.java:33)
      at org.geotools.data.shapefile.ShpFiles.unlockRead(ShpFiles.java:441)
      at org.geotools.data.shapefile.FileChannelDecorator.implCloseChannel(FileChannelDecorator.java:143)
      at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
      at org.geotools.data.shapefile.shp.IndexFile.close(IndexFile.java:182)
      at org.geotools.index.quadtree.QuadTree.close(QuadTree.java:357)
      at org.geotools.data.shapefile.indexed.ShapeFileIndexer.buildQuadTree(ShapeFileIndexer.java:269)
      at org.geotools.data.shapefile.indexed.ShapeFileIndexer.index(ShapeFileIndexer.java:175)
      at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.buildQuadTree(IndexedShapefileDataStore.java:978)
      at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.createSpatialIndex(IndexedShapefileDataStore.java:218) at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.<init>(IndexedShapefileDataStore.java:196)
      at org.geotools.data.shapefile.ShapefileDataStoreFactory.createDataStoreInstance(ShapefileDataStoreFactory.java:347)
      at org.geotools.data.shapefile.ShapefileDataStoreFactory.createDataStore(ShapefileDataStoreFactory.java:190)
      at org.geotools.data.shapefile.ShapefileDataStoreFactory.createDataStore(ShapefileDataStoreFactory.java:59)
      at org.vfny.geoserver.util.DataStoreUtils.getDataStore(DataStoreUtils.java:75)
      at org.vfny.geoserver.util.DataStoreUtils.acquireDataStore(DataStoreUtils.java:59)
      at org.vfny.geoserver.action.data.DataDataStoresEditorAction.execute(DataDataStoresEditorAction.java:135)
      at org.vfny.geoserver.action.ConfigAction.execute(ConfigAction.java:101)
      at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
      at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
      at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
      at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
      at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
      at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)

      I have tried running GeoServer using the default bin/startup.sh script. I've also tried using the following startup options:
      -server -Xms48m -Xmx256M
      The error occurs in both cases.

        Issue Links

          Activity

          Hide
          Paul added a comment -

          Interestingly, I am able to add some shapefile datastores. Just not all of them. I'm not sure if this is conclusive, but it seems the larger shapefiles are the ones GeoServer gags on. I successfully loaded a 28MB shapefile in one test I made. But it failed to load a 50MB shapefile.

          Show
          Paul added a comment - Interestingly, I am able to add some shapefile datastores. Just not all of them. I'm not sure if this is conclusive, but it seems the larger shapefiles are the ones GeoServer gags on. I successfully loaded a 28MB shapefile in one test I made. But it failed to load a 50MB shapefile.
          Hide
          Andrea Aime added a comment -

          I can succesfully load much larger files (1.3GB). Can you please add "-XX:+HeapDumpOnOutOfMemoryError" to the startup options, that will create a memory dump on OOM, a file named like java_pid<PID>.hprof (more information here: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/clopts.html)

          When you have the file, compress it (zip, bzip, 7zip, gz, whatever suits you) and have it delivered to me. The best way would be to make it available by http/ftp, but if you can't and the file is not too big, we can arrange a transfer by mail.

          Show
          Andrea Aime added a comment - I can succesfully load much larger files (1.3GB). Can you please add "-XX:+HeapDumpOnOutOfMemoryError" to the startup options, that will create a memory dump on OOM, a file named like java_pid<PID>.hprof (more information here: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/clopts.html ) When you have the file, compress it (zip, bzip, 7zip, gz, whatever suits you) and have it delivered to me. The best way would be to make it available by http/ftp, but if you can't and the file is not too big, we can arrange a transfer by mail.
          Hide
          Andrea Aime added a comment -

          Sorry about the last comment, it was for another jira

          Show
          Andrea Aime added a comment - Sorry about the last comment, it was for another jira
          Hide
          Andrea Aime added a comment -

          Hum, reached that page, clicked on "download", but it just seems to sit there and wait doing nothing...

          Show
          Andrea Aime added a comment - Hum, reached that page, clicked on "download", but it just seems to sit there and wait doing nothing...
          Hide
          Paul added a comment -

          Hmm... that's too bad. Guess MSN is having a bad day. So I searched for another internet file transfer site.
          Try this link: http://www.usbupload.com/21364_java_pid25293hprofzip.usb
          Please pardon the annoying advertising.

          Show
          Paul added a comment - Hmm... that's too bad. Guess MSN is having a bad day. So I searched for another internet file transfer site. Try this link: http://www.usbupload.com/21364_java_pid25293hprofzip.usb Please pardon the annoying advertising.
          Hide
          Andrea Aime added a comment -

          Ok, got it.
          The excessive memory consumption is due to an attempt to generate a spatial index for the shapefile. That operation requires a lot of memory, needs rewriting, but it's not trivial to do so, so don't hold your breath waiting for it

          In the meantime you can ask GeoServer not to spatial index your file (it's one of the options of the shapefile datastore) or generate a spatial index with an offline utility such as shptree.
          Apart from that, I don't understand why this was not happening in GeoServer 1.6.x, I mean, there is sure some extra debug code in GeoServer that changes the way it fails, but that should not affect the fact that there is not enough memory to build the .qix file...

          Show
          Andrea Aime added a comment - Ok, got it. The excessive memory consumption is due to an attempt to generate a spatial index for the shapefile. That operation requires a lot of memory, needs rewriting, but it's not trivial to do so, so don't hold your breath waiting for it In the meantime you can ask GeoServer not to spatial index your file (it's one of the options of the shapefile datastore) or generate a spatial index with an offline utility such as shptree. Apart from that, I don't understand why this was not happening in GeoServer 1.6.x, I mean, there is sure some extra debug code in GeoServer that changes the way it fails, but that should not affect the fact that there is not enough memory to build the .qix file...
          Hide
          Andrea Aime added a comment -

          Modified indexing so that it takes signifcantly less memory. A GeoServer with enough memory should be able to index shapefiles of pretty much any size (tried and succeded in indexing one that had 6 million features inside using 128MB heap for the indexer process alone, which means a GS with 256MB should be able to do the same)

          Show
          Andrea Aime added a comment - Modified indexing so that it takes signifcantly less memory. A GeoServer with enough memory should be able to index shapefiles of pretty much any size (tried and succeded in indexing one that had 6 million features inside using 128MB heap for the indexer process alone, which means a GS with 256MB should be able to do the same)

            People

            • Assignee:
              Andrea Aime
              Reporter:
              Paul
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: