GeoServer

Adding shapefile datastore causes java.lang.OutOfMemoryError: Java heap space

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.7.0 , 1.7.1, 1.7.2
  • Fix Version/s: 2.0-RC2
  • Component/s: None
  • Labels:
    None
  • Environment:
    Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
    Kernel 2.6.9-42.ELsmp #1 SMP i686
    Intel Pentium 4 CPU 3.06GHz
    2GB RAM
    Java JDK 1.5.0u12
  • Number of attachments :
    0

Description

In GeoServer 1.7.0 and newer, I get a OutOfMemoryError when I try to add a shapefile datastore to the configuration. This same test worked with 1.6.5 and earlier versions. I go to DataStores and click "new", select "shapefile" and specify an ID, and on the next screen I specify the path to the file. I leave the charset at its default ISO-8859-1, and the spatial index and memory mapped buffer options are left off.

When I click the "submit" button, I wait for 20+ seconds without seeing any response. Then I see this error on the console:

form connection params

{ url=file:data/shapefile_geography/LOW_Lakes.shp, charset=ISO-8859-1}

05 Feb 18:24:42 WARN [geoserver.action] - Unable to fetch a list of FeatureType names from datastore.
java.lang.OutOfMemoryError: Java heap space
at java.lang.String.<init>(String.java:208)
at java.lang.StringBuilder.toString(StringBuilder.java:431)
at org.geotools.data.shapefile.ShpFilesLocker.setTraceException(ShpFilesLocker.java:54)
at org.geotools.data.shapefile.ShpFilesLocker.<init>(ShpFilesLocker.java:33)
at org.geotools.data.shapefile.ShpFiles.unlockRead(ShpFiles.java:441)
at org.geotools.data.shapefile.FileChannelDecorator.implCloseChannel(FileChannelDecorator.java:143)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:97)
at org.geotools.data.shapefile.shp.IndexFile.close(IndexFile.java:182)
at org.geotools.index.quadtree.QuadTree.close(QuadTree.java:357)
at org.geotools.data.shapefile.indexed.ShapeFileIndexer.buildQuadTree(ShapeFileIndexer.java:269)
at org.geotools.data.shapefile.indexed.ShapeFileIndexer.index(ShapeFileIndexer.java:175)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.buildQuadTree(IndexedShapefileDataStore.java:978)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.createSpatialIndex(IndexedShapefileDataStore.java:218) at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.<init>(IndexedShapefileDataStore.java:196)
at org.geotools.data.shapefile.ShapefileDataStoreFactory.createDataStoreInstance(ShapefileDataStoreFactory.java:347)
at org.geotools.data.shapefile.ShapefileDataStoreFactory.createDataStore(ShapefileDataStoreFactory.java:190)
at org.geotools.data.shapefile.ShapefileDataStoreFactory.createDataStore(ShapefileDataStoreFactory.java:59)
at org.vfny.geoserver.util.DataStoreUtils.getDataStore(DataStoreUtils.java:75)
at org.vfny.geoserver.util.DataStoreUtils.acquireDataStore(DataStoreUtils.java:59)
at org.vfny.geoserver.action.data.DataDataStoresEditorAction.execute(DataDataStoresEditorAction.java:135)
at org.vfny.geoserver.action.ConfigAction.execute(ConfigAction.java:101)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:236)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)

I have tried running GeoServer using the default bin/startup.sh script. I've also tried using the following startup options:
-server -Xms48m -Xmx256M
The error occurs in both cases.

Issue Links

is related to

Improvement - An improvement or enhancement to an existing feature or task. GEOT-1498 Make sure the shapefile index generation won't eat excessive amounts of memory

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Improvement - An improvement or enhancement to an existing feature or task. GEOT-2326 ShpfilesLocker wastes cpu and memory by constantly building debug objects

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Paul added a comment -

Interestingly, I am able to add some shapefile datastores. Just not all of them. I'm not sure if this is conclusive, but it seems the larger shapefiles are the ones GeoServer gags on. I successfully loaded a 28MB shapefile in one test I made. But it failed to load a 50MB shapefile.

Show
Paul added a comment - Interestingly, I am able to add some shapefile datastores. Just not all of them. I'm not sure if this is conclusive, but it seems the larger shapefiles are the ones GeoServer gags on. I successfully loaded a 28MB shapefile in one test I made. But it failed to load a 50MB shapefile.
Hide
Permalink
Andrea Aime added a comment -

I can succesfully load much larger files (1.3GB). Can you please add "-XX:+HeapDumpOnOutOfMemoryError" to the startup options, that will create a memory dump on OOM, a file named like java_pid<PID>.hprof (more information here: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/clopts.html)

When you have the file, compress it (zip, bzip, 7zip, gz, whatever suits you) and have it delivered to me. The best way would be to make it available by http/ftp, but if you can't and the file is not too big, we can arrange a transfer by mail.

Show
Andrea Aime added a comment - I can succesfully load much larger files (1.3GB). Can you please add "-XX:+HeapDumpOnOutOfMemoryError" to the startup options, that will create a memory dump on OOM, a file named like java_pid<PID>.hprof (more information here: http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/clopts.html ) When you have the file, compress it (zip, bzip, 7zip, gz, whatever suits you) and have it delivered to me. The best way would be to make it available by http/ftp, but if you can't and the file is not too big, we can arrange a transfer by mail.
Hide
Permalink
Andrea Aime added a comment -

Sorry about the last comment, it was for another jira

Show
Andrea Aime added a comment - Sorry about the last comment, it was for another jira
Hide
Permalink
Andrea Aime added a comment -

Hum, reached that page, clicked on "download", but it just seems to sit there and wait doing nothing...

Show
Andrea Aime added a comment - Hum, reached that page, clicked on "download", but it just seems to sit there and wait doing nothing...
Hide
Permalink
Paul added a comment -

Hmm... that's too bad. Guess MSN is having a bad day. So I searched for another internet file transfer site.
Try this link: http://www.usbupload.com/21364_java_pid25293hprofzip.usb
Please pardon the annoying advertising.

Show
Paul added a comment - Hmm... that's too bad. Guess MSN is having a bad day. So I searched for another internet file transfer site. Try this link: http://www.usbupload.com/21364_java_pid25293hprofzip.usb Please pardon the annoying advertising.
Hide
Permalink
Andrea Aime added a comment -

Ok, got it.
The excessive memory consumption is due to an attempt to generate a spatial index for the shapefile. That operation requires a lot of memory, needs rewriting, but it's not trivial to do so, so don't hold your breath waiting for it

In the meantime you can ask GeoServer not to spatial index your file (it's one of the options of the shapefile datastore) or generate a spatial index with an offline utility such as shptree.
Apart from that, I don't understand why this was not happening in GeoServer 1.6.x, I mean, there is sure some extra debug code in GeoServer that changes the way it fails, but that should not affect the fact that there is not enough memory to build the .qix file...

Show
Andrea Aime added a comment - Ok, got it. The excessive memory consumption is due to an attempt to generate a spatial index for the shapefile. That operation requires a lot of memory, needs rewriting, but it's not trivial to do so, so don't hold your breath waiting for it In the meantime you can ask GeoServer not to spatial index your file (it's one of the options of the shapefile datastore) or generate a spatial index with an offline utility such as shptree. Apart from that, I don't understand why this was not happening in GeoServer 1.6.x, I mean, there is sure some extra debug code in GeoServer that changes the way it fails, but that should not affect the fact that there is not enough memory to build the .qix file...
Hide
Permalink
Andrea Aime added a comment -

Modified indexing so that it takes signifcantly less memory. A GeoServer with enough memory should be able to index shapefiles of pretty much any size (tried and succeded in indexing one that had 6 million features inside using 128MB heap for the indexer process alone, which means a GS with 256MB should be able to do the same)

Show
Andrea Aime added a comment - Modified indexing so that it takes signifcantly less memory. A GeoServer with enough memory should be able to index shapefiles of pretty much any size (tried and succeded in indexing one that had 6 million features inside using 128MB heap for the indexer process alone, which means a GS with 256MB should be able to do the same)

People

  • Assignee:
    Andrea Aime
    Reporter:
    Paul
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: