Details
Closed
Major
Description
Under heavy load of both GetFeature and DescribeFeatureType we get failures in GetFeature in roughly 10% of the GetFeature requests with the following stack trace:
2011-01-26 08:50:22,030 ERROR [geoserver.ows] - java.util.ConcurrentModificationException at org.eclipse.emf.common.util.BasicEList$EIterator.checkModCount(BasicEList.java:1327) at org.eclipse.emf.common.util.BasicEList$EIterator.next(BasicEList.java:1275) at org.geotools.xml.Encoder.encode(Encoder.java:726) at org.geotools.xml.Encoder.encode(Encoder.java:563) at org.geoserver.wfs.xml.GML3OutputFormat.encode(GML3OutputFormat.java:253) at org.geoserver.wfs.xml.GML3OutputFormat.write(GML3OutputFormat.java:241) at org.geoserver.wfs.WFSGetFeatureOutputFormat.write(WFSGetFeatureOutputFormat.java:141) at org.geoserver.ows.Dispatcher.response(Dispatcher.java:751) at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:233) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:809) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:501)
This is probably related to the substitution group handling in gt-xsd as the problem occurrs while visiting substitution groups for _Feature.
The patches committed on GeoTools at revision 35934,35936,35937 have been beneficial in this respect, as the same test against GS 2.0.2
results in well over 90% failures instead (which do start with the same stack trace, but eventually lead to NPE as some null finds its way into
the substitution group list), but I guess it's not enough since it's still possible to reproduce failures.
I'm going to attach a JMeter script that makes the problem evident: it creates two groups of 40 threads that do run concurrently, one makes
a lot of DFT (8000, as they are quick to process), the other one some GetFeature (800).
Issue Links
- depends upon
-
GEOT-3398
Encode might throw an concurrent modification exception while visiting abstract elements
-
Activity
Mumble... was too good to be true... I still have the occasional exception on 2.0.x:
java.lang.ArrayIndexOutOfBoundsException
at org.eclipse.emf.common.util.BasicEList.toArray(BasicEList.java:427)
at org.eclipse.emf.ecore.util.EcoreEList.toArray(EcoreEList.java:204)
at java.util.ArrayList.<init>(ArrayList.java:131)
at org.geotools.xml.Encoder.encode(Encoder.java:718)
at org.geotools.xml.Encoder.encode(Encoder.java:561)
This happens as the EMF list is being copied into the array.... is there any way to avoid this? Some sort of atomic copy for EMF lists? 
Hmmm... well ugly problems usually require ugly solutions I guess 
I don't know of any sort of atomic copy for EMF lists. Long story short they are not really designed to be used in the way we are using them. I think it is time to do the EMF upgrade I have been putting off and see if any of these issues are solved.
Another thing that might help the situation is to update SubstitutionGroupLeakPreventer to do an copy of the list, then update it and then setting it back in one transaction.
That said +1 on this patch if it works.
Here is the JMeter script and the csv file to drive it. It works against the standard GeoServer release directory