Concurrent modification exception in Configuration.setupContext

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Fixed
  • Affects Version/s: 2.5-M0
  • Fix Version/s: 2.4.3, 2.5-M1
  • Component/s: xml
  • Labels:
    None
  • Environment:
    Jetty

Description

I tried to test the performance of WFS in GeoServer with JMeter. Multiple threads cause errors. Here's the log from Jetty

java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.geotools.xml.Configuration.setupContext(Configuration.java:537)
at org.geotools.xml.impl.ParserHandler.startDocument(ParserHandler.java:196)
at org.apache.xerces.parsers.AbstractSAXParser.startDocument(Unknown Source)
at org.apache.xerces.impl.dtd.XMLDTDValidator.startDocument(Unknown Source)
at org.apache.xerces.impl.XMLDocumentScannerImpl.startEntity(Unknown Source)
at org.apache.xerces.impl.XMLVersionDetector.startDocumentParsing(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.geotools.xml.Parser.parse(Parser.java:179)
at org.geoserver.wfs.xml.v1_1_0.WfsXmlReader.read(WfsXmlReader.java:78)
at org.geoserver.ows.Dispatcher.parseRequestXML(Dispatcher.java:1094)
at org.geoserver.ows.Dispatcher.dispatch(Dispatcher.java:375)
at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:185)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:139)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:44)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:684)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:625)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:392)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:357)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
at org.vfny.geoserver.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:178)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)
at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.LoggingFilter.doFilter(LoggingFilter.java:69)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.geoserver.filters.GZIPFilter.doFilter(GZIPFilter.java:47)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:726)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:842)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:648)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:450)
07 Apr 14:22:01 INFO [geoserver.filters] - 192.168.2.136 "POST /geoserver/wfs" took 8ms

Issue Links

is depended upon by

Bug - A problem which impairs or prevents the functions of the product. GEOS-1856 ConcurrentModificationException under heavy load in xml module configuration

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Andrea Aime added a comment -
Interesting. I see you've assigned the issue to gt2 2.5.x series, so this means you're using GeoServer trunk to make your performance tests?
Show
Andrea Aime added a comment - Interesting. I see you've assigned the issue to gt2 2.5.x series, so this means you're using GeoServer trunk to make your performance tests?
Hide
Permalink
Andrea Aime added a comment -
Volker confirmed on #geoserver that he's using geoserver trunk
Show
Andrea Aime added a comment - Volker confirmed on #geoserver that he's using geoserver trunk
Hide
Permalink
Volker Mische added a comment -
I get the same error with GeoServer 1.6.x trunk (which uses gt2 2.4), too.
Show
Volker Mische added a comment - I get the same error with GeoServer 1.6.x trunk (which uses gt2 2.4), too.
Hide
Permalink
Andrea Aime added a comment -
Just as a clarification, GeoServer 1.6.x is: http://svn.codehaus.org/geoserver/branches/1.6.x
GeoServer trunk is: http://svn.codehaus.org/geoserver/trunk/
There is no such a thing as GeoServer 1.6.x trunk ;), though I guess by that you mean the source currently available in 1.6.x as opposed to the already released versions?
Show
Andrea Aime added a comment - Just as a clarification, GeoServer 1.6.x is: http://svn.codehaus.org/geoserver/branches/1.6.x GeoServer trunk is: http://svn.codehaus.org/geoserver/trunk/ There is no such a thing as GeoServer 1.6.x trunk ;), though I guess by that you mean the source currently available in 1.6.x as opposed to the already released versions?
Hide
Permalink
Volker Mische added a comment -
Thanks for clarification. You're right, i meant the current 1.6.x branch.
Show
Volker Mische added a comment - Thanks for clarification. You're right, i meant the current 1.6.x branch.
Hide
Permalink
Justin Deoliveira added a comment -
Gabriel: i am reassigning to you. To sum up the issue, it is that the configuration object is stored as a singleton object in spring without being thread safe. On trunk the fix is easy, dont make it a singleton, have it be instantated every time. The instantiation cost should be minimal now.

However on 1.6.x this is not so. Everytime the object is created the schema must be created as well... so i am not sure the best way to proceed. Perhaps just hack on Configuration to make sure it is thread safe?
Show
Justin Deoliveira added a comment - Gabriel: i am reassigning to you. To sum up the issue, it is that the configuration object is stored as a singleton object in spring without being thread safe. On trunk the fix is easy, dont make it a singleton, have it be instantated every time. The instantiation cost should be minimal now. However on 1.6.x this is not so. Everytime the object is created the schema must be created as well... so i am not sure the best way to proceed. Perhaps just hack on Configuration to make sure it is thread safe?
Hide
Permalink
Gabriel Roldán added a comment -
okay the issue is clear. Yet I wonder whether the configuration should or should not be a singleton for spring? I'm worried more on doing the right thing, how do we determine if a spring bean shall be or not a singleton?
Show
Gabriel Roldán added a comment - okay the issue is clear. Yet I wonder whether the configuration should or should not be a singleton for spring? I'm worried more on doing the right thing, how do we determine if a spring bean shall be or not a singleton?
Hide
Permalink
Gabriel Roldán added a comment -
Hi Volker, could you attach a JMeter test plan to reproduce the problem? so far I just made one that makes 10 concurrent GetFeature requests 10 times and nothing. Or may be just tell me which kind of requests you're sending so I made a similar test plan over the release configuration. That said, it makes all sense you're getting this sort of error, I just want to consistently reproduce before fixing.
Show
Gabriel Roldán added a comment - Hi Volker, could you attach a JMeter test plan to reproduce the problem? so far I just made one that makes 10 concurrent GetFeature requests 10 times and nothing. Or may be just tell me which kind of requests you're sending so I made a similar test plan over the release configuration. That said, it makes all sense you're getting this sort of error, I just want to consistently reproduce before fixing.
Hide
Permalink
Volker Mische added a comment -
I've just tried it again and I get far less errors than the last times. But it is still the same error. With this JMeter test plan I get about 20-30% errors.
Show
Volker Mische added a comment - I've just tried it again and I get far less errors than the last times. But it is still the same error. With this JMeter test plan I get about 20-30% errors.
Hide
Permalink
Andrea Aime added a comment -
Just tried on my machine and yes, I can get the errors as well against 1.6.3 as released on sf.
In the test output you'll just see failures of the assertion, it seems there is not even a service excetion being thrown (odd?) and the stack trace can be found in the logs.
Show
Andrea Aime added a comment - Just tried on my machine and yes, I can get the errors as well against 1.6.3 as released on sf. In the test output you'll just see failures of the assertion, it seems there is not even a service excetion being thrown (odd?) and the stack trace can be found in the logs.
Hide
Permalink
Gabriel Roldán added a comment -
Cool, so it being against the community schema modules sort of explains why we couldn't reproduce over the core modules. With this information at hand I'm just waiting for the build now to run wfs-c and fix it.
Show
Gabriel Roldán added a comment - Cool, so it being against the community schema modules sort of explains why we couldn't reproduce over the core modules. With this information at hand I'm just waiting for the build now to run wfs-c and fix it.
Hide
Permalink
Andrea Aime added a comment -
Sorry, I should have been more explicit. I did try against 1.6.3 as-is, no community schema stuff.
Show
Andrea Aime added a comment - Sorry, I should have been more explicit. I did try against 1.6.3 as-is, no community schema stuff.
Hide
Permalink
Gabriel Roldán added a comment -
Ok, thanks. I'm going to diagnose the issue based on the differences between 1.6.3 and 1.6.x to see if its actually fixed for good.
Show
Gabriel Roldán added a comment - Ok, thanks. I'm going to diagnose the issue based on the differences between 1.6.3 and 1.6.x to see if its actually fixed for good.
Hide
Permalink
Andrea Aime added a comment -
Gabriel, one more thing. I never managed to reproduce this issue in the past because in the tests I made at WHO we always hit GeoServer with GET requests as opposed to POST ones. It may well be still there in 1.6.x, I just don't know.
Show
Andrea Aime added a comment - Gabriel, one more thing. I never managed to reproduce this issue in the past because in the tests I made at WHO we always hit GeoServer with GET requests as opposed to POST ones. It may well be still there in 1.6.x, I just don't know.
Hide
Permalink
Gabriel Roldán added a comment -
ahhh... that's where xml Configurations are being hit concurrently for different purposes.. parsing and encoding. Perfect.
Show
Gabriel Roldán added a comment - ahhh... that's where xml Configurations are being hit concurrently for different purposes.. parsing and encoding. Perfect.
Hide
Permalink
Gabriel Roldán added a comment -
Yeah, its there in 1.6.x... What would I do without you Andrea...
Show
Gabriel Roldán added a comment - Yeah, its there in 1.6.x... What would I do without you Andrea...
Hide
Permalink
Gabriel Roldán added a comment -
Tracked down the issue to be Configuration.getProperties():List exposing its internal state without synchronization.
Since there're no setter for a Configuration property, but you have to do getProperties().add(...), solution seems to be two-fold:
- use a synchronized list so no concurrent modifications occur while modifying the properties contents
- use a safe copy at client code for iteration so structure modifications does not affect the iterator
Show
Gabriel Roldán added a comment - Tracked down the issue to be Configuration.getProperties():List exposing its internal state without synchronization. Since there're no setter for a Configuration property, but you have to do getProperties().add(...), solution seems to be two-fold: - use a synchronized list so no concurrent modifications occur while modifying the properties contents - use a safe copy at client code for iteration so structure modifications does not affect the iterator
Hide
Permalink
Gabriel Roldán added a comment -
Volker, the issue should be resolved. Please close it once you verify it. I guess you're testing over 1.6.x so you should be able of trying it out directly? otherwise just wait for tomorrows nightly
Show
Gabriel Roldán added a comment - Volker, the issue should be resolved. Please close it once you verify it. I guess you're testing over 1.6.x so you should be able of trying it out directly? otherwise just wait for tomorrows nightly
Hide
Permalink
Volker Mische added a comment -
The issue is fixed. Thanks a lot.
Show
Volker Mische added a comment - The issue is fixed. Thanks a lot.
Hide
Permalink
Volker Mische added a comment -
I forgot to say that I tested it with GeoServer 1.6.x and GeoTools 2.4.x.
Show
Volker Mische added a comment - I forgot to say that I tested it with GeoServer 1.6.x and GeoTools 2.4.x.
Hide
Permalink
Dan Paquette added a comment -
I see this issue is fixed; however, I'm still seeing this issue with 1.7.0 alpha 1 of GeoServer. I'm using WFS POSTs.

It is not clear to me the relationship between the 1.6.x and 1.7.x branches so perhaps this fix is just not yet promoted?
I only upgraded from 1.6.3 to 1.7.0 to get around this issue and I could not find a "1.6.x" newer than 1.6.3
Show
Dan Paquette added a comment - I see this issue is fixed; however, I'm still seeing this issue with 1.7.0 alpha 1 of GeoServer. I'm using WFS POSTs. It is not clear to me the relationship between the 1.6.x and 1.7.x branches so perhaps this fix is just not yet promoted? I only upgraded from 1.6.3 to 1.7.0 to get around this issue and I could not find a "1.6.x" newer than 1.6.3
Hide
Permalink
Andrea Aime added a comment -
The 1.7.0 alpha1 is actually older than 1.6.3. The development in the two branches goes in parallel, and 1.7.x is definitely alpha at the moment. If you cannot wait for 1.6.4 I suggest you grab a nightly from
 http://gridlock.openplans.org/geoserver/1.6.x/

Those are nightly generated build out of the 1.6.x branch, at the time of writing they do represent something in the middle between 1.6.3 and 1.6.4
Show
Andrea Aime added a comment - The 1.7.0 alpha1 is actually older than 1.6.3. The development in the two branches goes in parallel, and 1.7.x is definitely alpha at the moment. If you cannot wait for 1.6.4 I suggest you grab a nightly from  http://gridlock.openplans.org/geoserver/1.6.x/ Those are nightly generated build out of the 1.6.x branch, at the time of writing they do represent something in the middle between 1.6.3 and 1.6.4

People

  • Assignee:
    Gabriel Roldán
    Reporter:
    Volker Mische
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: