Huge WFS inserts fail 3+ meg POST

> I was wondering if anyone besides me has come across this error, and
> whether it is the result of my naiveté, or an actual problem? How are
> other people handling this?
I'm pretty sure it's an actual problem, as I've never tried a post request that large. I'll try to test it out myself and dig into the problem. If you can share your little client app that would be helpful, you can attach it to this jira task and people can download it. Otherwise I'll do what I can to make a request that large and try to replicate the problem.

Hi,

I suspect a Struts default limit. Maybe :
http://jakarta.apache.org/struts/doc-1.0.2/api/org/apache/struts/upload/MultipartIterator.html#setMaxSize(long)

... could help ?

My 2 cents,

p.b.

No, I don't think it's struts - I was hoping it could be something like that too. But an insert request does not go through struts, it goes directly to the servlet. Only the user interface stuff is struts - this is one of the core operations.

Hi all,

The client app I wrote to test the POST issue is a bit of a hack!

All my code does is load a file from disk and then fire it off to the WFS. I included the important chunk below. I was calling this from an asp.net app. I'll wrap it up into a more useful package if you'd like me to. (not sure how many of you use windows)

I tried a POST with 5522 inserts and the operation failed, though Catalina.out had 8692 entries with the features name (I gave them all the same name and just did a find all)

I had the finest logging on, so it looks like the geoserver waits to load all the features before doing any inserts, since I found no postgres insert lines. (which I had seen with single feature updates)
How else may I help?

----------------------------------------------------------------------------

Public Class tcpConnection
Dim servername As String
Dim servicename As String
Dim xmlrequest As String
Dim customService As String = ""
Public Sub New(ByVal in_servername As String, ByVal in_servicename As String, ByVal in_xmlrequest As String)
servername = in_servername
servicename = in_servicename
xmlrequest = in_xmlrequest
End Sub

Public Function request() As String
Dim serverURL As String = "http://192.168.0.3:8080/geoserver/wfs"
Dim myfile As System.IO.File
Dim myreader As System.IO.StreamReader
myreader = myfile.OpenText("c:\loadAgg7.xml") 'load my 3 meg file. I haven't tried teh 10 meg or 40 meg one yet.
xmlrequest = myreader.ReadToEnd

Try
Dim myReq As Net.HttpWebRequest = CType(Net.WebRequest.Create(serverURL), Net.HttpWebRequest)

myReq.Method = "POST"
myReq.ContentLength = xmlrequest.Length
myReq.Pipelined = True 'let multiple request go through a single socket
myReq.AllowWriteStreamBuffering = True
myReq.KeepAlive = True
'myReq.ReadWriteTimeout = 32000
myReq.SendChunked = True
'myReq.Timeout = 32000
'send our xml request to the server
Try
Dim writer As New System.IO.StreamWriter(myReq.GetRequestStream())
writer.Write(xmlrequest.ToString)
writer.Close()
Catch ee As Exception
End Try
'get the response back
'Dim results As String 'this will hold our response
Try
Dim reader As New System.IO.StreamReader(myReq.GetResponse.GetResponseStream)
Dim myrequest As String = reader.ReadToEnd()
request = myrequest
Dim len As Long = request.Length 'just an easy way to debug and see how large the return is.
reader.Close()
Catch ee As Exception
request = ee.Message 'the usual failure is caught here.

End Try
'If request = "" Then request = "ERROR"
Catch ee As Net.WebException
request = request + "," + ee.Message
Finally
End Try
End Function

I just tried a 700k insert and got the following return. This was on a geoserver wfs running on windows xp. When I attempt huge inserts on the mac I sometimes get no returns at all.

<?xml version="1.0" ?>

• <ServiceExceptionReport version="1.2.0" xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wfs/1.0.0/OGC-exception.xsd">
  <ServiceException>UNCAUGHT EXCEPTION: java.lang.IllegalArgumentException: Cannot parse class java.lang.String</ServiceException>
  </ServiceExceptionReport>

I tried a 100k insert and a 307k insert. The 100k worked fine, but the 307 produded the 'unable to parse java.lang.string' error.

I hope I am not bombarding you guys with too much information.

-john

First off, there's definitely no such thing as 'too much information' in bug reports, all this is definitely helpful. Could you include the stack trace from the logs of the parse exception? So I can see where it's happening? Do it as 'attach file' on jira, so that it won't clutter up the stream of comments, but all can download it. It may be a problem with just one of the features.

No worries about the hack for the requests - I wasn't expecting much more. If it was java I'd use it right away, but unfortunately I don't know .NET at all. So if it's easy to do turning it into an app would be great - I'm on windows so I at least can use it.

As for the loading of features, yeah, geoserver doesn't yet have streaming feature writing, only reading. Refractions is rewriting the GML reader, so when that's available we can integrate it and not use so much memory. Until then you will be memory constrained - but I feel like this current problem is something other than a pure memory constraint.

But yes, post the logs for what geoserver is doing when you get the various errors, on mac and pc - they should have stack traces so I can at least find where the problem is.

Another hint, since Tomcat is used :

http://jakarta.apache.org/tomcat/tomcat-5.0-doc/config/http.html

See Connector@maxPostSize.

But, definitely, a stack trace would help.

Cheers,

p.b.

I just got back from a wedding (not mine) so please forgive my delay.

p.b.--> I altered tomcat's server.xml file to include both
maxPostSize="-1" and
connectionTimeout="-1"
using the little .net app the operation timed out again.

Chris--> I had planned to attach catalina.out to this thread, but it is still growing! At the moment catalina.out is 196 megs and growing. Perhaps there were two problems, the post size limitation p.b. noticed, as well as a response timeout. My current post has been reading from the server for 5+ minutes now. I'll post a follow up once this operation concludes.

-john

No worries on the delay, open source makes you used to waiting for a long time for some responses - I have so many different areas to work on that I just switch gears and focus on other stuff. But I try to make users a first priority when they get information to me.

Is your logging at FINER or FINEST? FINER should be fine, though I imagine it will still produce huge amounts of logs.

The response time out sounds like it could have definitely been the source of no return. For the stack trace just tail the end of it when you get an error - it sounds like some sort of parsing error on a specific feature, and the end of the log may shed light on it.

We should make an issue to work on performace for large inserts. We should be able to stream them into the db, like we do for reading, right now we just hold things in memory, which slows things down for sure, especially with large requests.

The bottom of a 371 meg log file.

Wait, was there an error on this insert? The log doesn't seem to show any - says that it was handled. What returned?

Chris,

Some of this appears to just be a configuration problem. When I increased tomcat's memory to 500-700 megs, geoserver was able to use 550 megs of ram for a 500k insert. So I was able to make larger inserts with more memory allocated to tomcat. The current problem with the inserts is that while it takes ~3 seconds to post the data to the server, geoserver takes approximately a minute and a half to return the response. About 14 of my xml files to post were mal formed, which I think explains the SAX error related to parsing the string. It looks less like a bug, and more like a very hungry process. (550 megs of ram used and 100% cpu for 1.5 minutes is quite a bit! Perhaps this is caused by the validation?

-john

I would guess that it is not caused by validation, though I could be wrong. I imagine that it's much more due to the fact that TransactionFeatureReader has never been optimized at all really. This would be supported by the fact that you get parse errors after long amounts of time. If it was parsing quickly and taking a long time to validate then it would be validation errors that you would get. It's been a bitch just to get it working correctly, and no one has really hit it as hard as you are before.

I imagine you'd get some performance improvements just by removing some of the logging code and putting if (isLoggable)'s around the finer and finest ones, as the greedy string concats will hit your performance a bit. And I think there's one method in there that does a super unoptimized check to figure out which featureType it is. So a few code clean-up type operations might help a lot.

Beyond that what needs to happen is to have the results streamed to the database, not held in memory. I think this may be possible with the current set up, but I'm not sure, I'd have to look at our current architecture again. Once geotools has a new GML parser I'll definitely make sure to do that.

I would dig into this problem for you, since it's definitely an important one for GeoServer to hit, and my job is really just do what I deem the most important improvements, especially the non-flashy ones that no one wants to hit. Unfortunately I'm extremely over committed this month, so I don't have time to do it right. If you have good programming experience I can certainly point you in the right direction of where I think you can optimize and how to look into streaming writing. Or I can recommend a developer that you can contract with to do this for you, as a few of the core geoserver developers work for more consulting type companies. Or you can wait, and bug me in a few weeks, and I may have time then to do it.

But yeah, I think it's not really a bug, it's just slow - and unfortunately optimization is my lowest priority - see http://geoserver.sourceforge.net/documentation/1.2.0-rc2/developer/build_docs.html - the four primary goals of GeoServer. My boss didn't even let me think about optimizations until 1.0.0. So yeah, let me know, I'm really sorry I can't help you out more directly and just fix it for you, but this is now pretty high on my priority list - I just don't have much time this month at all to code.

This should get better as we plug in David's gml reading code.

Good news. I hope the reader supports complex features!

It is my impression that the current code will have troubles parsing a huge insert as well.
Justin? Is it working fully in memory or streaming?

Yup, all the features get parsed into memory. To do streaming on inserts would be hard. For one it would mean that we would have to temporarily save the request out somewhere to intermediate location since an entire request must be parsed before it is dispatched. Then we would have to come up with a feature collection that stream parses the saved out features. All in all probably something that won't be fixed without some funding.

I see. However, for the sake of this bug report, I guess we can accept a transaction that's 3MB, provided the container allows such a big POST (which they won't by default). Something we have to try it, if it works I'd say to go ahead and close it.