GeoServer

CQL filters causing StreamingRenderer failure

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 1.6.5
  • Component/s: ArcSDE
  • Labels:
    None
  • Patch Submitted:
    Yes
  • Number of attachments :
    1

Description

Consider the following scenario:

Geoserver 1.6.x latest revision: WMS GetMap on a bbox with a CQL filter like (PROP1 = 'value2')

1. In GetMapResponse.java on line 161 the parsed CQL gets turned into a combined filter (cql + layer def filter)
2. In GetMapResponse.java around line 313, the combined filter gets added to the query for that layer, however the propertyNames for that query are left at null (= all properties?)
3. In StreamingRenderer on line 715, the definition query from the layer and the bbox query from the StreamingRenderer are mixed together using DataUtilities.mixQueries(). However DataUtilities.mixQueries interprets query.propertyNames = null as NO properties (as opposed to ALL properties as in point 2.) See line 1401 in DataUtilities.java.
4. This means that the query to the datastore only asks for the explicitly named properties (namely the geometry property from the bbox filter), but that the call to iterator.next() (StreamingRenderer.java line 1599) will probably fail as the datastore is returning a single attribute and the iterator is expecting at least two attributes.

I was able to fix this with the attached patch. It explicitly extracts all required attributes out of the combined filter and applies them to the query created in 2. I tried to run the tests but got other (extraneous?) failures in SVG and KML rendering that I'll look into.

Note that the comments around line 687 make me think that this new queryLayer() implementation might have exactly these sorts of issues. I think the discrepancy between how 4. treats the propertyNames = null and 2. treats propertyNames = null should be addressed at some level. This patch just makes things explicit while not solving that problem.

Issue Links

This issue is related to:
GEOS-1283 error "org.geotools.renderer.lite.StreamingRenderer - Wrong number of attributes expected 1 got 2" Major Closed
This issue depends upon:
GEOT-1848 ArcSDEFeatureSource.getFeatures returns the filter property names when it shouldnt Major Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Andrea Aime added a comment -

Query.getProperties() javadoc says:

The properties array is used to specify the attributes that should be selected for the return feature collection.
*ALL_NAMES: null
If no properties are specified (getProperties returns ALL_NAMES or null) then the full schema should be used (all attributes).

  • NO_NAMES: new String[0]
    If getProperties returns an array of size 0, then the datasource should return features with no attributes, only their ids.

GetMapResponse line 161 is blank, which revision are you looking at? The code creating the query is at line 238 afaik

The evaluation at point 3 is not fully exact, the javadoc of mixQueries says that what you're seeing is to be expected.
In my opinion the renderer is working properly, but the datastore is broken. The renderer does not need any attribute specified in the filter, only the datastore does.
For example, if I attach a cql filter such as "STATE_FIPS = 56" against the topp:states shapefile using the population style, the renderer will end up
issuing a query that requires just the_geom, STATE_ABBR and PERSONS, with a filter that uses STATE_FIPS and the_geom.
The datastore will have to read the "STATE_FIPS" attribute to perform the query, but it won't return it because it has not been requested. Afaik the problem is
the opposite of what you diagnosed, that is, somewhere in the datastore more attributes than requested are returned... but it's hard to say since there
is no stack trace to look.

Show
Andrea Aime added a comment - Query.getProperties() javadoc says:
The properties array is used to specify the attributes that should be selected for the return feature collection. *ALL_NAMES: null If no properties are specified (getProperties returns ALL_NAMES or null) then the full schema should be used (all attributes).
  • NO_NAMES: new String[0] If getProperties returns an array of size 0, then the datasource should return features with no attributes, only their ids.
GetMapResponse line 161 is blank, which revision are you looking at? The code creating the query is at line 238 afaik The evaluation at point 3 is not fully exact, the javadoc of mixQueries says that what you're seeing is to be expected. In my opinion the renderer is working properly, but the datastore is broken. The renderer does not need any attribute specified in the filter, only the datastore does. For example, if I attach a cql filter such as "STATE_FIPS = 56" against the topp:states shapefile using the population style, the renderer will end up issuing a query that requires just the_geom, STATE_ABBR and PERSONS, with a filter that uses STATE_FIPS and the_geom. The datastore will have to read the "STATE_FIPS" attribute to perform the query, but it won't return it because it has not been requested. Afaik the problem is the opposite of what you diagnosed, that is, somewhere in the datastore more attributes than requested are returned... but it's hard to say since there is no stack trace to look.
Hide
Permalink
Andrea Aime added a comment -

Briefly looked at the patch, I cannot accept it, since it's tries to workaround the problem, and in the wrong place too. Imho if a patch is needed, it has to be put inside the renderer.
Moreover, the fact that only ArcSDE is showing this behaviour is a good indication that something is at odds in the datastore, not in the renderer (it is also possible that all other datastores have been fixed to cope with some renderer inconsistency, but it's less likely).

Show
Andrea Aime added a comment - Briefly looked at the patch, I cannot accept it, since it's tries to workaround the problem, and in the wrong place too. Imho if a patch is needed, it has to be put inside the renderer. Moreover, the fact that only ArcSDE is showing this behaviour is a good indication that something is at odds in the datastore, not in the renderer (it is also possible that all other datastores have been fixed to cope with some renderer inconsistency, but it's less likely).
Hide
Permalink
Andrea Aime added a comment -

Actually had an idea to check my hyphotesis. Is ArcSDE able to answer a WFS query where you ask for certain attributes and the filter contains others?

Show
Andrea Aime added a comment - Actually had an idea to check my hyphotesis. Is ArcSDE able to answer a WFS query where you ask for certain attributes and the filter contains others?
Hide
Permalink
Aleda Freeman added a comment -

Andrea,

In answer to your question - yes - ArcSDE is able to answer a WFS query where you ask for certain attributes and the filter contains others.

This query works:

wget "http://env-fp-phoenix:8080/geoserver/wfs?request=getfeature&version=1.0.0&service=wfs&propertyname=TOWN&typename=massgis:GISDATA.TOWNS_POLYM&CQL_FILTER=POP2000>100000"

WFS CQL queries seem fine, WMS no.

Show
Aleda Freeman added a comment - Andrea, In answer to your question - yes - ArcSDE is able to answer a WFS query where you ask for certain attributes and the filter contains others. This query works: wget "http://env-fp-phoenix:8080/geoserver/wfs?request=getfeature&version=1.0.0&service=wfs&propertyname=TOWN&typename=massgis:GISDATA.TOWNS_POLYM&CQL_FILTER=POP2000>100000" WFS CQL queries seem fine, WMS no.
Hide
Permalink
Andrea Aime added a comment -

This is very odd... then again, we have to determine why ArcSDE datastore breaks whilst other do work in the same situation. Adding complexity to the GeoServer code for an ArcSDE oddity is not the solution imho. I would do that myself, but I never played with ArcSDE and I have no way to use one...

Show
Andrea Aime added a comment - This is very odd... then again, we have to determine why ArcSDE datastore breaks whilst other do work in the same situation. Adding complexity to the GeoServer code for an ArcSDE oddity is not the solution imho. I would do that myself, but I never played with ArcSDE and I have no way to use one...
Hide
Permalink
Andrea Aime added a comment -

If my memory serves me right Jody was having the same problem with the ArcSDE datastore on GeoTools trunk + uDig and he patched the datastore to fix it? Jody, what did you do? (if you did anything at all?)

Show
Andrea Aime added a comment - If my memory serves me right Jody was having the same problem with the ArcSDE datastore on GeoTools trunk + uDig and he patched the datastore to fix it? Jody, what did you do? (if you did anything at all?)
Hide
Permalink
Andrea Aime added a comment -

Is GEOS-1283 what you're seeing? If so it looks a lot like an ArcSDE datastore issue, which is grabbing one extra attribute from the data and then using it to build a feature that's supposed to have one less.

Show
Andrea Aime added a comment - Is GEOS-1283 what you're seeing? If so it looks a lot like an ArcSDE datastore issue, which is grabbing one extra attribute from the data and then using it to build a feature that's supposed to have one less.
Hide
Permalink
Gabriel Roldán added a comment -

hmmm these are my findings so far:

Using a CQL_FILTER like this: &CQL_FILTER=ZONE='24TH-MISSION'

  • StreamingRenderer.queryLayer calls FeatureSource.getFeatures(Query).
    Query.propertyNames = ["GEOM"]
    Query.filter = [[Filter.INCLUDE AND [ ZONE = 24TH-MISSION ]] AND [ GEOM bbox POLYGON ((...........)) ]]

So, when it gets down to ArcSDEDataStore, it returns a FeatureSource whose schema contains both the GEOM and the ZONE property, which is wrong. Gonna add an issue for the arcsde module.

Show
Gabriel Roldán added a comment - hmmm these are my findings so far: Using a CQL_FILTER like this: &CQL_FILTER=ZONE='24TH-MISSION'
  • StreamingRenderer.queryLayer calls FeatureSource.getFeatures(Query). Query.propertyNames = ["GEOM"] Query.filter = [[Filter.INCLUDE AND [ ZONE = 24TH-MISSION ]] AND [ GEOM bbox POLYGON ((...........)) ]]
So, when it gets down to ArcSDEDataStore, it returns a FeatureSource whose schema contains both the GEOM and the ZONE property, which is wrong. Gonna add an issue for the arcsde module.
Hide
Permalink
Gabriel Roldán added a comment -

changed issue component to ArcSDE

Show
Gabriel Roldán added a comment - changed issue component to ArcSDE

People

Vote (1)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.