Fail to read GBK attribute in dbf

Andrea Aime added a comment - 21/Jul/09 5:11 AM It does, but you have to specify the target charset while constructing it. There is no way to guess it that I know of.

limonolith added a comment - 21/Jul/09 6:55 AM There is no doubt that we can specified the string charset in terms of params. In addition, we can set the charset by "setStringCharset" method. However, errors occur when retreivalling the attribute. For example Name(20) | Adress(20) 中国石油 | 北京市 feature.getAttribute("Name").toString() = "中国石油 北京市 " feature.getAttribute("Adress").toString() = "" The reason why it occurs is that the procedu ignore the fact that a chinese character take two bytes.Therefore, the "readObject" method is improper for chinese character.

limonolith added a comment - 21/Jul/09 7:07 AM I modified the DbaseFileReader.java and it works now.

Andrea Aime added a comment - 21/Jul/09 7:36 AM What did you modify? We do have a test in our test suite that loads chinese chars, the code looks like: {code} public void testLoadChineseChars() throws Exception {         try {             FeatureCollection<SimpleFeatureType, SimpleFeature> fc = loadFeatures(CHINESE, Charset                     .forName("GB18030"), null);             SimpleFeature first = firstFeature(fc);             String s = (String) first.getAttribute("NAME");             assertEquals("\u9ed1\u9f99\u6c5f\u7701", s);         } catch (UnsupportedCharsetException notInstalledInJRE){                 // this just means you have not installed                 // chinese support into your JRE                 // (as such it represents a bad configuration                 // rather than a test failure)                 // we only wanted to ensure that if you have Chinese support                 // available - GeoTools can use it             }         } {code} Can you provide a test case failing for you so that I can extend the test suite?

limonolith added a comment - 21/Jul/09 7:58 AM Thanks for your testcase. I must confess that the errror may not occur in that case. I've upload two images which indicates the difference between QGIS and uDig. ps: gt-shapefile 2.5 was used in uDig.

limonolith added a comment - 21/Jul/09 8:02 AM "modifeid.jpg" shows the result of modified library which adopts the above mentioned modification.

Andrea Aime added a comment - 21/Jul/09 8:20 AM Images are not helping me much. In order to keep the library working today and tomorrow in face of changes I need to add a test case that will fail when the behaviour is changed. Do you have some royalty free (and very small) shapefile I can use to add a test case for the bug you've encountered?

limonolith added a comment - 21/Jul/09 9:47 AM package com.limonolith.experiment; import java.io.BufferedReader; import java.io.File; import java.io.FileReader; import java.io.IOException; import java.io.Serializable; import java.net.MalformedURLException; import java.util.HashMap; import java.util.Map; import org.geotools.data.DataStoreFinder; import org.geotools.data.FeatureSource; import org.geotools.data.FeatureStore; import org.geotools.data.shapefile.ShapefileDataStore; import org.geotools.data.shapefile.ShapefileDataStoreFactory; import org.geotools.feature.FeatureCollection; import org.geotools.feature.FeatureCollections; import org.geotools.feature.FeatureIterator; import org.geotools.feature.simple.SimpleFeatureBuilder; import org.geotools.feature.simple.SimpleFeatureTypeBuilder; import org.geotools.geometry.jts.JTSFactoryFinder; import org.opengis.feature.simple.SimpleFeature; import org.opengis.feature.simple.SimpleFeatureType; import com.vividsolutions.jts.geom.Coordinate; import com.vividsolutions.jts.geom.GeometryFactory; import com.vividsolutions.jts.geom.Point; import junit.framework.TestCase; public class DbfTest extends TestCase { private String filePath = "C:\\Temp\\test.shp"; private String name = "&#21271;&#20140;&#22823;&#23398;"; private String address = "&#28023;&#28096;"; private void createDataStore() { // Create the datastore ShapefileDataStore dataStore = null; Map<String, Serializable> params = new HashMap<String, Serializable>(); File file = new File(filePath); try { params.put("url", file.toURI().toURL()); params.put("charset", "GBK"); } catch (MalformedURLException e1) { e1.printStackTrace(); } params.put("create spatial index", Boolean.TRUE); ShapefileDataStoreFactory factory = new ShapefileDataStoreFactory(); try { dataStore = (ShapefileDataStore) factory.createNewDataStore(params); } catch (IOException e) { dataStore = null; System.out.println("create fail:" + e.getMessage()); } SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder(); String name = file.getName(); int index = name.indexOf("."); name = name.substring(0, index); builder.setName(name); builder.setSRS("EPSG:4326"); builder.add("Point", Point.class); builder.length(150).add("Name", String.class); builder.length(150).add("Address", String.class); builder.add("Weight", Float.class); SimpleFeatureType featureType = builder.buildFeatureType(); try { dataStore.createSchema(featureType); } catch (IOException e) { dataStore = null; System.out.println("Fail to create schema!" + e.getMessage()); } // add some data to file FeatureStore<SimpleFeatureType, SimpleFeature> featureStore; try { featureStore = (FeatureStore<SimpleFeatureType, SimpleFeature>) dataStore .getFeatureSource(); FeatureCollection<SimpleFeatureType, SimpleFeature> collection = FeatureCollections .newCollection(); SimpleFeatureType type = featureStore.getSchema(); GeometryFactory geoFactory = JTSFactoryFinder .getGeometryFactory(null); Point point = geoFactory.createPoint(new Coordinate(110, 39)); SimpleFeature feature = SimpleFeatureBuilder.build(type, new Object[] { point, this.name, this.address, 1.0 }, null); System.out.println(feature.getAttribute("Name") + "\t" + feature.getAttribute("Address")); collection.add(feature); featureStore.addFeatures(collection); } catch (IOException e1) { e1.printStackTrace(); } } private void validateAttribute(){ File file = new File(this.filePath); try { Map<String, Serializable> connectParameters = new HashMap<String, Serializable>(); connectParameters.put("url", file.toURI().toURL()); connectParameters.put("charset", "GBK"); connectParameters.put("create spatial index", true); ShapefileDataStore dataStore = (ShapefileDataStore) DataStoreFinder.getDataStore(connectParameters); String[] typeNames = dataStore.getTypeNames(); String typeName = typeNames[0]; FeatureSource<SimpleFeatureType, SimpleFeature> featureSource; FeatureCollection<SimpleFeatureType, SimpleFeature> collection; FeatureIterator<SimpleFeature> iterator; featureSource = dataStore.getFeatureSource(typeName); collection = featureSource.getFeatures(); iterator = collection.features(); try { while (iterator.hasNext()) { SimpleFeature feature = iterator.next(); String name = feature.getAttribute("Name").toString(); String address = feature.getAttribute("Address").toString(); System.out.println("Name: " + name); System.out.println("Address: " + address); assertEquals(name, this.name); assertEquals(address, this.address); } } finally { if (iterator != null) { iterator.close(); } } } catch (Exception ex) { ex.printStackTrace(); } } public void testReadGBK() { createDataStore(); validateAttribute(); } }

limonolith added a comment - 21/Jul/09 9:51 AM We can create a shapefile programaticlly and read the attributes again to validate the function. In the DbfTest, I add a point with attributes(Name,Adress,Weight).When I read the attribute from again the name and address changed.

Michael Steigemann added a comment - 07/Aug/09 8:09 AM Two shapefile sets with switched column order. They can be used to find the filtering error, if there are special characters like ß in a column.

Stefan Alfons Tzeggai added a comment - 06/Oct/09 1:41 PM I had some UTF8 problems. It turned out that the DbaseFileReader.java attached by limonolith just fixed all my problems. My attached utf8supportAndTests.tar.gz contains 1. a small and royalty-free shapefile with an UTF8 encoded DBF 2. a patch that ** adds some JUnit-Tests that access the utf8 attributes ** replaces the existing DbaseFileReader with the one given by limonolith ** adds the optimizations from http://jira.codehaus.org/browse/GEOT-2683 to the new DbfFileReader. "mvn test" just passed... so i would be happy to see this comitted next week.

Stefan Alfons Tzeggai added a comment - 08/Oct/09 10:14 AM After Andrea reviewed the patch and made it compatible with Java 1.5, i tested the patch again and did some minimal changes: * removed try catch UnsupportedCharset() around the utf8 test, because utf8 is a standard charset of every JVM * oneBytePerChar = encoder.maxBytesPerChar() == 1.0f removed the second double check: "&& encoder.averageBytesPerChar() == 1.0f; " * added final modifies where possible for performance and removed unused imports Please don't forget to commit the binary resource files contained in the earlier .tar.gz for the JUnit tests.

Andrea Aime added a comment - 08/Oct/09 10:52 AM Applied on trunk. Thanks for all the help everybody.