Excel and Text exports use Windows Latin-1 encoding

fabrizio giustina added a comment - 15/Nov/04 4:22 PM testcase added: in rc2 encoding is always preserved during export. If you set utf-8 encoding in page with <jsp:directive.page contentType="text/html; charset=UTF8" /> or response.setContentType("text/html; charset=utf-8"); displaytag correctly preserve encoding and csv/excel output is written using utf8. Are you sure you are setting the output encoding correctly? Are you facing this problem with the displaytag filter enabled or not? Which version of displaytag are you using? Please reopen the bug if you are trying with the latest snapshot and you are sure everything has been done correctly (i.e. charset is set to utf8 when not exporting). You can look in displaytag test code for DISPL-107 .jsp and Displ107.java for an example of a working utf8 export.

J. Patterson Waltz III added a comment - 15/Nov/04 9:24 PM In my Tiles master layout, I have the following directive, which is the non-XML equivalent to what you suggested: <%@ page pageEncoding="UTF-8" contentType="text/html; charset=UTF-8" %> When I look at Firefox's Page Info window, it tells me the page is UTF-8 encoded. Same thing with curl: jpw$ curl -i ' http://cerenit.local:8080/SAGE/client.do?retrieve=&d-16544-p=2&status=active&#39; HTTP/1.1 200 OK Set-Cookie: JSESSIONID=42665940856D4110CCB07BD62FD7F523; Path=/SAGE Content-Type: text/html;charset=UTF-8 Transfer-Encoding: chunked Date: Tue, 16 Nov 2004 02:04:45 GMT Server: Apache-Coyote/1.1 But when I export a CSV, it's Windows Latin-1 encoded... :-( All the above is on my OS X box running JBoss 3.2.5. I have the same app running on a Linux box and it sends back pages which have both ISO-8859 *and* UTF-8 listed in the headers: $ curl -I ' http://xxxxx.dyndns.org:8080/SAGE/client.do?retrieve=&d-16544-p=2&status=active&#39; HTTP/1.1 200 OK Set-Cookie: JSESSIONID=27E58087A1A139543FD72477379D826F; Path=/SAGE Content-Type: text/html; charset=ISO-8859-1;charset=UTF-8 Content-Length: 0 Date: Tue, 16 Nov 2004 02:12:39 GMT Server: Apache-Coyote/1.1 Exporting from that machine in CSV also gives Windows Latin-1 encoding, perhaps less surprisingly, in light of the weird double encoding header. Che palle !

fabrizio giustina added a comment - 16/Nov/04 3:54 AM Sorry, but I am still unable to reproduce it, I will need addictional details: - which version of displaytag are you using, is it the latest rc2 snapshot or a previous release? - do you use the export filter? And a couple of addictional tests: - try putting the encoding declaration directly in the page containing the display:table tag (not only in the master tile layout) - can you look at the content-type header during an export? Does it contain the double declaration?

Anonymous added a comment - 16/Nov/04 3:58 PM I am using a slightly modified version of an RC2 pulled from CVS shortly before the 10th of November. The mods implement the Collator-based sorting described in DISPL-77 , in the checkNullsAndCompare method of RowSorter, thus should have no effect on export file formats. If you think it matters, I could retry everything using an unmodified CVS snapshot. Otherwise, I *am* using the export filter. Not sure if it would make any difference, but my web application uses a servlet 2.4/JSP 2.0-style web.xml file. I added an additional <%@ page pageEncoding="UTF-8" contentType="text/html; charset=UTF-8" %> delcaration to the .jsp which contains the actual tile where the displaytag table is. After redeploying the application, I got the same results (web page charset=UTF-8 and CSV export in ISO-8859): $ curl -i ' http://cerenit.local:8080/SAGE/client.do?6578706f7274=1&d-16544-e=1&status=active&retrieve=&#39; HTTP/1.1 200 OK Set-Cookie: JSESSIONID=E08718BE8363353F1BC71B0A2C522229; Path=/SAGE Content-Type: text/csv;charset=ISO-8859-1 Content-Length: 2567 Date: Tue, 16 Nov 2004 20:13:58 GMT Server: Apache-Coyote/1.1 I have not yet tried uploading an updated .war to the server with the "double charset" declarations, as it is a semi-production server, and I'm not getting the results I want even on my development server which doesn't have the double charset problem.

Anonymous added a comment - 14/Dec/04 1:55 AM i have the same problem,and i believe the reason is that the excel just using the system default decoding and i haven't find a way to change that .perhaps the ms excel does not support utf-8.

fabrizio giustina added a comment - 30/Dec/04 12:55 PM sorry, postponing a fix since I'm still unable to reproduce the problem. Unit tests give a positive results and sniffing the content type on a live web application gives me correctly UTF8. :/ I'll try to run some test on a similar environment, but at the moment I can't do anything to fix it. If you need a quick solution for the excel export you could use a customized export view using POI, which generates binary excel files.

Anonymous added a comment - 26/May/05 8:14 AM ?

Ray Tsang added a comment - 26/Oct/05 10:39 PM I am experiencing the same issue. The exported file is indeed UTF8 encoded, but Excel does not open it correctly when opened directly (through the browser or by double clicking). If you start Excel first, then open the file through "Open File" operation, it would detect the encoding correctly. Moreover, if you resave the file from Excel as tab delimited file again, it would open correctly in the future. Turns out, when saving Tab-delimited file from Excel, it does not save w/ the surrounding quotes (").

fabrizio giustina added a comment - 29/Oct/05 10:16 AM since the problem is related to Excel handling of UTF8 encoded files, the fix here is to use the new binary excel format which will be included in displaytag 1.1. Tests for for the csv/xml format also show that encoding is preserved during export and I can't see any other possible fox to displaytag...