URL query in utf8 character

Hi,

How are you sure that request.setCharacterEncoding() has been called before any of the content is retrieved?

I would recommend you try setting the System property org.eclipse.jetty.util.UrlEncoding.charset to whatever charset you want the url to be interpreted as.

If that doesn't work, then it would be helpful if you could try a newer version of jetty, say jetty-7.1.4 and report back.

thanks
Jan

I believe this was fixed by JETTY-1151.
with the currently release, I can run the test webapp and hit http://localhost:8080/dump/info?q=%E4%B8%AD%E6%96%87 without a problem.

already fixed

surely, you can run the test webapp and hit http://localhost:8080/dump/info?q=%E4%B8%AD%E6%96%87 without a problem.

but you can't run htt p://localhost:8080/dump/info?q=测试 without error!!!
you can copy the url string in your ie6, try it!
because firefox encode the url to ascii automatically but ie6 wont!
Chinese chars in byte is negative, so in Utf8StringBuilder.append(byte b), if the url is not encoded to ascii, the following codes will always been touch:

(line 114 in Utf8StringBuilder.java, 7.0.2.v20100331 , and won't be fixed in the lasted version)
if ((b&0xc0)==0xc0) // <-- here b < 0 , and this condition always true

that's why i said you can't stop client submits non-utf8 requests

I have tried 7.1.4.v20100610, same problem!
Here is my servlet

protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException

Exception:

java.lang.IllegalArgumentException: !utf8
at org.eclipse.jetty.util.Utf8StringBuilder.append(Utf8StringBuilder.java:119)
at org.eclipse.jetty.util.Utf8StringBuilder.append(Utf8StringBuilder.java:49)
at org.eclipse.jetty.http.HttpURI.toUtf8String(HttpURI.java:488)
at org.eclipse.jetty.http.HttpURI.decodeQueryTo(HttpURI.java:647)
at org.eclipse.jetty.server.Request.extractParameters(Request.java:210)
at org.eclipse.jetty.server.Request.getParameter(Request.java:649)
at com.weiqipu.dapu.servlet.JSApi.service(JSApi.java:46)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

and i also have tried encoding "GBK"!

I'm a bit confused... is your browser sending UTF-8 or not?
I just did a telnet to 8080 and typed in a HTTP GET directly and used the UTF-8 characters (not % encoded):

[332] telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /dump/info?q=测试 HTTP/1.1
Host: localhost:8080

the response came back fine with

<th align="right">getQueryString: </th><td>q=测试</td></tr><tr>

and

<th align="left" colspan="2"><big><br/>Request Parameters:</big></th></tr><tr>
<th align="right">q: </th><td>测试</td></tr><tr>

I expect your browser is in some non-utf-8 mode and is not sending utf-8 characters (% encoded or otherwise).

Can you do a tcpdump or wireshark capture so I can see the exact bytes being sent over the wire?

regards

I tried IE-6. It doesn't display the characters at all - just boxes and ?.
But server does not throw an exception.

here is the data sent by ie6

GET http://www.weiqipu.com/search?q=\300\356\312\300\312\257 HTTP/1.1\r\n

0000 47 45 54 20 68 74 74 70 3a 2f 2f 77 77 77 2e 77 GET http://www.w
0010 65 69 71 69 70 75 2e 63 6f 6d 2f 73 65 61 72 63 eiqipu.com/searc
0020 68 3f 71 3d 5c 33 30 30 5c 33 35 36 5c 33 31 32 h?q=\300\356\312
0030 5c 33 30 30 5c 33 31 32 5c 32 35 37 20 48 54 54 \300\312\257 HTT
0040 50 2f 31 2e 31 0d 0a
P/1.1..

This is really strange encoding. Firstly it is in octal with \ encoding, when URLs are meant either to be in UTF-8 or % encoded hex. Then the numbers themselves are not in hex.

Can you try a different IE6 installation.

Ok, I find the reason how this happened:

assume i'm requesting "/search?q=测试", and set encoding to "utf-8".

the invoke stack of jvm should be:

org.eclipse.jetty.util.Utf8StringBuilder.append(Utf8StringBuilder.java:119)
org.eclipse.jetty.util.Utf8StringBuilder.append(Utf8StringBuilder.java:49)
org.eclipse.jetty.http.HttpURI.toUtf8String(HttpURI.java:414)
org.eclipse.jetty.http.HttpURI.decodeQueryTo(HttpURI.java:573)

when i debug in toUtf8String, i found the value of _raw is:
[71, 69, 84, 32, 47, 115, 101, 97, 114, 99, 104, 63, 113, 61, -78, -30, -54, -44, 32, 72, 84, 84, 80, 47 .... more

as i said, bytes of Chinese is negative, so i make a simple test:

String str = "测试";
byte[] bs = str.getBytes("GBK");
for(byte b : bs)

the output is:
-78 -30 -54 -44

in Utf8StringBuilder.append(Utf8StringBuilder.java:119),
((b&0xc0)==0xc0), only -78 return false, so there always throw IllegalArgumentException whatever the Encoding or Character is.

i'm not very experienced on http protocol, not sure why (b&0xc0)==0xc0 throw new IllegalArgumentException......

Hi Wilkins, I found the problem, could you take a look, can this be fixed?

As I said, this issue never been fixed!!!
How disappointed.

donf.yang, we are dealing with this issue as fast as we can, so please DO NOT COMPLAIN about the support that we are giving you for free. If you want faster support you are welcome to take out a support contract.

I still do not see what there is to fix here. As your test shows, the raw bytes being sent by your browser are in GBK encoding, not in UTF-8. So if you set the encoding on the server to UTF-8, the it will have an error because GBK is not legal UTF-8.

You need to try to make your browser send UTF-8. Sometimes forcing the content type of the page served to the browser prior to this request can help... however, many browsers will just erroneously send there own encoding.

I don't understand what you want us to fix? We cannot make our UTF-8 handling accept GBK.