HTTP headers decoded with platform's default encoding

Hi David,

I've fixed those, thanks. You mention it occurs in other places, do you have a list of them?

thanks
Jan

I did a simple scan for Buffer.toString() and it's called in various places. Since this method is fundamentally broken (uses platform encoding for converting a buffer to a string) then I'd assume any call that relies on it will break sooner or later given non-lower-ascii default codepage. What I would do is replace all calls to Buffer.toString or even make it return something that is not a direct representation of the buffer ("Buffer: ASCII contents=XXX").

Dawid,

I'm not sure Buffer.toString() is fundamentally broken. The toString (charset) method is there to be called when you know the charset of the bytes represented by the buffer. If you don't know what the bytes represent, then guessing the platform encoding is as good as anything.

By "broken" I meant exactly that – the bytes in a Buffer will rarely match the platform's default encoding. The Buffer is (from my brief analysis) constructed from fragments of incoming HTTP request (headers, for example). This implicit assumption that the codepage of headers/ other data and the platform will match is just plain wrong so any use of Buffer's toString() method is, from my perspective, a source of potential problems. From what I remember, toString() is used not only for diagnostics (debugging) but also for other program logic. I'd say toString() should be allowed only when assertions are enabled (so that debugging can be done) but a specialized toString(Charset) should be exposed for program logic. Or whatever else but verifying that the byte buffer matches the request codepage.

Obviously you may disagree with this, but it is really easy to verify in practice – run your jetty instance with -Dfile.encoding=UTF-32. Does it work and pass all the tests? If so, it's fine and doesn't depend on the default codepage. If it doesn't, it's broken to me.

Hi Dawid,

Setting file.encoding does not work because the jvm does not allow it.

In any case, my point still stands: we want to use the encoding when it is known, and use the platform's when it is not known.

I'm going to close this issue for now as I believe we've fixed the instance where we had a known encoding type but were failing to use it. If you have any other specific instances, then I'd appreciate it if you open a new issue over at Jetty's bugzilla at Eclipse, which is Jetty's home: https://bugs.eclipse.org/bugs/buglist.cgi?cmdtype=runnamed&namedcmd=jetty-bugs&list_id=3859265

cheers
Jan

Setting file.encoding does not work because the jvm does not allow it.

You're wrong in thinking so. All JVMs I know of allow the default file encoding to be overridden. More importantly – platform encoding sensitive methods are legacy inheritance that should be avoided because they make your software (and bug reports) reliable on an unknown context. Plain and simple. I don't think there is any reason for software like Jetty to rely on default platform encoding, in particular when handling HTTP requests.

Anyway, up to you of course.

Hi,
it may be interesting to you to read: http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html or http://blog.joda.org/2012/12/annotating-jdk-default-data.html

Jetty is a server-based software and the character encoding used by the client is completely unrelated to the default encoding by the server platform. In the case of HTTP, the HTTP/1.1 standard specifies the encoding to be used for constructing URLs, parsing headers (US-ASCII or UTF-8, depends on).

To Jetty, also the big issue maybe Locale-specific defaults, used e.g. by lowercasing strings. I hope you have tested your server software on a system with tr_TR default locale (at Lucene we choose a random locale and charset for every test run)! If your tests pass, then you seem to correctly handle case-insensitivity, but if they fail in that locale you should really think of passing Locale.ROOT to things like String.toLowerCase(), when they are language-unspecific! I am just thinking of matching HTTP-headers in a case insensitive way against a internal map.