Maven Doxia

Character encoding autodetection fails for APT source files

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Not A Bug
  • Affects Version/s: 1.0-alpha-11
  • Fix Version/s: None
  • Component/s: Module - Apt
  • Labels:
    None
  • Environment:
    Mac OS X 10.5.6, Java 1.6.0_07
  • Number of attachments :
    1

Description

Doxia unnecessarily forces all APT source files to be encoded in ISO-8859-1. Files encoded in UTF-8 can have garbage characters as a result. Doxia should be able to autodetect the encoding of the APT file to prevent this problem, as it already does for XML (see DOXIA-133).

A test case is attached. It includes two APT source files, one encoded in ISO-8859-1 and another encoded in UTF-8. Both contain the copyright symbol. To reproduce the problem, simply run "mvn site" on the project and open the target/site/test-utf8.html and target/site/test-iso-8859-1.html. The file encoded with ISO-8859-1 should display the copyright symbol correctly, while the one encoded with UTF-8 contains a garbage character immediately before the symbol.

Issue Links

relates to

Bug - A problem which impairs or prevents the functions of the product. DOXIA-133 default XML encoding (UTF-8) or XML encoding set in XML files is ignored: inputEncoding is used instead

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Bug - A problem which impairs or prevents the functions of the product. DOXIA-345 UTF-8 problems with APT and XDOC format

  • Major - Major loss of function.
  • Closed - The issue is considered finished, the resolution is correct. Issues which are not closed can be reopened.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Trevor Harmon added a comment -

Another user complained of this problem a few years ago:

http://www.mailinglistarchive.com/users@maven.apache.org/msg21983.html

Show
Trevor Harmon added a comment - Another user complained of this problem a few years ago: http://www.mailinglistarchive.com/users@maven.apache.org/msg21983.html
Hide
Permalink
Trevor Harmon added a comment -

As a workaround, it is possible to tell Doxia to use UTF-8 explicitly by adding the following plugin declaration:

<plugin>
<artifactId>maven-site-plugin</artifactId>
<configuration>
<inputEncoding>UTF-8</inputEncoding>
</configuration>
</plugin>

However, this has the side effect of breaking all non-UTF-8 files. Doxia should be able to detect the input encoding automatically.

Show
Trevor Harmon added a comment - As a workaround, it is possible to tell Doxia to use UTF-8 explicitly by adding the following plugin declaration: <plugin> <artifactId>maven-site-plugin</artifactId> <configuration> <inputEncoding>UTF-8</inputEncoding> </configuration> </plugin> However, this has the side effect of breaking all non-UTF-8 files. Doxia should be able to detect the input encoding automatically.
Hide
Permalink
Herve Boutemy added a comment -

auto-detecting encoding isn't a bullet-proof feature: nobody can assure to really detect encoding of a byte stream, the better that can be done is a guess, without any guarantee

XML encoding selection is possible because encoding is written into the XML document in a precise manner: here, we have automatic encoding selection. If the stream effective encoding is different from the encoding in <? xml encoding="..."?>, there will be broken characters because the parser is using what is told is the header.

FYI, there has already been a long discussion in Maven dev list about this

APT format does not provide such a convention: it's pure text, without encoding information.
If a convention similar to the XML convention was added.

an APT file starting with ~~ encoding="xxx" should be considered as being written in the specified encoding

we could implement a text reader using it.
Don't know if such a comment at APT file start is copmpatible with title headers though...

Last point: the user complaining about encoding problems you show was hitting a real bug, when encoding wasn't properly handled in Doxia and maven-site-plugin
this is fixed now: see MSITE-314 and POM Element for Source File Encoding

Show
Herve Boutemy added a comment - auto-detecting encoding isn't a bullet-proof feature: nobody can assure to really detect encoding of a byte stream, the better that can be done is a guess, without any guarantee XML encoding selection is possible because encoding is written into the XML document in a precise manner : here, we have automatic encoding selection . If the stream effective encoding is different from the encoding in <? xml encoding="..."?> , there will be broken characters because the parser is using what is told is the header. FYI, there has already been a long discussion in Maven dev list about this APT format does not provide such a convention: it's pure text, without encoding information. If a convention similar to the XML convention was added. an APT file starting with ~~ encoding="xxx" should be considered as being written in the specified encoding we could implement a text reader using it. Don't know if such a comment at APT file start is copmpatible with title headers though... Last point: the user complaining about encoding problems you show was hitting a real bug, when encoding wasn't properly handled in Doxia and maven-site-plugin this is fixed now: see MSITE-314 and POM Element for Source File Encoding

People

  • Assignee:
    Herve Boutemy
    Reporter:
    Trevor Harmon
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: