added a comment - - edited
This bug is a big problem and repeatably kills other servlets. Everything seems to work at first, until GeoServer receives a WFS request with outputformat=GML2 or version=1.0.0, which triggers the system property change, and causes any other servlet relying on the platform TransformerFactory and not having a xalan jar to fail.
The fault is now at line 181-182 in org.geoserver.wfs.xml.GML2OutputFormat.prepare() on trunk.
System.setProperty("javax.xml.transform.TransformerFactory",
"org.apache.xalan.processor.TransformerFactoryImpl");
You just cannot do this sort of thing in a servlet and expect to coexist with other servlets. It kills deegree running in the same Tomcat stone dead, because deegree was expecting to use the JDK 1.5 xalan (Sun's repackaged copy), and GeoServer just told everyone to use xalan 2.7.0, which it has, but nobody else has on their classpath. Anyone servlet using another TransformerFactory, such as Sun's repackaged copy of Xalan, or Saxon, is stuffed.
The workaround is to copy GeoServer's xalan jar into every other TransformerFactory-using servlet, so they can survive the property change.
There is one sane alternative: use SPI:
http://java.sun.com/j2se/1.4.2/docs/api/javax/xml/transform/TransformerFactory.html#newInstance%28%29
The third option is to use an SPI resource:
META-INF/services/javax.xml.transform.TransformerFactory
This looks like the only servlet-friendly option.
I guess you're right we shouldn't be tied to any specific xml api impl.
Setting the expected transformer factory impl via System.setProperty is there since the early days of geoserver, and since the reason is not documented I don't know if there is a specific need to expect that one, though as said, we shouldn't be doing that. That to say it is pretty much possible that the guilty for that line of code being there can well be myself... Or at least I remember letting it there because it scared me to remove it...
Now, I know there are at least one more place where a specific xml impl is referenced, this time through direct invocation, so it becomes a hard dependency over xerces. Guess it is the xml Encoder in geotools which directly calls xerces' XMLSerializer.
So the question is two fold: