RGBA raster + direct raster path + JPEG output -> inverted colors

Andrea Aime added a comment - 05/Feb/11 11:05 AM This is the result I get cascading topp:states from another server on linux 64bit with all jai/jai imageio native extensions active (and of course native jpeg encoding on)

Andrea Aime added a comment - 05/Feb/11 12:12 PM Ok, figured it out. There are two considerations to be made. The first is that the direct rendering path does not shave off the alpha band in the resulting image even if the output has not been asked as transparent. Now, this is a bug, if I ask for format=png and transparency=false I must get as output a RGB image, not an RGBA one. So I added code to shave off the alpha band. However, that alone does not solve the issue. This is quite evident as the ImageWorker.writeJPEG was already shaving off the extra band. What is still tripping the JPEG writer (and also the PNG one too, once we remove the alpha band, it also starts to write out inverted colors) is that the image is tiled: in order to get the right colors we still need to ensure the image getting into the encode step is untiled. The following patch does the trick for me: diff --git a/src/wms/src/main/java/org/geoserver/wms/map/RenderedImageMapOutputFormat.java b/src/wms/src/main/java/org/g index 6814481..b9bfcc5 100644 --- a/src/wms/src/main/java/org/geoserver/wms/map/RenderedImageMapOutputFormat.java +++ b/src/wms/src/main/java/org/geoserver/wms/map/RenderedImageMapOutputFormat.java @@ -1054,6 +1049,19 @@ public class RenderedImageMapOutputFormat extends AbstractMapOutputFormat { ( float )mapHeight, null ); } + + // last step, if the output image is supposed not to be transparent and the input has + // alpha we have to remote it (the previous mosaic operation already set the bg color) + ColorModel finalColorModel = image.getColorModel(); + if (!transparent && finalColorModel.hasAlpha()) { + ImageWorker iwAlpha = new ImageWorker(image); + iwAlpha.setRenderingHint(JAI.KEY_IMAGE_LAYOUT, layout); + iwAlpha.retainBands(iwAlpha.getNumBands() - 1); + image = iwAlpha.getRenderedImage(); + } + + + return image; } The trick is setting the image layout as the band is removed, otherwise while removing the band the image worker will also make it tiled (I guess it's using the default tile size from JAI or something). To recap, to avoid color inversion on linux 64 bit with native JPEG writer enabled two things are required, and the above patch does both of them: avoid having the alpha band around make sure the image is untiled Given the nature of the issue I think we should also patch the ImageWorker so that a tiled image getting into the writeJPEG and writePNG methods is flattened if the native encoders are in use.

Andrea Aime added a comment - 06/Feb/11 9:39 AM - edited Forget about this patch, the one attached to GEOS-4360 also solves this issue by different means

Andrea Aime added a comment - 10/Feb/11 11:21 AM Anyways the problem is also in ImageWorker, it forces tiling in the image while removing the alpha band Will open an issue in gt2

Andrea Aime added a comment - 20/Apr/11 3:09 AM Bulk closing all resolved issue that have not gotten any more feedback or comment in the last month