FEST

ImageAssert says two identical images are not equal

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: FEST-Assert 1.1
  • Component/s: Assert
  • Labels:
    None
  • Environment:
    JDK versions: jdk1.6.0_12 (32 bits on XP, 64 bits on Vista)
  • Testcase included:
    yes
  • Number of attachments :
    2

Description

From Jean-Francois:

Now I am still stuck with my problems comparing screenshots between Windows XP and Vista. This occurs only with Java6 (I have solved the problem with Java5 now).

To make sure the screenshots are comparable between both OS, first of all, I have removed Java6 native font rendering (I have passed "-Dawt.useSystemAAFontSettings=false" to my tests).

Now I get screenshots that really look the same under XP and Vista (see attachment for the simplest case), although the bytestream is different (I am not sure why because both screenshots were taken by FEST).

I have used an image comparison software (http://www.bolidesoft.com/imagecomparer.html free evaluation) to compare these files and it tells me they are 100% identical! So I deduced there may be a problem inside FEST ImageAssert.

Here is the relevant code I use in my tests for checking the screenshots:

BufferedImage screenshot = _screenshot.takeScreenshotOf(_frame.panel("TOP").component());
_screenshot.saveImage(screenshot, snapshot);
         // Compare with previously recorded snapshots
Assertions.assertThat(screenshot).as(expected).isEqualTo(ImageAssert.read(expected));

where _screenshot is a ScreenshotTaker instance, snapshot is the path to which I save the new snapshot, expected is the path where the "reference" snapshot is stored.

the exception I get is:

java.lang.AssertionError: [src/test/resources/screenshots/java6/basics/Basics9RealWorldExample3.png] images do not have the same color(s)
  at org.fest.assertions.Fail.fail(Fail.java:85)
  at org.fest.assertions.GenericAssert.fail(GenericAssert.java:211)
  at org.fest.assertions.ImageAssert.isEqualTo(ImageAssert.java:166)
  at net.java.dev.designgridlayout.AbstractGuiTest.checkSnapshot(AbstractGuiTest.java:167)
  at net.java.dev.designgridlayout.AbstractGuiTest.checkSnapshot(AbstractGuiTest.java:143)
  at net.java.dev.designgridlayout.AbstractGuiTest.checkExample(AbstractGuiTest.java:109)
  at net.java.dev.designgridlayout.basics.BasicFeaturesTest.checkJGoodiesSkeleton(BasicFeaturesTest.java:68)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.testng.internal.MethodHelper.invokeMethod(MethodHelper.java:580)
  at org.testng.internal.Invoker.invokeMethod(Invoker.java:478)
  at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:617)
  at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:885)
  at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
  at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:110)
  at org.testng.TestRunner.runWorkers(TestRunner.java:712)
  at org.testng.TestRunner.privateRun(TestRunner.java:582)
  at org.testng.TestRunner.run(TestRunner.java:477)
  at org.testng.SuiteRunner.runTest(SuiteRunner.java:324)
  at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:319)
  at org.testng.SuiteRunner.privateRun(SuiteRunner.java:292)
  at org.testng.SuiteRunner.run(SuiteRunner.java:198)
  at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:821)
  at org.testng.TestNG.runSuitesLocally(TestNG.java:788)
  at org.testng.TestNG.run(TestNG.java:708)
  at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:62)
  at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:141)
  at org.apache.maven.surefire.Surefire.run(Surefire.java:177)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:345)
  at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1009)

The attachment contains 2 files:

  • the "reference" screenshot that was first taken on Vista 64 bits (in expected-VISTA folder)
  • the screenshot taken during the tests on Windows XP (in actual-XP folder)

Both JDK versions are the same: jdk1.6.0_12 (32 bits on XP, 64 bits on Vista).

Well I think that's all information I can give you about this problem (but if you need more, please feel free to ask!)

If you could take a look at what can be wrong, that would help me a lot!

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Jean-Francois Poilpret added a comment -

Thank you Alex for opening this issue. The exception stack trace I sent you is actually not the one matching the screenshots in the attachment (made a copy/paste mistake from my test log).
The actual stack trace is:

java.lang.AssertionError: [src/test/resources/screenshots/java6/basics/Basics1SimpleGrid.png] images do not have the same color(s)
at org.fest.assertions.Fail.fail(Fail.java:85)
at org.fest.assertions.GenericAssert.fail(GenericAssert.java:211)
at org.fest.assertions.ImageAssert.isEqualTo(ImageAssert.java:166)
at net.java.dev.designgridlayout.AbstractGuiTest.checkSnapshot(AbstractGuiTest.java:167)
at net.java.dev.designgridlayout.AbstractGuiTest.checkSnapshot(AbstractGuiTest.java:143)
at net.java.dev.designgridlayout.AbstractGuiTest.checkExample(AbstractGuiTest.java:109)
at net.java.dev.designgridlayout.basics.BasicFeaturesTest.checkSimpleGrid(BasicFeaturesTest.java:33)
....

It is essentially the same as the original one except that it was run from another test case and the screenshot reference is correct that time.

After a quick look at the source of ImageAssert, I could not find any obvious hint that would explain this behavior.
It might be interesting to run the TC in debug mode and check how the 2 BufferedImages are different (I suspect their ColorModels might differ but there is no certitude).
By the way it might be useful that the ImageAssert.isEqualTo() creates an exception with a more detailed message, in particular which pixel (x,y) was found different, what was the exepected and the actual colors?

I'll see tonight if I can try to check my tests in debug mode (I never run them from eclipse normally, only from maven...)

Show
Jean-Francois Poilpret added a comment - Thank you Alex for opening this issue. The exception stack trace I sent you is actually not the one matching the screenshots in the attachment (made a copy/paste mistake from my test log). The actual stack trace is: java.lang.AssertionError: [src/test/resources/screenshots/java6/basics/Basics1SimpleGrid.png] images do not have the same color(s) at org.fest.assertions.Fail.fail(Fail.java:85) at org.fest.assertions.GenericAssert.fail(GenericAssert.java:211) at org.fest.assertions.ImageAssert.isEqualTo(ImageAssert.java:166) at net.java.dev.designgridlayout.AbstractGuiTest.checkSnapshot(AbstractGuiTest.java:167) at net.java.dev.designgridlayout.AbstractGuiTest.checkSnapshot(AbstractGuiTest.java:143) at net.java.dev.designgridlayout.AbstractGuiTest.checkExample(AbstractGuiTest.java:109) at net.java.dev.designgridlayout.basics.BasicFeaturesTest.checkSimpleGrid(BasicFeaturesTest.java:33) .... It is essentially the same as the original one except that it was run from another test case and the screenshot reference is correct that time. After a quick look at the source of ImageAssert, I could not find any obvious hint that would explain this behavior. It might be interesting to run the TC in debug mode and check how the 2 BufferedImages are different (I suspect their ColorModels might differ but there is no certitude). By the way it might be useful that the ImageAssert.isEqualTo() creates an exception with a more detailed message, in particular which pixel (x,y) was found different, what was the exepected and the actual colors? I'll see tonight if I can try to check my tests in debug mode (I never run them from eclipse normally, only from maven...)
Hide
Permalink
Jean-Francois Poilpret added a comment -

My first comments on trying to debug this issue.
I confirm that both BufferedImage have a different ColorModel. A few details about respective properties for both:

ACTUAL SCREENSHOT (XP, from Robot)

  • image type TYPE_INT_RGB
  • color model DirectColorModel
  • raster IntegerInterleavedRaster

EXPECTED SCREENSHOT (VISTA64, from file)

  • image type TYPE_CUSTOM
  • color model ComponentColorModel
  • raster ByteInterleavedRaster

while debugging through ImageAssert.hasEqualColor() I was expecting a failure on the first pixel but no! There was no error in the first pixels line!
I didn't have the patience to step through each of the 115x173 pixels checks!
Thanks to Eclipse conditional breakpoints, I found out that the first different pixel is at (x,y)=(13,15). The Color values returned by getRGB() were:

actual = -1314056 (0xFFEBF2F8)
expected = -1248520 (0xFFECF2F8)

hence, a 1-bit difference in the Red component (EB Vs. EC)!

Now what I don't know is if this difference was due to internal conversions performed by getRGB() or if the 2 PNG are really different!

I need to find some way to investigate that point further...

Show
Jean-Francois Poilpret added a comment - My first comments on trying to debug this issue. I confirm that both BufferedImage have a different ColorModel. A few details about respective properties for both: ACTUAL SCREENSHOT (XP, from Robot)
  • image type TYPE_INT_RGB
  • color model DirectColorModel
  • raster IntegerInterleavedRaster
EXPECTED SCREENSHOT (VISTA64, from file)
  • image type TYPE_CUSTOM
  • color model ComponentColorModel
  • raster ByteInterleavedRaster
while debugging through ImageAssert.hasEqualColor() I was expecting a failure on the first pixel but no! There was no error in the first pixels line! I didn't have the patience to step through each of the 115x173 pixels checks! Thanks to Eclipse conditional breakpoints, I found out that the first different pixel is at (x,y)=(13,15). The Color values returned by getRGB() were: actual = -1314056 (0xFFEBF2F8) expected = -1248520 (0xFFECF2F8) hence, a 1-bit difference in the Red component (EB Vs. EC)! Now what I don't know is if this difference was due to internal conversions performed by getRGB() or if the 2 PNG are really different! I need to find some way to investigate that point further...
Hide
Permalink
Jean-Francois Poilpret added a comment -

Bad news! After checking both screenshots files with MS Digital Image, and viewing the pixel color they effectively differ!
This means the problem is probably not in ImageAssert, but either in:

  • ScreenshotTaker.takeScreenshotOf()
  • or ScreenshotTaker.saveImage()

Hard to tell, I feel a bit lost now, don't really know where to look (and how) to fix that problem...

Anyway, I think this issue should at least enhance the exception message so that we see immediately which pixel differs and the respective colors for expected and actual pixels.

Show
Jean-Francois Poilpret added a comment - Bad news! After checking both screenshots files with MS Digital Image, and viewing the pixel color they effectively differ! This means the problem is probably not in ImageAssert, but either in:
  • ScreenshotTaker.takeScreenshotOf()
  • or ScreenshotTaker.saveImage()
Hard to tell, I feel a bit lost now, don't really know where to look (and how) to fix that problem... Anyway, I think this issue should at least enhance the exception message so that we see immediately which pixel differs and the respective colors for expected and actual pixels.
Hide
Permalink
Jean-Francois Poilpret added a comment -

In addition, another enhancement could also be to have some threshold in pixel color comparison, passed as an argument to ImageAssert (default = 0).

for each actual pixel
for each of R, G, B value
if abs(expected - actual) > threshold then fail

that looks more like a workaround, but I would already be pleased with it Hopefully it would solve all my problems of screenshot checks between Vista and XP (I wonder if ImageIO or Robot may be different between both platforms).

One next check I will do is to compare, on Vista, the actual screenshot (BufferedImage taken by Robot) with the saved PNG file. This way I would know if the problem is in Robot or ImageIO...

I'll check that later on tonight and post my conclusions.

Show
Jean-Francois Poilpret added a comment - In addition, another enhancement could also be to have some threshold in pixel color comparison, passed as an argument to ImageAssert (default = 0). for each actual pixel for each of R, G, B value if abs(expected - actual) > threshold then fail that looks more like a workaround, but I would already be pleased with it Hopefully it would solve all my problems of screenshot checks between Vista and XP (I wonder if ImageIO or Robot may be different between both platforms). One next check I will do is to compare, on Vista, the actual screenshot (BufferedImage taken by Robot) with the saved PNG file. This way I would know if the problem is in Robot or ImageIO... I'll check that later on tonight and post my conclusions.
Hide
Permalink
Alex Ruiz added a comment -

Thanks Jean-Francois for looking into this bug in so much detail!

Like you said, it is kind of hard to fix. If the bug is in either takeScreenshotOf or in saveImage, the bug will be actually in the JDK, since FEST simply delegates to methods in the JDK that perform those actions. My guess is that there are actual unexpected differences in JDK implementations for different OSs.

Both your suggestions are excellent! (as usual.) I'll be adding more details to the exception message and a threshold for comparison.

Thanks,
-Alex

Show
Alex Ruiz added a comment - Thanks Jean-Francois for looking into this bug in so much detail! Like you said, it is kind of hard to fix. If the bug is in either takeScreenshotOf or in saveImage, the bug will be actually in the JDK, since FEST simply delegates to methods in the JDK that perform those actions. My guess is that there are actual unexpected differences in JDK implementations for different OSs. Both your suggestions are excellent! (as usual.) I'll be adding more details to the exception message and a threshold for comparison. Thanks, -Alex
Hide
Permalink
Alex Ruiz added a comment -

Hi Jean-Francois,

I just added the changes you suggested and it works perfectly for the images you sent me. The code in the test org.fest.assertions.FEST107_EqualImagesAreConsideredDifferent looks like this:

BufferedImage actual = reader.read(file("Basics1SimpleGridActual.png"));
    BufferedImage expected = reader.read(file("Basics1SimpleGridExpected.png"));
    assertThat(actual).isEqualTo(expected, threshold(1));

This is a lot better than what we previously had!

Thanks a lot!
-Alex

Show
Alex Ruiz added a comment - Hi Jean-Francois, I just added the changes you suggested and it works perfectly for the images you sent me. The code in the test org.fest.assertions.FEST107_EqualImagesAreConsideredDifferent looks like this: 
BufferedImage actual = reader.read(file("Basics1SimpleGridActual.png"));
    BufferedImage expected = reader.read(file("Basics1SimpleGridExpected.png"));
    assertThat(actual).isEqualTo(expected, threshold(1));
This is a lot better than what we previously had! Thanks a lot! -Alex
Hide
Permalink
Jean-Francois Poilpret added a comment -

Thanks Alex for this prompt fix.
Is there any way I can get it (the easiest way for me would be a maven snapshot)?

I think you are right about the fact we won't be able to solve the problems of different screenshots between XP and Vista; maybe I won't waste time to try to spot the origin, Robot or ImageIO, since anyway this would not change the fact we cannot do anything!

Cheers

Jean-Francois

Show
Jean-Francois Poilpret added a comment - Thanks Alex for this prompt fix. Is there any way I can get it (the easiest way for me would be a maven snapshot)? I think you are right about the fact we won't be able to solve the problems of different screenshots between XP and Vista; maybe I won't waste time to try to spot the origin, Robot or ImageIO, since anyway this would not change the fact we cannot do anything! Cheers Jean-Francois
Hide
Permalink
Alex Ruiz added a comment -

Hi Jean-Francois,

I'll be uploading snapshots to our new, shiny Maven repo tonight (I'll out for the whole day)...Once it's done, I'll send you instructions on how to access the new repo

Thanks!
-Alex

Show
Alex Ruiz added a comment - Hi Jean-Francois, I'll be uploading snapshots to our new, shiny Maven repo tonight (I'll out for the whole day)...Once it's done, I'll send you instructions on how to access the new repo Thanks! -Alex
Hide
Permalink
Alex Ruiz added a comment -

Hi Jean-Francois,

I just uploaded a snapshot release for FEST-Swing (which will use a snapshot release of FEST-Assert.)

The new Maven repo can be found http://svn.codehaus.org/fest/m2/repo/ (both work fine). The groupId has changed from from 'fest' to 'org.easytesting' (to follow the rules of the central Maven repo and eventually upload FEST there.)

The version that contains fixes for the bugs you filed is fest-swing-1.2a1-SNAPSHOT. Please let me know if this is working OK

Thanks,
-Alex

Show
Alex Ruiz added a comment - Hi Jean-Francois, I just uploaded a snapshot release for FEST-Swing (which will use a snapshot release of FEST-Assert.) The new Maven repo can be found http://svn.codehaus.org/fest/m2/repo/ (both work fine). The groupId has changed from from 'fest' to 'org.easytesting' (to follow the rules of the central Maven repo and eventually upload FEST there.) The version that contains fixes for the bugs you filed is fest-swing-1.2a1-SNAPSHOT. Please let me know if this is working OK Thanks, -Alex
Hide
Permalink
Jean-Francois Poilpret added a comment -

Hello Alex,

thanks for uploading the snapshot.

Unfortunately on the new maven repo, you didn't upload fest-swing-testng jar (all my tests are based on TestNG) and that makes it impossible for me to test the latest fest-swing snapshot (I cannot even try to tune maven artifacts loading to get both jars from the old and new maven repos because the group names have changed so the "old" fest-swing-testng would not find the new fest-swing.

You also mentioned uploading a snapshot fest-swing release elsewhere but I could not find it in any location other than the new maven repo. Is there a new download location for FEST? Is it normal that the docs on codehaus still refer to locations on code.google?

Sorry to bother you a bit more with that but it would be hard for me to revert from TestNG (which I like much) to JUnit 4 (which I still don't appreciate much).

Hope I can test the whole stuff tonight and report my results here.

Cheers,

Jean-Francois

Show
Jean-Francois Poilpret added a comment - Hello Alex, thanks for uploading the snapshot. Unfortunately on the new maven repo, you didn't upload fest-swing-testng jar (all my tests are based on TestNG) and that makes it impossible for me to test the latest fest-swing snapshot (I cannot even try to tune maven artifacts loading to get both jars from the old and new maven repos because the group names have changed so the "old" fest-swing-testng would not find the new fest-swing. You also mentioned uploading a snapshot fest-swing release elsewhere but I could not find it in any location other than the new maven repo. Is there a new download location for FEST? Is it normal that the docs on codehaus still refer to locations on code.google? Sorry to bother you a bit more with that but it would be hard for me to revert from TestNG (which I like much) to JUnit 4 (which I still don't appreciate much). Hope I can test the whole stuff tonight and report my results here. Cheers, Jean-Francois
Hide
Permalink
Jean-Francois Poilpret added a comment -

Hello Alex,

now everything is fine with the snapshot in maven repository. All my tests run well now on both XP and VISTA.
Thanks a lot!

Jean-Francois

Show
Jean-Francois Poilpret added a comment - Hello Alex, now everything is fine with the snapshot in maven repository. All my tests run well now on both XP and VISTA. Thanks a lot! Jean-Francois
Hide
Permalink
Alex Ruiz added a comment -

Thanks a lot Jean-Francois! I'm glad both FEST-106 and FEST-107 are fixed! Thank you so much for the time and effort you spent making sure this is fixed. I truly appreciate it

Best regards,
-Alex

P.S. I'm planning to have a full (non-SNAPSHOT) release by the end of April

Show
Alex Ruiz added a comment - Thanks a lot Jean-Francois! I'm glad both FEST-106 and FEST-107 are fixed! Thank you so much for the time and effort you spent making sure this is fixed. I truly appreciate it Best regards, -Alex P.S. I'm planning to have a full (non-SNAPSHOT) release by the end of April

People

Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Codehaus. Try JIRA - bug tracking software for your team.