FEST
  1. FEST
  2. FEST-475

Ability to assert with timeout

    Details

    • Type: New Feature New Feature
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: FEST-Assert 1.4
    • Fix Version/s: FEST-Assert 2.0
    • Component/s: Assert
    • Labels:
      None
    • Number of attachments :
      0

      Description

      There is a very handy for testing interactions in multiple thread feature in Mockito [1][2] which allows to verify with timeout. In the case an interaction did not occur already Mockito waits specified amount of time and just then reports failure (in case interaction occurs within that period verification immediately stops blocking and reports success).
      There are cases when it would be very useful to have an ability to specify a timeout with "normal" assertions with fest-assert (e.g. asserting counter for message received from a queue). I don't know API internals, but maybe it would be possible to easily implement something looking like:
      assertThat(foo).isEqualTo(bar).withTimeout(200);
      (possibly at the generic matcher level)

      In my opinion it is a great feature for version 2.0 .

      [1] - http://docs.mockito.googlecode.com/hg/org/mockito/Mockito.html#22
      [2] - http://grepcode.com/file/repo1.maven.org/maven2/org.mockito/mockito-core/1.9.0/org/mockito/internal/verification/VerificationWithTimeoutImpl.java - internal implementation (doesn't look complicated)

        Activity

        Hide
        Marcin Zajaczkowski added a comment -

        matcher level -> assertion level

        Show
        Marcin Zajaczkowski added a comment - matcher level -> assertion level
        Hide
        Joel Costigliola added a comment - - edited

        That could be nice indeed, we must look at this more thoroughly.

        The code example won't work because when isEqualTo is called it performs the assertion verification right away.
        We must let assertion knows that it must wait before performing verification, something like this would work :

        assertThat(foo).withTimeout(200).isEqualTo(bar);
        

        Feel free to fork fest assert 2.x repo and hack (repo is here : https://github.com/alexruiz/fest-assert-2.x)

        Show
        Joel Costigliola added a comment - - edited That could be nice indeed, we must look at this more thoroughly. The code example won't work because when isEqualTo is called it performs the assertion verification right away. We must let assertion knows that it must wait before performing verification, something like this would work : assertThat(foo).withTimeout(200).isEqualTo(bar); Feel free to fork fest assert 2.x repo and hack (repo is here : https://github.com/alexruiz/fest-assert-2.x )
        Hide
        Marcin Zajaczkowski added a comment -

        I took a look at code and generally was able to implement quick and dirty, but working PoC. Nevertheless there are a few issues.

        Thinking about possible implementation I did a sequence diagram, but I was done on paper in a very unreadable form, so firstly I will describe my idea, which would lead to the need to write quite noticeable number of lines of code, so maybe you will have a better idea.

        A PoC implementation for ListAssert.hasSize() assertion (to further generalization with generics for some cases)

        ListAssert.java
          (...)
          public TimeoutListAssert withTimeout(int millis) {
            return new TimeoutListAssert(millis, this);
          }
        
        TimeoutListAssert.java
        public class TimeoutListAssert { //should extends ListAssert?
        
          private static final int THRESHOLD = 100;	//configurable?
        
          private final ListAssert listAssert;
          private final AssertWithTimeoutImpl assertWithTimeout;
        
          /*protected*/ public TimeoutListAssert(int millis, ListAssert listAssert) {	//TODO: TimeoutAssertions with factory methods
            this.listAssert = listAssert;
            assertWithTimeout = new AssertWithTimeoutImpl(THREASHOLD, millis);
          }
        
          public TimeoutListAssert /* ListAssert? */ hasSize(final int expected) {
            AssertDelegate hasSizeAssertDelegate = new AssertDelegate() {
              public void doAssertion() {
                listAssert.hasSize(expected);
              }
            };
            assertWithTimeout.doAssertion(hasSizeAssertDelegate);
            return this;  //listAssert?
          }
        
          //other methods
        }
        

        AssertDelegate - simple interface with just one method, AssertWithTimeoutImpl - similar to mentioned Mockito implementation

        Problems with my solution:

        • I don't see a clear way to transparently wrap methods in the *Assert classes to support timeout. In the end application there could be AOP used, but in a library like fest-assert it wouldn't be a good idea . As the result - additional code
        • Java doesn't allow to pass functions as a method parameter, so it is needed in my solution to create an anonymous delegate class for every method which should be assertable with timeout
        • I started to write it for StringAssert and I faced a problem of immutability. It wouldn't be probably possible to add an assertion with timeout for immutable types (including primitive numbers passed as a value) what limits a little bit a potential of proposed feature
        • probably some others

        I hope you have some better idea how to implement it.

        Show
        Marcin Zajaczkowski added a comment - I took a look at code and generally was able to implement quick and dirty, but working PoC. Nevertheless there are a few issues. Thinking about possible implementation I did a sequence diagram, but I was done on paper in a very unreadable form, so firstly I will describe my idea, which would lead to the need to write quite noticeable number of lines of code, so maybe you will have a better idea. A PoC implementation for ListAssert.hasSize() assertion (to further generalization with generics for some cases) ListAssert.java (...) public TimeoutListAssert withTimeout( int millis) { return new TimeoutListAssert(millis, this ); } TimeoutListAssert.java public class TimeoutListAssert { //should extends ListAssert? private static final int THRESHOLD = 100; //configurable? private final ListAssert listAssert; private final AssertWithTimeoutImpl assertWithTimeout; /* protected */ public TimeoutListAssert( int millis, ListAssert listAssert) { //TODO: TimeoutAssertions with factory methods this .listAssert = listAssert; assertWithTimeout = new AssertWithTimeoutImpl(THREASHOLD, millis); } public TimeoutListAssert /* ListAssert? */ hasSize( final int expected) { AssertDelegate hasSizeAssertDelegate = new AssertDelegate() { public void doAssertion() { listAssert.hasSize(expected); } }; assertWithTimeout.doAssertion(hasSizeAssertDelegate); return this ; //listAssert? } //other methods } AssertDelegate - simple interface with just one method, AssertWithTimeoutImpl - similar to mentioned Mockito implementation Problems with my solution: I don't see a clear way to transparently wrap methods in the *Assert classes to support timeout. In the end application there could be AOP used, but in a library like fest-assert it wouldn't be a good idea . As the result - additional code Java doesn't allow to pass functions as a method parameter, so it is needed in my solution to create an anonymous delegate class for every method which should be assertable with timeout I started to write it for StringAssert and I faced a problem of immutability. It wouldn't be probably possible to add an assertion with timeout for immutable types (including primitive numbers passed as a value) what limits a little bit a potential of proposed feature probably some others I hope you have some better idea how to implement it.
        Hide
        Joel Costigliola added a comment -

        I haven't thought thoroughly yet to the problem but for the time I don't see any elegant solution ... I will try to find some time to think about it and I'll let you know.
        Keep hacking !

        thanks

        Show
        Joel Costigliola added a comment - I haven't thought thoroughly yet to the problem but for the time I don't see any elegant solution ... I will try to find some time to think about it and I'll let you know. Keep hacking ! thanks
        Hide
        Joel Costigliola added a comment - - edited

        I was thinking that it may be possible with a dynamic proxy (as long as the assertion is defined in an interface).

        withTimeout(int) method creates and returns a proxy of the Assertion interface (let's say Assert<S, A>), the chained assertion will be executed by the proxy.
        Instead of executing the assertion right away, the proxy creates a Callable performing the assertion task and executes it through an ExecutorService, it uses the returned Future to get the result within the given timeout, a TimeoutException being thrown if the timeout is not honored.

        ExecutorService executor = Executors.newCachedThreadPool();
        Callable<Object> task = new Callable<Object>() {
           public Object call() {
              // proceed the assertion
              return assert;
           }
        };
        Future<Object> future = executor.submit(task);
        try {
           Object result = future.get(5, TimeUnit.SECONDS); 
        } catch (TimeoutException ex) {
           // throw an assertion error with an explicit message 
        } catch (InterruptedException e) {
           // rethrow it in a runtime exception
        } catch (ExecutionException e) {
           // rethrow it in a runtime exception
        } finally {
           future.cancel(); // stop the assertion after deadline
        }
        

        what do you think ? (I haven't coded it so it may not work)

        Show
        Joel Costigliola added a comment - - edited I was thinking that it may be possible with a dynamic proxy (as long as the assertion is defined in an interface). withTimeout(int) method creates and returns a proxy of the Assertion interface (let's say Assert<S, A>), the chained assertion will be executed by the proxy. Instead of executing the assertion right away, the proxy creates a Callable performing the assertion task and executes it through an ExecutorService, it uses the returned Future to get the result within the given timeout, a TimeoutException being thrown if the timeout is not honored. ExecutorService executor = Executors.newCachedThreadPool(); Callable< Object > task = new Callable< Object >() { public Object call() { // proceed the assertion return assert ; } }; Future< Object > future = executor.submit(task); try { Object result = future .get(5, TimeUnit.SECONDS); } catch (TimeoutException ex) { // throw an assertion error with an explicit message } catch (InterruptedException e) { // rethrow it in a runtime exception } catch (ExecutionException e) { // rethrow it in a runtime exception } finally { future .cancel(); // stop the assertion after deadline } what do you think ? (I haven't coded it so it may not work)
        Hide
        Marcin Zajaczkowski added a comment -

        I like your idea with dynamic proxy. I was thinking how to make it even more generic and allow to use methods not only from interface and I ended with ProxyFactory from Javassist.

        ProxyFactory allows to create proxy for given class which intercept only calls for specific methods (even from classes higher in a hierarchy) in a convenient way (a filter can select for example the methods with "@Timeoutable" annotation). I was able to rewrite my previous PoC to use that mechanism to wrap contains() or hasSize() with a timeout mechanism. Currently it is just for one method, but doesn't depend directly on class specific information and should be possible to move it to some abstract as an one (ugly) method - much better than the next extra method for every existing method (however there could be some issues with different constructor parameters, I have to check it).

        Sample code for ListAssert (note: it is just a spike - it would be polished and extract to some separate class in case of following this path)

          public ListAssert /*could be probably S */ withTimeout(final int millis) {
            try {
              ProxyFactory pf = new ProxyFactory();
              pf.setSuperclass(mySelf.getClass());
              pf.setFilter(new MethodFilter() {
                public boolean isHandled(Method method) {
                  //only those with @Timeoutable annotation
                }
              });
              MethodHandler handler = new MethodHandler() {
                public Object invoke(final Object self, final Method thisMethod, final Method proceed, final Object[] args) throws Throwable {
                  AssertDelegate delegate = new AssertDelegate() {
                    public void doAssertion() {
                      try {
                        //TODO: Ignore result - are assert methods always void?
                        proceed.invoke(self, args);
                      } catch (IllegalAccessException e) {
                        throw new RuntimeException(e); //TODO: some more concrete runtime exception will be used
                      } catch (InvocationTargetException e) {
                        if (e.getCause() instanceof AssertionError) {
                          throw (AssertionError)e.getCause();
                        } else {
                          throw new RuntimeException(e);
                        }
                      }
                    }
                  };
                  new AssertWithTimeoutImpl(100, millis).doAssertion(delegate);
                  return null; //assumed void methods
                }
              };
              ListAssert proxy = (ListAssert)pf.create(new Class[]{List.class /*actual.getClass() for ArrayList needs tweaking*/}, new Object[]{actual}, handler);
              return proxy;
        
            } catch (InstantiationException e) {
              throw new RuntimeException(e);
            } catch (IllegalAccessException e) {
              throw new RuntimeException(e);
            } catch (NoSuchMethodException e) {
              throw new RuntimeException(e);
            } catch (InvocationTargetException e) {
              throw new RuntimeException(e);
            }
        

        I like all those checked exceptions in Java reflection methods... (fest-reflect is very handy when I need to work with reflection )

        I'm not sure what would be more convenient - separate thread with Future and Executor or previously mentioned AssertWithTimeoutImpl working on the same thread.

        Issues:

        • Javassist cannot proxy final methods (like hasSize()) - it would be required to remove final modifiers or create an alternative/separate methods hierarchy (for those final methods only) to use by timeout proxies
        • Javassist doesn't like protected constructors - it could be possibly mitigated with some more reflection magic
        • the need to add Javassist as a fest-assert dependency (would it be a problem?)
        • possible problems with constructor parameters - it has to be investigated

        How do you like it?

        Show
        Marcin Zajaczkowski added a comment - I like your idea with dynamic proxy. I was thinking how to make it even more generic and allow to use methods not only from interface and I ended with ProxyFactory from Javassist . ProxyFactory allows to create proxy for given class which intercept only calls for specific methods (even from classes higher in a hierarchy) in a convenient way (a filter can select for example the methods with "@Timeoutable" annotation). I was able to rewrite my previous PoC to use that mechanism to wrap contains() or hasSize() with a timeout mechanism. Currently it is just for one method, but doesn't depend directly on class specific information and should be possible to move it to some abstract as an one (ugly) method - much better than the next extra method for every existing method (however there could be some issues with different constructor parameters, I have to check it). Sample code for ListAssert (note: it is just a spike - it would be polished and extract to some separate class in case of following this path) public ListAssert /*could be probably S */ withTimeout( final int millis) { try { ProxyFactory pf = new ProxyFactory(); pf.setSuperclass(mySelf.getClass()); pf.setFilter( new MethodFilter() { public boolean isHandled(Method method) { //only those with @Timeoutable annotation } }); MethodHandler handler = new MethodHandler() { public Object invoke( final Object self, final Method thisMethod, final Method proceed, final Object [] args) throws Throwable { AssertDelegate delegate = new AssertDelegate() { public void doAssertion() { try { //TODO: Ignore result - are assert methods always void? proceed.invoke(self, args); } catch (IllegalAccessException e) { throw new RuntimeException(e); //TODO: some more concrete runtime exception will be used } catch (InvocationTargetException e) { if (e.getCause() instanceof AssertionError) { throw (AssertionError)e.getCause(); } else { throw new RuntimeException(e); } } } }; new AssertWithTimeoutImpl(100, millis).doAssertion(delegate); return null ; //assumed void methods } }; ListAssert proxy = (ListAssert)pf.create( new Class []{List.class /*actual.getClass() for ArrayList needs tweaking*/}, new Object []{actual}, handler); return proxy; } catch (InstantiationException e) { throw new RuntimeException(e); } catch (IllegalAccessException e) { throw new RuntimeException(e); } catch (NoSuchMethodException e) { throw new RuntimeException(e); } catch (InvocationTargetException e) { throw new RuntimeException(e); } I like all those checked exceptions in Java reflection methods... (fest-reflect is very handy when I need to work with reflection ) I'm not sure what would be more convenient - separate thread with Future and Executor or previously mentioned AssertWithTimeoutImpl working on the same thread. Issues: Javassist cannot proxy final methods (like hasSize()) - it would be required to remove final modifiers or create an alternative/separate methods hierarchy (for those final methods only) to use by timeout proxies Javassist doesn't like protected constructors - it could be possibly mitigated with some more reflection magic the need to add Javassist as a fest-assert dependency (would it be a problem?) possible problems with constructor parameters - it has to be investigated How do you like it?
        Hide
        Joel Costigliola added a comment - - edited

        I have to say that we are reluctant to external dependencies, I suggest thus to explore other approaches first (and the dynamic proxy among them) to see what it gives.
        To allow us experimenting different options I have created a FEST-475 branch on fest-assert-2.x repo .

        As I'm developing other features, I won't be able to work on this right away - but don't get me wrong, I think it's a nice addition to Fest !

        Please don't hesitate to hack on FEST-475 branch, github is so convenient to share code that it would be a pity not to use it.

        Show
        Joel Costigliola added a comment - - edited I have to say that we are reluctant to external dependencies, I suggest thus to explore other approaches first (and the dynamic proxy among them) to see what it gives. To allow us experimenting different options I have created a FEST-475 branch on fest-assert-2.x repo . As I'm developing other features, I won't be able to work on this right away - but don't get me wrong, I think it's a nice addition to Fest ! Please don't hesitate to hack on FEST-475 branch, github is so convenient to share code that it would be a pity not to use it.
        Hide
        Marcin Zajaczkowski added a comment -

        Hello again. Recently I have accidentally found a library awaitility which the main purpose is to make assertions with timeout possible to do in a convenient way. Currently it supports only Hamcrest, but a little bit of hacking I was able to implement functionality similar to previously posted without touching fest-assert classes at all. There is already all required timeout related stuff implemented I think it could be a better idea that hacking into fest-assert internal classes.

        There is also one important thing. Thanks to its custom proxy mechanism, it is possible to make something I already deleted from my original the features list - ability to assert with timeout on immutable types (with some limitations).
        In the following example (with Hamcrest):

        await().untilCall(to(fakeRepository).getValue(), greaterThan(0));

        to() static method creates a proxy which "retake" an immutable value from the object on each assertion withing specified time period. But there is one problem .

        I like very much type safe fluent API in fest-assert, but it makes it harder to use it from other libraries. I was able with some hints from the awaitility author to using some hacks (and 4 Callable structures) make it work (still PoC) with fest-assert and immutable types:

        awaitFest().untilCall(to(fakeRepository).getValue()).isGreaterThan(0);

        but it is needed to call assertThat directly in the implementation inside awaitility to keep fluent interface faced to the user and still control it. It needs a lot of boilerplate code (twice as much like in Assertions class). The similar proxy on *Assert to pasted in my last comment was also used, but the main problem remains the same (to that in pure fest-assert solution) - final methods in fest-assert.
        It seems to be not possible to proxy them. Even worst - they are silently ignored. There are unfortunately many final method in *Assert classes (e.g. AbstractComparableAssert) which make assertion with timeout to work only partially.

        I have two question.
        1. In case you had some time to look into maybe you would have some better idea how fest-assert could be integrated into awaitlity (the discussion, my PoC, awaitility documentation)?
        2. I know that methods marked as final should not be overridden in deriving assertion classes, but what would it an impact of removing it (final) in fest-assert 2.0? Or maybe you see the other way how would they be proxied?

        Show
        Marcin Zajaczkowski added a comment - Hello again. Recently I have accidentally found a library awaitility which the main purpose is to make assertions with timeout possible to do in a convenient way. Currently it supports only Hamcrest, but a little bit of hacking I was able to implement functionality similar to previously posted without touching fest-assert classes at all. There is already all required timeout related stuff implemented I think it could be a better idea that hacking into fest-assert internal classes. There is also one important thing. Thanks to its custom proxy mechanism, it is possible to make something I already deleted from my original the features list - ability to assert with timeout on immutable types (with some limitations). In the following example (with Hamcrest): await().untilCall(to(fakeRepository).getValue(), greaterThan(0)); to() static method creates a proxy which "retake" an immutable value from the object on each assertion withing specified time period. But there is one problem . I like very much type safe fluent API in fest-assert, but it makes it harder to use it from other libraries. I was able with some hints from the awaitility author to using some hacks (and 4 Callable structures) make it work (still PoC) with fest-assert and immutable types: awaitFest().untilCall(to(fakeRepository).getValue()).isGreaterThan(0); but it is needed to call assertThat directly in the implementation inside awaitility to keep fluent interface faced to the user and still control it. It needs a lot of boilerplate code (twice as much like in Assertions class). The similar proxy on *Assert to pasted in my last comment was also used, but the main problem remains the same (to that in pure fest-assert solution) - final methods in fest-assert. It seems to be not possible to proxy them. Even worst - they are silently ignored. There are unfortunately many final method in *Assert classes (e.g. AbstractComparableAssert) which make assertion with timeout to work only partially. I have two question. 1. In case you had some time to look into maybe you would have some better idea how fest-assert could be integrated into awaitlity ( the discussion , my PoC , awaitility documentation )? 2. I know that methods marked as final should not be overridden in deriving assertion classes, but what would it an impact of removing it (final) in fest-assert 2.0? Or maybe you see the other way how would they be proxied?
        Hide
        Joel Costigliola added a comment -

        Awaitility seems indeed promising, we really need to have a deep look at your suggestion (which may takes time).
        Concerning the final methods, I have no problem to "unfinal" them unless there was a really good reason for them to be final - I'll take care of that point (https://github.com/alexruiz/fest-assert-2.x/issues/52).

        Thanks for your work !

        Show
        Joel Costigliola added a comment - Awaitility seems indeed promising, we really need to have a deep look at your suggestion (which may takes time). Concerning the final methods, I have no problem to "unfinal" them unless there was a really good reason for them to be final - I'll take care of that point ( https://github.com/alexruiz/fest-assert-2.x/issues/52 ). Thanks for your work !
        Hide
        Marcin Zajaczkowski added a comment -

        Sure, take your time. It doesn't look easy, but I hope you will have some cleaver proposal to make the integration simpler and cleaner.

        Great to hear about final methods. I will remove them locally for testing purposes and synchronize with original 2.x branch later. In case you find a good reason to keep some final modifiers please let me know (here or in the issue 52).

        Btw, I've already seen you reached the milestone 5. Do you have any time schedule for 2.0-final release (not to hustle you, just it would help me to schedule timeout related work - to not be surprised that today is the last day to propose changes for 2.0 )?

        Show
        Marcin Zajaczkowski added a comment - Sure, take your time. It doesn't look easy, but I hope you will have some cleaver proposal to make the integration simpler and cleaner. Great to hear about final methods. I will remove them locally for testing purposes and synchronize with original 2.x branch later. In case you find a good reason to keep some final modifiers please let me know (here or in the issue 52). Btw, I've already seen you reached the milestone 5. Do you have any time schedule for 2.0-final release (not to hustle you, just it would help me to schedule timeout related work - to not be surprised that today is the last day to propose changes for 2.0 )?
        Hide
        Joel Costigliola added a comment -

        I don't have a final release date for 2.0, I would like to close the old google code fest site, update easytesting.org site and clean some stuff before making the official 2.0 release.

        I'm gonna continue making milestones until then but don't worry even if 2.0 is released, if something is worth to be added then I'll make another release, the idea there is to release often even if the release is small (much like jenkins).

        final methods will removed in 2.0M6 that will be released next week very likely.

        Show
        Joel Costigliola added a comment - I don't have a final release date for 2.0, I would like to close the old google code fest site, update easytesting.org site and clean some stuff before making the official 2.0 release. I'm gonna continue making milestones until then but don't worry even if 2.0 is released, if something is worth to be added then I'll make another release, the idea there is to release often even if the release is small (much like jenkins). final methods will removed in 2.0M6 that will be released next week very likely.
        Hide
        Marcin Zajaczkowski added a comment -

        Just a notice for people finding this issue though the search engine. In the end I was able to implement AssertJ integration in Awaitility using lambda expressions from Java 8. More info in my blog post: http://solidsoft.wordpress.com/2014/04/29/using-assertj-and-awaitility-together-thanks-to-java-8-and-lambdas/

        Show
        Marcin Zajaczkowski added a comment - Just a notice for people finding this issue though the search engine. In the end I was able to implement AssertJ integration in Awaitility using lambda expressions from Java 8. More info in my blog post: http://solidsoft.wordpress.com/2014/04/29/using-assertj-and-awaitility-together-thanks-to-java-8-and-lambdas/

          People

          • Assignee:
            Unassigned
            Reporter:
            Marcin Zajaczkowski
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated: