gpars (Groovy Parallel Systems)
  1. gpars (Groovy Parallel Systems)
  2. GPARS-221

A wait_for_all function should be implemented for df operators

    Details

    • Type: Improvement Improvement
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: Dataflow
    • Labels:
      None
    • Number of attachments :
      0

      Description

      It should be possible to stop a df network once all messages have been processed.
      We could either use a central "graph" object monitoring the whole network, consider using a management channel, or propagate an EOF event through the regular channels.
      In that case, channels should provide open/close methods to count interested parties and send EOF down the stream once the counter reaches zero. Both operators and selectors need to handle EOF correctly. Selectors close only after all unguarded input channels report EOF.
      Also, EOFs should be filterred out from the network output channels so that thay do not propagade beyond the network boundaries.

      Multiple readers of a queue need all to receive EOF and PoisonPill
      When maxForks>1 special care must be taken to close only after all forks finish - both operators and selectors

      onEOF lifecycle event handler might be added to read channels to optionally overwrite EOF or to throw a custom exception

      In case a central graph object is created, consider these:
      Unify Kanban-flow and graph
      Allow for multiple PGroups per graph
      Introduce the notion of nodes, edges and graphs

        Activity

        Hide
        Nick Chen added a comment -

        Vaclav, thanks for the suggestion on augmenting PoisonPill.

        I want to point out that as a proof-of-concept, I have been trying to implement the central graph feature. You can find the details on my flowgraph-prototype branch on Github. You can find the diff here. Not all features of the dataflow are implemented yet. The general ideas are:

        1. Create a FlowGraph class that knows the operators participating in the network.
        2. Create a specialized version of AsyncMessagingCore that will "count" the number of messages and transitions from Active/Passive
        3. Used the new CountedCompleters introduced recently by Doug Lea in jsr166y

        I've added a couple of test cases that mimic the existing ones inside GPars for operators. On one simple image processing app that I have, the performance hit was about ~2% on a Core i7.

        On a related note (I might post more on the gpars-dev mailing list), I have also been trying to get some microbenchmarks for measurements. I have been looking at the benchmarks package but it seems to be rather sparse at the moment. I think it would be valuable to use something like Google caliper to do the benchmarks. Then we can have a record of the performance of the different constructs on different configurations (machine, jvm, etc) for comparison and performance testing, e.g., Google caliper can produce graphs.

        Show
        Nick Chen added a comment - Vaclav, thanks for the suggestion on augmenting PoisonPill . I want to point out that as a proof-of-concept, I have been trying to implement the central graph feature. You can find the details on my flowgraph-prototype branch on Github. You can find the diff here . Not all features of the dataflow are implemented yet. The general ideas are: Create a FlowGraph class that knows the operators participating in the network. Create a specialized version of AsyncMessagingCore that will "count" the number of messages and transitions from Active/Passive Used the new CountedCompleters introduced recently by Doug Lea in jsr166y I've added a couple of test cases that mimic the existing ones inside GPars for operators. On one simple image processing app that I have, the performance hit was about ~2% on a Core i7. On a related note (I might post more on the gpars-dev mailing list), I have also been trying to get some microbenchmarks for measurements. I have been looking at the benchmarks package but it seems to be rather sparse at the moment. I think it would be valuable to use something like Google caliper to do the benchmarks. Then we can have a record of the performance of the different constructs on different configurations (machine, jvm, etc) for comparison and performance testing, e.g., Google caliper can produce graphs .
        Hide
        Nick Chen added a comment -

        Vaclav, I just remembered that using {{PoisonPill} from Java with parameterized channels has a potential limitation:

        DataflowQueue<Integer> channel = new DataflowQueue<Integer>();
        channel.bind(1); // No problem
        channel.bind(PoisonPill.getInstance()); // this is a type error
        

        So to use this from Java you would need to forgo the convenience of the type parameter.

        Show
        Nick Chen added a comment - Vaclav, I just remembered that using {{PoisonPill} from Java with parameterized channels has a potential limitation: DataflowQueue< Integer > channel = new DataflowQueue< Integer >(); channel.bind(1); // No problem channel.bind(PoisonPill.getInstance()); // this is a type error So to use this from Java you would need to forgo the convenience of the type parameter.
        Hide
        Vaclav Pech added a comment -

        Nick, thanks for your comments. I'll take a look at your implementation (busy times over here, so I might need some time, unfortunately). I'm sure it will help me clarify my thoughts about the matter.
        In fact, dataflow operators is the last remaining big piece of functionality that we have to finish before we can start polishing our way towards the 1.0 release.

        Show
        Vaclav Pech added a comment - Nick, thanks for your comments. I'll take a look at your implementation (busy times over here, so I might need some time, unfortunately). I'm sure it will help me clarify my thoughts about the matter. In fact, dataflow operators is the last remaining big piece of functionality that we have to finish before we can start polishing our way towards the 1.0 release.
        Hide
        Vaclav Pech added a comment -

        Now, with operator lifecycle events in place I'm experimenting with a more generic way to monitor the state of operators and their input channels. Since all the monitoring would be taken care of by plain listeners, we could use this graceful shutdown mechanism to all networks, even if the operators have been created by the usual factory mehods.

        Show
        Vaclav Pech added a comment - Now, with operator lifecycle events in place I'm experimenting with a more generic way to monitor the state of operators and their input channels. Since all the monitoring would be taken care of by plain listeners, we could use this graceful shutdown mechanism to all networks, even if the operators have been created by the usual factory mehods.
        Show
        Vaclav Pech added a comment - http://gpars.org/SNAPSHOT/guide/guide/dataflow.html#dataflow_operators_shutdown

          People

          • Assignee:
            Vaclav Pech
            Reporter:
            Vaclav Pech
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: