gpars (Groovy Parallel Systems)
  1. gpars (Groovy Parallel Systems)
  2. GPARS-105

MapReduce improvements (implicit groupBy calls and value-only for reduce) for easier usage

    Details

    • Type: Improvement Improvement
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11
    • Fix Version/s: 0.11
    • Component/s: Parallel collections
    • Labels:
      None
    • Number of attachments :
      0

      Description

      This example is based on http://jira.codehaus.org/browse/GPARS-104, just some more improvements to make more 'classic' mapreduce:

      http://gist.github.com/566942

      map/reduce in this case should be probably renamed/changed signature not to make confusion with old one

        Activity

        Hide
        Ildar Karimov added a comment -

        Actually and unfortunately I don't know too much about MR so I can't say exactly. Probably the best solution for this is
        1) to study some MR implementations like Hadoop or GridGain
        and
        2) ask some people that do know about MR - post link to this discussion to the gpars mail list (-dev and probably -user)

        Show
        Ildar Karimov added a comment - Actually and unfortunately I don't know too much about MR so I can't say exactly. Probably the best solution for this is 1) to study some MR implementations like Hadoop or GridGain and 2) ask some people that do know about MR - post link to this discussion to the gpars mail list (-dev and probably -user)
        Hide
        Vaclav Pech added a comment -

        OK, makes sence.
        A demo showing possible uses of the current 'combine' implementation is at http://github.com/vaclav/GPars/blob/master/src/test/groovy/groovyx/gpars/samples/DemoParallelWordCount.groovy

        Show
        Vaclav Pech added a comment - OK, makes sence. A demo showing possible uses of the current 'combine' implementation is at http://github.com/vaclav/GPars/blob/master/src/test/groovy/groovyx/gpars/samples/DemoParallelWordCount.groovy
        Hide
        Vaclav Pech added a comment -

        Since the implementation currently reflects my (perhaps limited) view of the map/reduce world, if no-one objects I'd close the issue.

        Show
        Vaclav Pech added a comment - Since the implementation currently reflects my (perhaps limited) view of the map/reduce world, if no-one objects I'd close the issue.
        Hide
        Ildar Karimov added a comment -

        ok, I'll open another issue when I study MR implementations and have some specific (and reasonable) ideas (if combine will be not enough)

        Show
        Ildar Karimov added a comment - ok, I'll open another issue when I study MR implementations and have some specific (and reasonable) ideas (if combine will be not enough)
        Hide
        Vaclav Pech added a comment -

        Sounds good to me. I'll always welcome your feedback.

        Show
        Vaclav Pech added a comment - Sounds good to me. I'll always welcome your feedback.

          People

          • Assignee:
            Vaclav Pech
            Reporter:
            Ildar Karimov
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: