MapReduce improvements (implicit groupBy calls and value-only for reduce) for easier usage

May I ask for a bit more details on the rationale behind the request to be able to assess it properly.

Sure.

Implicit groupBy is similar to the combiner in the Hadoop implementation:

http://wiki.apache.org/hadoop/HadoopMapReduce
http://hadoop.apache.org/mapreduce/docs/current/mapred_tutorial.html#Walk-through

The reasons for using it are:
1) stricter map() step that produces 'real' KV pairs
2) some internal optimizations are possible when using combine instead of groupBy (e.g. memory optimizations)
3) probably we could even make combine() step available to the gpars mapreduce - that would be the best fix for this issue
Actual combine step for word count example would be very similar to inject, maybe just a bit parametrized(Object, MapEntry->Object) or probably even not (Object, Key->Object)

Value-only is used to hide mostly unused stuff and write only really used code.

I hope it is more clear now. If not, I can try to write new word count example using combine step and explain steps behind it.

Thanks for the explanation. From the docs it seems combine in Hadoop is more of an optimization step, identical in its logic with the reduce step, but perhaps you may need to correct my assumptions here.

Will the following use come closer to what you propose?

return arg.parallel.map

.combine {{0}, a, b -> a + b}.getParallel().sort {-it.value}.collection

We can't really live without an initial value for the combine operation here, I'm afraid, unless we force the user to deal with null values in accumulators.

yes, very close.

a bit more complex example with both combine in reduce(though actually producing the same result but in a slightly different way) would look like:

arg.parallel.map

.combine([])

.reduce

'combine' here uses the same syntax as inject

In your recent example, does combine remove the 'key' from further processing so that reduce only gets a collection of lists?

Also, since 'combine' accumulates items concurrently, the initial value may actually be needed multiple-times. If the combination/accumulation closure modifies the first argument (list in our case), the initial value must be cloneable (List is fine in this respect).

1) combine does not remove the key, but simply reduce does not use it (but probably might use it in some cases)
2) oops, forgot about concurrency in this step. but if we have good enough cloneable list implementation it will be ok.

I was puzzled since the reduce operation in your example
arg.parallel.map

.combine([])

.reduce

assumes the value is a number, not a key->value pair.

ok, we should not probably oversimplify and use
.reduce

.reduce

like in map.each{}

Shouldn't we use 'map' instead of 'reduce' here? I tend to think of reduce as a function that takes two items and merges them into one (possibly different), while map replaces items with new, derived ones, which seems more like what we're trying to achieve as the step after 'combine'.

Actually and unfortunately I don't know too much about MR so I can't say exactly. Probably the best solution for this is
1) to study some MR implementations like Hadoop or GridGain
and
2) ask some people that do know about MR - post link to this discussion to the gpars mail list (-dev and probably -user)

OK, makes sence.
A demo showing possible uses of the current 'combine' implementation is at http://github.com/vaclav/GPars/blob/master/src/test/groovy/groovyx/gpars/samples/DemoParallelWordCount.groovy

Since the implementation currently reflects my (perhaps limited) view of the map/reduce world, if no-one objects I'd close the issue.

ok, I'll open another issue when I study MR implementations and have some specific (and reasonable) ideas (if combine will be not enough)

Sounds good to me. I'll always welcome your feedback.