wang zhengkui's Page

Is there a reason you want to get multiple map outputs to a single reduce task?
Do you want the data to be fully sorted and grouped by key?

The simplest way is to change the partitioner class so that you get all of the data you want in one single map output.

There is nothing stopping you from creating multiple output files in hdfs or other shared file system, in your map tasks, and passing the names of these files to your reducer via the output collect, or some other mechanism.

You would loose out on the framework handling the sorting for you.

As another alternative that is somewhat io expensive, is to have 2 map/reduce jobs, one of which has only a map output,
the other only a reduce, where the parttioner assigns the reduce task based on the output file form the previous job, such that you get all of the outputs you want in each of your reduces.

The downside of this is that the output data comes from hdfs through the map task back to hdfs
then back into the identity mapper, through the local disk, then http to the reducers.
so you have two extra passes through hdfs and an extra pass through the identitymapper.