A Community for Hadoop Users
- wang zhengkui
- Share
- Share on Twitter
- Share on Facebook
- Blog Posts (1)
- Discussions (4)
- Events
- Groups (1)
- Photos
- Photo Albums
wang zhengkui's Discussions
How many reducers should I set suitably?
1 Reply
I want to know for a job If I have know the datasize, how many reducers do I need to set? How to count the suitable value of reducers number in my program? Is there any formulation of it? Thanks!
Started this discussion. Last reply by Jason Venner Oct. 30, 2009.
How to disable sort in hadoop
3 Replies
Dear all, If in my application, I do not need the hadoop to sort the intermediate result for me. How can I disable the sort in the application? Because sorting needs time. But actually, I don`t want…
Started this discussion. Last reply by Jason Venner Oct. 9, 2009.
Two requirements for Hadoop
2 Replies
There are two requirements which I want to implement based on Hadoop. But , by now, I do not think that hadoop support them now. I am looking forward to your suggestion how to implement these. First…
Started this discussion. Last reply by amogh vasekar Sep. 23, 2009.
wang zhengkui's Page
Gifts Received
wang zhengkui has not received any gifts yet
Latest Activity
Profile Information
- Hadoop Experience Level
- Intermediate
- Available for Consulting
- Yes
Wang zhengkui's Blog
Two requirements on Hadoop
Firstly, if I want to let the reducers to fetch more partitions files from map out put, is that ok? For instance, now reducer one can fetch all the partition 1 from mappers, how I implement that reducer one can fetch all the partition 1 and also 2 to go to reducer 1? If can , How could I implement that?… Continue
Posted on September 16, 2009 at 7:04am —
Comment Wall (1 comment)
- At 11:50pm on October 5, 2009,
Jason Venner said…
- Is there a reason you want to get multiple map outputs to a single reduce task?
Do you want the data to be fully sorted and grouped by key?
The simplest way is to change the partitioner class so that you get all of the data you want in one single map output.
There is nothing stopping you from creating multiple output files in hdfs or other shared file system, in your map tasks, and passing the names of these files to your reducer via the output collect, or some other mechanism.
You would loose out on the framework handling the sorting for you.
As another alternative that is somewhat io expensive, is to have 2 map/reduce jobs, one of which has only a map output,
the other only a reduce, where the parttioner assigns the reduce task based on the output file form the previous job, such that you get all of the outputs you want in each of your reduces.
The downside of this is that the output data comes from hdfs through the map task back to hdfs
then back into the identity mapper, through the local disk, then http to the reducers.
so you have two extra passes through hdfs and an extra pass through the identitymapper.
About
Groups
-
Karma Sphere Studio Users
1 member
-
HBase Users
5 members
-
Hadoop Cascading
1 member
-
Hadoop Administrators
2 members
-
Katta Users
1 member
© 2010 Created by Jason Venner on Ning. Create a Ning Network!
