A Community for Hadoop Users
- wang zhengkui
- Share
- Share on Twitter
- Share on Facebook
- Blog Posts (1)
- Discussions (6)
- Events
- Groups (1)
- Photos
- Photo Albums
wang zhengkui's Discussions
Is there any difference between the following two cases?
1 Reply
My application needs to have very big computation requirement, which means that the processed data size is not big, but the computation is complicated. What I would like to ask is if I set one machin…
Started this discussion. Last reply by Jason Venner May 15.
An error occurs when did long time processing
Dear all, when I use my program to do small data processing, it works well. But when I use the program to do the longer time processing, it throws the errors. Does anyone have any idea why this happ…
Started May 13
How many reducers should I set suitably?
1 Reply
I want to know for a job If I have know the datasize, how many reducers do I need to set? How to count the suitable value of reducers number in my program? Is there any formulation of it? Thanks!
Started this discussion. Last reply by Jason Venner Oct. 30, 2009.
How to disable sort in hadoop
3 Replies
Dear all, If in my application, I do not need the hadoop to sort the intermediate result for me. How can I disable the sort in the application? Because sorting needs time. But actually, I don`t want…
Started this discussion. Last reply by Jason Venner Oct. 9, 2009.
wang zhengkui's Page
Gifts Received
wang zhengkui has not received any gifts yet
Latest Activity
Profile Information
- Hadoop Experience Level
- Intermediate
- Available for Consulting
- Yes
Wang zhengkui's Blog
Two requirements on Hadoop
Firstly, if I want to let the reducers to fetch more partitions files from map out put, is that ok? For instance, now reducer one can fetch all the partition 1 from mappers, how I implement that reducer one can fetch all the partition 1 and also 2 to go to reducer 1? If can , How could I implement that?… Continue
Posted on September 16, 2009 at 7:04am —
Comment Wall (1 comment)
- At 11:50pm on October 5, 2009,
Jason Venner said…
- Is there a reason you want to get multiple map outputs to a single reduce task?
Do you want the data to be fully sorted and grouped by key?
The simplest way is to change the partitioner class so that you get all of the data you want in one single map output.
There is nothing stopping you from creating multiple output files in hdfs or other shared file system, in your map tasks, and passing the names of these files to your reducer via the output collect, or some other mechanism.
You would loose out on the framework handling the sorting for you.
As another alternative that is somewhat io expensive, is to have 2 map/reduce jobs, one of which has only a map output,
the other only a reduce, where the parttioner assigns the reduce task based on the output file form the previous job, such that you get all of the outputs you want in each of your reduces.
The downside of this is that the output data comes from hdfs through the map task back to hdfs
then back into the identity mapper, through the local disk, then http to the reducers.
so you have two extra passes through hdfs and an extra pass through the identitymapper.
Groups
-
NoSql
2 members
-
Karma Sphere Studio Users
1 member
-
HBase Users
7 members
-
Hadoop Cascading
1 member
-
Hadoop Administrators
3 members
© 2010 Created by Jason Venner.
Powered by 
.
