A Community for Hadoop Users
How to disable sort in hadoop
Dear all, If in my application, I do not need the hadoop to sort the intermediate result for me. How can I disable the sort in the application? Because sorting needs time. But actually, I don`t want it to be sorted. Thanks!
Views: 231
Replies to This Discussion
-
Permalink Reply by Jason Venner on -
If you set the number of reduce tasks to 0, there will be no sorting. There will also be no reduce phase.
In hadoop through 19, the JobConf object provides a method setNumReduceTasks, and the parameter behind it is mapred.reduce.tasks.
I do not know the hadoop 20 equivalents.
-
Permalink Reply by wang zhengkui on -
Thanks Jason. Does this mean, if my numReduceTask doesn`t equal to 0, hadoop must sort the intermediate result? If my reduce number doesn`t equal to 0, is there anyway I do not let it sort my intermediate result?
-
Permalink Reply by Jason Venner on
-
If your number of reduce tasks is not 0, the hadoop framework will sort your results. there is no way around it.
Groups
-
NoSql
6 members
-
Karma Sphere Studio Users
4 members
-
HBase Users
11 members
-
Hadoop Cascading
1 member
-
Hadoop Administrators
9 members
© 2012 Created by Jason Venner.
