A Community for Hadoop Users
HI, I am facing a problem during job execution. The exception is like: "Task attempt_20110404183_0009_r_0000_0" failed to report status for 600 seconds. killing! I think the exception is because I have a very time consuming operations in the Close() function in the reducers.
I think one of the solutions is: change the "mapred.task.timeout" to a bigger values. But I am still wondering, is there any other way to solve this problem? Thanks.
My application requirement is: I need to get all the data passed to reducer, and then based on all the data I have got in one reducer, I need to do one analysis task. Because the operation can only start after getting all the values in the reducer, I have to store the data in memory and do the time-consuming task in the Close() function.
Tags:
6 members
4 members
11 members
1 member
9 members
© 2012 Created by Jason Venner.