Hadoop Professionals

A Community for Hadoop Users

HI, I am facing a problem during job execution. The exception is like: "Task attempt_20110404183_0009_r_0000_0" failed to report status for 600 seconds. killing! I think the exception is because I have a very time consuming operations in the Close() function in the reducers.

 

I think one of the solutions is: change the "mapred.task.timeout" to a bigger values. But I am still wondering, is there any other way to solve this problem? Thanks.

 

My application requirement is: I need to get all the data passed to reducer, and then based on all the data I have got in one reducer, I need to do one analysis task.  Because the operation can only start after getting all the values in the reducer, I have to store the data in memory and do the time-consuming task in the Close() function.

Views: 36

Reply to This




Groups

© 2012   Created by Jason Venner.

Badges  |  Report an Issue  |  Terms of Service