Adeel's Discussions

do all mappers finish before reducer starts
6 Replies

Tags: hadoop

Started this discussion. Last reply by ankit Mar 23, 2010.

View All

Adeel's Page

Gifts Received

Adeel has not received any gifts yet

Give Adeel a Gift

Latest Activity

ankit replied to Adeel's discussion 'do all mappers finish before reducer starts'

I am writing a temporary file from map function and would like to read in configure function of Reduce Class after all the maps are finished. My Question is "How do I make the configure to run after all maps"

Mar 23, 2010

dachuan huang replied to Adeel's discussion 'do all mappers finish before reducer starts'

based on my understanding and experiences, the reducers starts before mappers finish. because there are four stages in "reduce": copy phase, append phase, sort phase, reduce phase. exactly speaking, append phase starts after all mappers…

Mar 7, 2010

Prasen replied to Adeel's discussion 'do all mappers finish before reducer starts'

Reduce operation has 3 stages : copy,sort and then the actual-reduce is performed on copied/sorted data. Copy/Sort can start even before mapper has finished.

Feb 22, 2010

Jason Venner replied to Adeel's discussion 'do all mappers finish before reducer starts'

For all intents and purposes your reduce doesn't start until the reduce % hits 60% the parts that run prior to that are involved in preparing the data for your reduce tasks. It the job output is a confusing information presentation.

Jan 28, 2010

Adeel replied to Adeel's discussion 'do all mappers finish before reducer starts'

Thanks for your reply Jason .. and that makes perfect sense .. only thing is that when the job is runnung .. before the map tasks reach 100% status .. reducer task starts running as well ..so you will see something like this in terminal ***** map 0%…

Jan 27, 2010

Jason Venner replied to Adeel's discussion 'do all mappers finish before reducer starts'

A reducer can not start until all of the data that will be it's input is fully ordered. The ordering can not complete until all of the map tasks have finished, as any map may have data that will go to any reducer (reduce task). The reduce task…

Jan 27, 2010

do all mappers finish before reducer starts

I just have a conceptual question. My understanding is that all the mappers have to complete their job for the reducers to start working because mappers dont know about each other so we need values for a given key from all the different mappers so we have to wait until all mappers have collectively given the system all possible values for a key .so that then that can be passed on the reducer .. but when I ran these jobs .. almost everytime before the mappers are all done the reducers start…See More

Discussion posted by Adeel Jan 27, 2010

6 Comments

Adeel is now a member of Hadoop Professionals Jan 27, 2010

Welcome Them!