Adeel's Discussions

do all mappers finish before reducer starts
6 Replies

I just have a conceptual question. My understanding is that all the mappers have to complete their job for the reducers to start working because mappers dont know about each other so we need values f…

Tagged: hadoop

Started this discussion. Last reply by ankit Mar 23.

View All

Adeel's Page

Gifts Received

Adeel has not received any gifts yet

Give Adeel a Gift

Latest Activity

ankit replied to Adeel's discussion 'do all mappers finish before reducer starts'

I am writing a temporary file from map function and would like to read in configure function of Reduce Class after all the maps are finished. My Question is "How do I make the configure to run after all maps"

March 23

dachuan huang replied to Adeel's discussion 'do all mappers finish before reducer starts'

based on my understanding and experiences, the reducers starts before mappers finish. because there are four stages in "reduce": copy phase, append phase, sort phase, reduce phase. exactly speaking, append phase starts after all mappers finishes, bu…

March 6

Prasen replied to Adeel's discussion 'do all mappers finish before reducer starts'

Reduce operation has 3 stages : copy,sort and then the actual-reduce is performed on copied/sorted data. Copy/Sort can start even before mapper has finished.

February 21

Jason Venner replied to Adeel's discussion 'do all mappers finish before reducer starts'

For all intents and purposes your reduce doesn't start until the reduce % hits 60% the parts that run prior to that are involved in preparing the data for your reduce tasks. It the job output is a confusing information presentation.

January 27

Adeel replied to Adeel's discussion 'do all mappers finish before reducer starts'

Thanks for your reply Jason .. and that makes perfect sense .. only thing is that when the job is runnung .. before the map tasks reach 100% status .. reducer task starts running as well ..so you will see something like this in terminal ***** map 0%…

January 27

Jason Venner replied to Adeel's discussion 'do all mappers finish before reducer starts'

A reducer can not start until all of the data that will be it's input is fully ordered. The ordering can not complete until all of the map tasks have finished, as any map may have data that will go to any reducer (reduce task). The reduce task ofte…

January 26

Adeel added a discussion

do all mappers finish before reducer starts

January 26

Adeel is now a member of Hadoop Professionals

Welcome Them!

January 26