Hadoop Professionals

A Community for Hadoop Users

Adarsh
  • Male
  • Auburn, AL
  • United States
Share on Facebook Share Twitter

Adarsh's Friends

  • Amy Tierney

Adarsh's Discussions

Measure bandwidth and shuffle time during hadoop job

Hi,I am trying to evaluate the dependency of hadoop on the underlying networks and hence need to find the bandwidth and shuffle time during a hadoop job. Can anyone please let me know how to get the…Continue

Started Nov 15, 2010

Shuffle time during hadoop jobs

Hi,Can anyone please tell me how to get the status of shuffle stage during a hadoop job.I want to see the time taken for shuffling.Thanks in advance.Continue

Started Oct 26, 2010

 

Adarsh's Page

Gifts Received

Gift

Adarsh has not received any gifts yet

Give Adarsh a Gift

Latest Activity

Adarsh posted a blog post

Measure shuffle time and bandwidth

Hi,I am trying to evaluate the dependency of hadoop on theunderlying networks and hence need to find the bandwidth and shuffletime during a hadoop job. Can anyone please let me know how to get thebandwidth and the shuffle time during a job.Thanks in advanceregards,AdarshSee More
Nov 18, 2010
0 Comments
Adarsh posted a discussion

Measure bandwidth and shuffle time during hadoop job

Hi,I am trying to evaluate the dependency of hadoop on the underlying networks and hence need to find the bandwidth and shuffle time during a hadoop job. Can anyone please let me know how to get the bandwidth and the shuffle time during a job.Thanks in advanceregards,AdarshSee More
Nov 15, 2010
0 Comments
Jason Venner commented on Adarsh's blog post Upper limit for bandwidth in hadoop
"This is the bandwidth used by the balancing task, when leveling the hdfs space utilization on your data nodes. Overall for Map/Reduce tasks the bandwidth is limited by your networking bandwidth."
Nov 15, 2010
Adarsh posted a blog post

Upper limit for bandwidth in hadoop

Hi,I was going through the hdfs-default.xml and found an entry dfs.balance.bandwidthPerSec. So my question was, is there an upper limit set for bandwidth in hadoop. Please let me know if my understanding is correct.Thanks in advance,AdarshSee More
Nov 15, 2010
1 Comment
Adarsh updated their profile
Nov 10, 2010
Adarsh and Amy Tierney are now friends
Nov 10, 2010
Adarsh posted a discussion

Shuffle time during hadoop jobs

Hi,Can anyone please tell me how to get the status of shuffle stage during a hadoop job.I want to see the time taken for shuffling.Thanks in advance.See More
Oct 26, 2010
0 Comments
Adarsh is now a member of Hadoop Professionals
Oct 26, 2010
Welcome Them!

Profile Information

Hadoop Experience Level
Beginner
Interests
Parallel computing, high performance computing networks
Expertise
Java,c
Past Projects
Worked for 3 years at Infosys Technologies Ltd, Bangalore,India.
Current Project
Reasearch assisstantship at Auburn University, Alabama
Available for Consulting
Yes

Adarsh's Blog

Measure shuffle time and bandwidth

Hi,

I am trying to evaluate the dependency of hadoop on the underlying networks and hence need to find the bandwidth and shuffle
time during a hadoop job. Can anyone please let me know how to get the
bandwidth and the shuffle time during a job.

Thanks in advance

regards,
Adarsh

Posted on November 18, 2010 at 3:18pm

Upper limit for bandwidth in hadoop

Hi,

I was going through the hdfs-default.xml and found an entry dfs.balance.bandwidthPerSec. So my question was, is there an upper limit set for bandwidth in hadoop. Please let me know if my understanding is correct.

Thanks in advance,
Adarsh

Posted on November 14, 2010 at 9:57pm — 1 Comment

Comment Wall

You need to be a member of Hadoop Professionals to add comments!

Join Hadoop Professionals

  • No comments yet!
 
 
 



Groups

© 2012   Created by Jason Venner.

Badges  |  Report an Issue  |  Terms of Service

Sign in to chat!