Raj Vishwanathan's Discussions

Seperate Server pools for Map and Reduce Systems
2 Replies

Started this discussion. Last reply by Raj Vishwanathan Jul 27, 2010.

View All

Raj Vishwanathan's Page

Gifts Received

Raj Vishwanathan has not received any gifts yet

Give Raj Vishwanathan a Gift

Latest Activity

Raj Vishwanathan replied to Raj Vishwanathan's discussion 'Seperate Server pools for Map and Reduce Systems'

Thanks Jason. I do have full control over my cluster , so I did do what you suggested, split the cluster into two and run two map/reduce jobs. It was a nuisance but worked.

Jul 27, 2010

Jason Venner replied to Raj Vishwanathan's discussion 'Seperate Server pools for Map and Reduce Systems'

There does not appear to be any simple way to split the pools. I have had similar job constraints - building solr indexes in the reduce step. There may be a way to do this with the scheduling system. The first solution that comes to mind is to run…

Jul 27, 2010

Seperate Server pools for Map and Reduce Systems

We have lots of servers but have limited storage pool. My map jobs are handle lots of small input files (approx 300Mb Compressed) but the reduce input is huge ( about 100Gb) requiring lots of temporary and local storage. I would like to divide my server pool into two kinds - one set with a small disks ( for map jobs) and a few with big storage ( for the combine and reduce jobs).Is there something I can do that lets me force the reduce job to run on a specific nodes?I have done google searching…See More

Discussion posted by Raj Vishwanathan Jul 21, 2010

2 Comments

Raj Vishwanathan is now a member of Hadoop Professionals Jul 21, 2010

Welcome Them!