Elton Tian's Discussions

Set LOG.Info in FileInputFormat.java doesn't work!?
1 Reply

Hello everyone,problem here. I am trying to get some runtime info in FileInputFormat.java and I try to use LOG.info(). However it never seems write anything in log file. I tried to grep the logs dir…

Started this discussion. Last reply by Elton Tian Jun 15.

How Hadoop MR delete maps output files in reduce's local disk when clean up?
1 Reply

Hello all,I am going thru source code of mapreduce part. For experiment purpose, I try to retain the tmp directories created on node's local file system when a mapreduce job is running, i.e. "map.loc…

Tagged: output, map, cleanup

Started this discussion. Last reply by Elton Tian Jun 2.

What it's gonna happend when it comes to large number of maps?
2 Replies

Hello everyone, I know when map func generates intermediate output, reduce func will pull data directly from all maps' local disk. Although we can use combiner func to minimize the amount of data, wh…

Started this discussion. Last reply by Elton Tian Apr 19.

Question: One instance of Hadoop can only run one Map Reduce Job at once?

I got a question as title. It just popped up in my head. And I think it's right.Unless, Hadoop instance can start multiple JobTrackers, and there's some resource manager coordinates JobTrackers from…

Started Mar 17

View All

Elton Tian's Page

Gifts Received

Elton Tian has not received any gifts yet

Give Elton Tian a Gift

Latest Activity

Elton Tian replied to Elton Tian's discussion 'Set LOG.Info in FileInputFormat.java doesn't work!?'

so... there's no one else run into this before? Is there any other way we can get value of variables at runtime? please advice Cheers

June 14

Elton Tian added a discussion

Set LOG.Info in FileInputFormat.java doesn't work!?

June 3

Elton Tian replied to Elton Tian's discussion 'How Hadoop MR delete maps output files in reduce's local disk when clean up?'

PS: I have tried to set "keep.task.files.pattern" to "attemp_*", in order to keep all the attemp folders in local/tasktracker/jobcache folder. Still the same ....

June 2

Elton Tian added a discussion

How Hadoop MR delete maps output files in reduce's local disk when clean up?

June 1

Elton Tian replied to Elton Tian's discussion 'What it's gonna happend when it comes to large number of maps?'

Thanks for reply Jason, Hmmm... so I think that point worths some optimization. Maybe the combiner can be extended a bit to, say, rack level, so intermediate output produced from nodes on the same rack can be merged and stored (somewhere?) before p…

April 18

Jason Venner replied to Elton Tian's discussion 'What it's gonna happend when it comes to large number of maps?'

I regularly run jobs with 20k map tasks. The shuffle can take quite a while, and if the jobs pass a lot of data to the reduce phase, load down the networking layer substantially. It does just work though.

April 16

Elton Tian added a discussion

What it's gonna happend when it comes to large number of maps?

April 15

Elton Tian added a discussion

Question: One instance of Hadoop can only run one Map Reduce Job at once?

March 17

Elton Tian added a discussion

When should I jump on HBase rather than RDBMS?

Hello everyone,I read through some literature and end up with some ideas on HBase and RDBMS. Please correct me if I am wrong:* Use HBase if the application is going to handle large datasets, like Petabytes. That means when scalability is a big conce…

March 15

Elton Tian is now a member of Hadoop Professionals

Welcome Them!

March 15