Hadoop Professionals

A Community for Hadoop Users

Jason Venner

Setting the default file system or job tracker for a hadoop job.

You have 3 basic ways to set the defaults.

you can set them in your -site.xml file (hadoop-site.xml, or mapred-site.xml and hdfs-site.xml file for 20+).
you can set them on the command line

if you are using the bin/hadoop script of your distribution and running your jobs via the jar command,
-fs file_system_url or -jt jobtracker_host:port will override whatever the default is, when placed on the command line after the jar and before the actual jar_file.

Finally you can set them on the configuration object

fs.default.name is the paramter for the file system and mapred.job.tracker is the parameter for the jobtracker.

If conf is a JobConf object,

conf.set("fs.default.name", "hdfs://host:port"); will set the default hdfs filesystem
and
conf.set("mapred.job.tracker","host:port"); will set the job tracker.

Both of these have to be set before the job is submitted :).

Views: 12

Comment

You need to be a member of Hadoop Professionals to add comments!

Join Hadoop Professionals




Groups

© 2012   Created by Jason Venner.

Badges  |  Report an Issue  |  Terms of Service