Hadoop Professionals

A Community for Hadoop Users

Aprile Matteo

Launching a shell command in reduce method with Java

Hi guys,
We're some students of University of Fisciano(SA-Italy).  We're wrinting a MapReduce program for a chemistry problem and we need to launch a shell command in our reduce method. This is the code:

*Process p = Runtime.getRuntime().exec("hadoop fs -cp /source/* /dest");
try
{
p.waitFor();
}
catch (Exception e) 
{
e.getMessage();
}

Thi is another mode to call a shell command but we have the same negative result....
Shell.execCommand("hadoop fs -cp /projects/* /" + name_dir);

and when we launch the cluster, we've this stack trace error:

java.io.IOException: Cannot run program "hadoop fs -cp /projects/* /1": java.io.IOException: error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
at org.apache.hadoop
*.util.Shell.execCommand(Shell.java:354)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:337)
at cloud.yasara.MapReduceYasara$YasaraReducer.reduce(MapReduceYasara.java:95)
at cloud.yasara.MapReduceYasara$YasaraReducer.reduce(MapReduceYasara.java:1)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1214)
at 
*org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1227)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1091)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:512)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:585)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child
*.main(Child.java:170)
Caused by: java.io.IOException: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 15 more
 Can someone help us???
Thanks in advance
Matteo Aprile




Views: 136

Reply to This

Replies to This Discussion

I doubt that the hadoop script is in your path at task execution time.
You can check the value of the PATH environment variable. You may also try giving an explicit path starting from /.

In addition most of the commands you are trying to invoke via the shell in your example are available to you via the FsShell class.
thx a lot for your answer but I must execute an esternal script that call a molecolar program. This command is "./yasara ........"
I alreay tried to use the FsShell class but it compare the same exception. I think that I could be a security error ...
My inclination is to thing this is a PATH environment variable issue or an issue with the command string you are trying to execute.

When I have to debug this class of problem, I pass a simple shell script via the distributed cache
-file test.sh
which must be executable

and within my java, I would launch ./test.sh

in the test.sh file. This should give you a feel for the runtime environment for any program you are trying to run.

#! /bin/sh

pwd
ls -l
echo $PATH
exit 0

Aprile Matteo said:
thx a lot for your answer but I must execute an esternal script that call a molecolar program. This command is "./yasara ........"
I alreay tried to use the FsShell class but it compare the same exception. I think that I could be a security error ...

Reply to Discussion

RSS




Groups

© 2012   Created by Jason Venner.

Badges  |  Report an Issue  |  Terms of Service