hadoop I use the command hadoop jar contrib/streaming/hadoop-streaming-1.0.0.jar-input input-output output-mapper / bin / cat-file test.sh-reducer ; test.sh
shell script test.sh:
#! / bin / bash
hadoop fs-get input/1.avi ~ / hadoop-1.0.0/tmp/1.avi
error message is:
12/06/01 16:31:25 ERROR streaming.StreamJob: Job not successful. Error: # of failed Reduce Tasks exceeded allowed ; limit. FailedCount: 1. LastFailedTask: task_201206011340_0024_r_000000
12/06/01 16:31:25 INFO streaming.StreamJob: killJob ...
Streaming Job Failed!
------ Solution ------------------------------------ --------
reduce tasks have to hang up, you can see on the web console jobtracker specific fault stack http://jobmaster:port, port is probably 50,070 or 50,030 Port
------ For reference only ------------------------------------- -
finally someone responds.
because in reduce.sh call the hadoop fs-get commands like grep or cat if the change to do what no problem,
reduce the return value back to the task is not 0, hadoop default will try four times, four times, after they reported the entire mission fails,
Is there a shell script called hadoop command experience, how to solve it? Seek advice
------ For reference only ------------------------------------ ---
looked wrong as you say, or do not know where is the problem, I hadoop novice, help see what is the problem?
java.lang.RuntimeException: PipeMapRed.waitOutputThreads (): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads (PipeMapRed.java: 311)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished (PipeMapRed.java: 545)
at org.apache.hadoop.streaming.PipeMapper.close (PipeMapper.java: 132)
at org.apache.hadoop.mapred.MapRunner.run (MapRunner.java: 57)
at org.apache.hadoop.streaming.PipeMapRunner.run (PipeMapRunner.java: 36)
at org.apache.hadoop.mapred.MapTask.runOldMapper (MapTask.java: 436)
at org.apache.hadoop.mapred.MapTask.run (MapTask.java: 372)
at org.apache.hadoop.mapred.Child $ 4.run (Child.java: 255)
at java.security.AccessController.doPrivileged (Native Method)
at javax.security.auth.Subject.doAs (Subject.java: 396)
at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java: 1083)
at org.apache.hadoop.mapred.Child.main (Child.java: 249)
------ For reference only -------------- -------------------------
after suffering solve some problems, does not recognize the hadoop command, you need to shell script export hadoop environment Variables
------ For reference only ------------------------------------- -
Well then good ah
没有评论:
发表评论