最近在Yarn上使用Spark,不管是yarn-cluster模式还是yarn-client模式,都出现了以下的异常:
Application application_1434099279301_123706 failed 2 times due to AM Container for appattempt_1434099279301_123706_000002 exited with exitCode: 127 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Container exited with a non-zero exit code 127.Failing this attempt.. Failing the application.
而且ApplicationMaster所在机器的日志里面有下面的信息提示:
/bin/bash: {{JAVA_HOME}}/bin/java: No such file or directory
经过分析,原来是我编译Spark所使用的Hadoop版本和线上Hadoop版本不一致导致的,后来使用线上Hadoop版本重新编译了Spark,这个问题解决了。
本博客文章除特别声明,全部都是原创!原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载。
本文链接: 【Spark on yarn上遇到的问题】(https://www.iteblog.com/archives/1393.html)