最近修改了Spark的一些代码,然后编译Spark出现了以下的异常信息:
error file=/iteblog/spark-1.3.1/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala message=File line length exceeds 100 characters line=279 error file=/iteblog/spark-1.3.1/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala message=File line length exceeds 100 characters line=612 Saving to outputFile=/iteblog/spark-1.3.1/streaming/scalastyle-output.xml Processed 79 file(s) Found 2 errors Found 0 warnings Found 0 infos Finished in 946 ms [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM .......................... SUCCESS [3.381s] [INFO] Spark Project Networking .......................... SUCCESS [13.187s] [INFO] Spark Project Shuffle Streaming Service ........... SUCCESS [6.580s] [INFO] Spark Project Core ................................ SUCCESS [3:24.312s] [INFO] Spark Project Bagel ............................... SUCCESS [9.088s] [INFO] Spark Project GraphX .............................. SUCCESS [25.691s] [INFO] Spark Project Streaming ........................... FAILURE [24.015s] [INFO] Spark Project Catalyst ............................ SKIPPED [INFO] Spark Project SQL ................................. SKIPPED [INFO] Spark Project ML Library .......................... SKIPPED [INFO] Spark Project Tools ............................... SKIPPED [INFO] Spark Project Hive ................................ SKIPPED [INFO] Spark Project REPL ................................ SKIPPED [INFO] Spark Project YARN ................................ SKIPPED [INFO] Spark Ganglia Integration ......................... SKIPPED [INFO] Spark Project Assembly ............................ SKIPPED [INFO] Spark Project External Twitter .................... SKIPPED [INFO] Spark Project External Flume Sink ................. SKIPPED [INFO] Spark Project External Flume ...................... SKIPPED [INFO] Spark Project External MQTT ....................... SKIPPED [INFO] Spark Project External ZeroMQ ..................... SKIPPED [INFO] Spark Project External Kafka ...................... SKIPPED [INFO] Spark Project Examples ............................ SKIPPED [INFO] Spark Project YARN Shuffle Service ................ SKIPPED [INFO] Spark Project External Kafka Assembly ............. SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4:46.989s [INFO] Finished at: Wed May 20 14:56:59 CST 2015 [INFO] Final Memory: 82M/530M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.scalastyle:scalastyle-maven-plugin:0.4.0:check (default) on project spark-streaming_2.10: Failed during scalastyle execution: You have 2 Scalastyle violation(s). -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :spark-streaming_2.10
从上面出错的第一行和第二行可以看出是第279行和第612行的代码长度超过100个字符的原因,因为Spark在编译的时候会检测Scala的代码风格是否符合要求,我们可以通过下面方法解决:
修改Spark源码的pom.xml文件,找到下面的行:
<failOnViolation>true</failOnViolation>
可以修改成:
@@ -123,6 +123,7 @@ <log4j.version>1.2.17</log4j.version> <hadoop.version>1.0.4</hadoop.version> <protobuf.version>2.4.1</protobuf.version> + <scalastyle.failonviolation>true</scalastyle.failonviolation> <yarn.version>${hadoop.version}</yarn.version> <hbase.version>0.94.6</hbase.version> <flume.version>1.4.0</flume.version> @@ -1071,7 +1072,7 @@ <version>0.4.0</version> <configuration> <verbose>false</verbose> - <failOnViolation>true</failOnViolation> + <failOnViolation>${scalastyle.failonviolation}</failOnViolation> <includeTestSourceDirectory>false</includeTestSourceDirectory> <failOnWarning>false</failOnWarning> <sourceDirectory>${basedir}/src/main/scala</sourceDirectory>
上面的+是添加的,-是删除的。然后可以通过下面的方式编译:
[iteblog@iteblog ~]$ ./make-distribution.sh --tgz -Phadoop-2.6 -Pyarn -DskipTests \ -Dhadoop.version=2.6.0 -Phive -Pspark-ganglia-lgpl -Dscalastyle.failOnViolation=false
或者直接将failOnViolation
选项修改成false:
<failOnViolation>false</failOnViolation>
然后再去编译Spark代码, 这样就不会检测Scala代码规范了。
本博客文章除特别声明,全部都是原创!原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载。
本文链接: 【Spark编译错误笔记】(https://www.iteblog.com/archives/1365.html)