由于需要在Flume里面加入一些我需要的代码,这时候就需要重新编译Flume代码,因为在编译Flume源码的时候出现了很多问题,所以写出这篇博客,以此分享给那些也需要编译代码的人一些参考,这里以如何编译Flume-0.9.4源码为例进行说明。
首先下载Flume0.9.4源码(可以到https://repository.cloudera.com/content/repositories/releases/com/cloudera/flume-distribution/0.9.4-cdh4.0.0/里面下载)之后,解压及编译:
[wyp@master Desktop]$ unzip flume-distribution-0.9.4-cdh4.0.0-project.zip [wyp@master Desktop]$ cd flume-0.9.4-cdh4.0.0/ [wyp@master flume-0.9.4-cdh4.0.0]$ mvn package -Pdist -DskipTests -Dtar
这时候会出现下面的错误:
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Flume ............................................. SUCCESS [0.003s] [INFO] Flume Core ........................................ FAILURE [5.016s] [INFO] Flume Master Config Web Application ............... SKIPPED [INFO] Flume Node Web .................................... SKIPPED [INFO] Flume Distribution Project ........................ SKIPPED [INFO] A log4j appender for Flume ........................ SKIPPED [INFO] Flume Hello World Plugin .......................... SKIPPED [INFO] Flume HBase Plugin ................................ SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5.798s [INFO] Finished at: Wed Jan 22 16:35:13 CST 2014 [INFO] Final Memory: 15M/136M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "dist" could not be activated because it does not exist. [ERROR] Failed to execute goal org.apache.thrift.tools:maven-thrift-plugin: 0.1.10:compile (default) on project flume-core: thrift did not exit cleanly. Review output for more information. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :flume-core
这是因为编译Flume的时候需要用到thrift插件,可以参考flume-core/pom.xml文件里面的配置:
<plugin> <groupId>org.apache.thrift.tools</groupId> <artifactId>maven-thrift-plugin</artifactId> <version>0.1.10</version> <configuration> <thriftExecutable>${thrift.executable}</thriftExecutable> </configuration> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> </plugin>
所以需要安装thrift,可以参考这里进行安装:《Flume-0.9.4源码编译依赖的thrift插件安装》安装完thrift,再环境变量里面设置一下THRIFT_HOME=/usr/local,因为在flume-core/pom.xml里面会用到thrift,如下:
<properties> <!-- NB: The version of the thrift compiler must match that of the dependency on the jar file below. --> <thrift.executable>${env.THRIFT_HOME}/bin/thrift</thrift.executable> <!-- Set default encoding to UTF-8 to remove maven complaints --> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <!-- defaults for flaky test and focused test exclusions --> <test.exclude.pattern>$</test.exclude.pattern> <!-- junk pattern --> <test.include.pattern>**/Test*.java</test.include.pattern> </properties>
接着再重新编译Flume源码,这时候将会出现大量类似以下的错误信息:
[WARNING] The requested profile "dist" could not be activated because it does not exist. [ERROR] Failed to execute goal org.apache.maven.plugins: maven-compiler-plugin:2.3.2:compile (default-compile) on project flume-core: Compilation failure: Compilation failure: [ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/ thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[9,39] 错误: 程序包org.apache.commons.lang3.builder不存在 [ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/ thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[10,31] 错误: 程序包org.apache.thrift.scheme不存在 [ERROR] /home/wyp/Desktop/flume-src/flume-core/target/generated-sources/ thrift/com/cloudera/flume/conf/thrift/ThriftFlumeClientServer.java:[11,31] 错误: 程序包org.apache.thrift.scheme不存在 ######################################################################### 下面省略了很多类似的错误信息 #########################################################################
请将pom.xml里面libthrift依赖版本修改为0.9.1,如下:
<dependency> <groupid>org.apache.thrift</groupid> <artifactid>libthrift</artifactid> <version>0.6.0-cdh</version> </dependency> 改为 <dependency> <groupid>org.apache.thrift</groupid> <artifactid>libthrift</artifactid> <version>0.9.1</version> </dependency>
修改完后再编译。紧接着的错误是类似下面的错误信息:
[WARNING] The requested profile "dist" could not be activated because it does not exist. [ERROR] Failed to execute goal on project flume-config-web: Could not resolve dependencies for project com.cloudera:flume-config-web:war:0.9.4-cdh4.0.0: Failure to find org.apache.hadoop:hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT in https://repository.cloudera.com/content/groups/public/ was cached in the local repository, resolution will not be reattempted until the update interval of com.cloudera.repository.releases has elapsed or updates are forced -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors,re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :flume-config-web
从错误信息可以得到编译Flume的时候依赖的hadoop-core:jar:0.20.2-cdh3u3-SNAPSHOT不能成功下载,解决办法是到pom.xml里面进行如下修改
<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.2-cdh3u3-SNAPSHOT</version> <exclusions> <exclusion> <groupId>com.cloudera.cdh</groupId> <artifactId>hadoop-ant</artifactId> </exclusion> </exclusions> </dependency> 改为 <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-core</artifactId> <version>0.20.2-cdh3u3</version> <exclusions> <exclusion> <groupId>com.cloudera.cdh</groupId> <artifactId>hadoop-ant</artifactId> </exclusion> </exclusions> </dependency>
修改完保存再编译源码。这时候应该可以顺利的进行编译,最后编译成功的界面如下:
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Flume ............................................. SUCCESS [0.003s] [INFO] Flume Core ........................................ SUCCESS [12.344s] [INFO] Flume Master Config Web Application ............... SUCCESS [3.243s] [INFO] Flume Node Web .................................... SUCCESS [0.885s] [INFO] Flume Distribution Project ........................ SUCCESS [25.381s] [INFO] A log4j appender for Flume ........................ SUCCESS [0.251s] [INFO] Flume Hello World Plugin .......................... SUCCESS [0.133s] [INFO] Flume HBase Plugin ................................ SUCCESS [20:42.199s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 21:25.138s [INFO] Finished at: Thu Jan 23 00:50:28 CST 2014 [INFO] Final Memory: 42M/286M [INFO] ------------------------------------------------------------------------ [WARNING] The requested profile "dist" could not be activated because it does not exist. [wyp@master flume-0.9.4-cdh4.0.0]$
恭喜你,编译成功了!
本博客文章除特别声明,全部都是原创!原创文章版权归过往记忆大数据(过往记忆)所有,未经许可不得转载。
本文链接: 【Flume-0.9.4源码编译及一些编译出错解决方法】(https://www.iteblog.com/archives/909.html)