在今年的5月22号，Flume-ng 1.5.0版本正式发布，关于Flume-ng 1.5.0版本的新特性可以参见本博客的《Apache Flume-ng 1.5.0正式发布》进行了解。关于Apache flume-ng 1.4.0版本的编译可以参见本博客《基于Hadoop-2.2.0编译flume-ng 1.4.0及错误解决》。本文将讲述如何用Maven编译Apache flume-ng 1.5.0源码。

一、到官方网站下载相应版本的flume-ng源码

# wget http://mirror.bit.edu.cn/apache/flume/1.5.0/apache-flume-1.5.0-src.tar.gz
# tar -zxf apache-flume-1.5.0-src.tar.gz
# cd apache-flume-1.5.0-src

二、用maven在相应的Hadoop上编译Flume-ng 1.5.0源码

Flume-ng 1.5.0默认的Hadoop版本是1.0.1，Hbase是0.92.1。我们可以在编译的时候加上一些参数，指定Hadoop 2.2.0版本进行编译：

-Phadoop-2

三、Hadoop、Hbase对应的test不能下载

　　如果在编译的时候出现了类似如下的情况，请将相应pom.xml文件中的Hadoop-test和Hbase-test依赖包去掉：

<!--
 User: 过往记忆
 Date: 14-6-16
 Time: 23:45
 bolg: 
 本文地址：/archives/1043
 过往记忆博客，专注于hadoop、hive、spark、shark、flume的技术博客，大量的干货
 过往记忆博客微信公共帐号：iteblog_hadoop
-->
<dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase</artifactId>
        <version>${hbase.version}</version>
        <classifier>tests</classifier>
        <scope>test</scope>
</dependency>

<dependency>
         <groupId>org.apache.hadoop</groupId>
         <artifactId>hadoop-test</artifactId>
         <version>${hadoop.version}</version>
</dependency>

四、elasticsearch-0.90.1.jar read错误

　　如果你在编译的过程中出现一下异常：

[ERROR] Failed to execute goal org.apache.maven.plugins:
maven-compiler-plugin:2.3.2:compile (default-compile) on 
project flume-ng-elasticsearch-sink: Compilation failure
[ERROR] error: error reading 
/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar; 
error in opening zip file
[ERROR] -> [Help 1]

　　只需要到你maven本地仓库将elasticsearch-0.90.1.jar文件删除，然后重新下载即可：

# rm elasticsearch-0.90.1.jar
wget http://central.maven.org/maven2/org/elasticsearch/elasticsearch/0.90.1/elasticsearch-0.90.1.jar

五、ua-parser-1.3.0下载失败

编译过程中，下载ua-parser-1.3.0.pom可能失败，出现如下错误：

[ERROR] Failed to execute goal on project flume-ng-morphline-solr-sink:
 Could not resolve dependencies for project 
org.apache.flume.flume-ng-sinks:flume-ng-morphline-solr-sink:jar:1.5.0: 
Failed to collect dependencies for [org.apache.flume:flume-ng-core:jar:1.5.0 
(compile), org.slf4j:slf4j-api:jar:1.6.1 (compile), 
org.kitesdk:kite-morphlines-all:pom:0.12.0 (compile?),
 org.slf4j:jcl-over-slf4j:jar:1.6.1 (provided), 
org.apache.solr:solr-test-framework:jar:4.3.0 (test), 
org.kitesdk:kite-morphlines-solr-core:jar:tests:0.12.0 (test), 
junit:junit:jar:4.10 (test)]: Failed to read artifact descriptor 
for ua_parser:ua-parser:jar:1.3.0: Could not transfer artifact 
ua_parser:ua-parser:pom:1.3.0 from/to maven-twttr (http://maven.twttr.com):
 Connection to http://maven.twttr.com refused: Connection timed out -> [Help 1]

只需要加一个新的repository即可：

<repository>
  <id>nexus.axiomalaska.com</id>
  <url>http://nexus.axiomalaska.com/nexus/content/repositories/public</url>
</repository>

六、编译Flume-ng 1.5.0源码

# mvn install -Phadoop-2 -DskipTests -Dtar

编译需要一段时间，这个需要看你编译时的网速。如果一切顺利的话，你将会看到以下输出，说明你的Flume-ng 1.5.0编译成功：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Flume ...................................... SUCCESS [1.360s]
[INFO] Flume NG SDK ...................................... SUCCESS [2.903s]
[INFO] Flume NG Configuration ............................ SUCCESS [0.483s]
[INFO] Flume NG Core ..................................... SUCCESS [2.531s]
[INFO] Flume NG Sinks .................................... SUCCESS [0.110s]
[INFO] Flume NG HDFS Sink ................................ SUCCESS [1.668s]
[INFO] Flume NG IRC Sink ................................. SUCCESS [0.484s]
[INFO] Flume NG Channels ................................. SUCCESS [0.074s]
[INFO] Flume NG JDBC channel ............................. SUCCESS [0.579s]
[INFO] Flume NG file-based channel ....................... SUCCESS [0.712s]
[INFO] Flume NG Spillable Memory channel ................. SUCCESS [0.552s]
[INFO] Flume NG Node ..................................... SUCCESS [0.619s]
[INFO] Flume NG Embedded Agent ........................... SUCCESS [1.001s]
[INFO] Flume NG HBase Sink ............................... SUCCESS [2.373s]
[INFO] Flume NG ElasticSearch Sink ....................... SUCCESS [0.899s]
[INFO] Flume NG Morphline Solr Sink ...................... SUCCESS [2.483s]
[INFO] Flume NG Kite Dataset Sink ........................ SUCCESS [35.925s]
[INFO] Flume Sources ..................................... SUCCESS [0.054s]
[INFO] Flume Scribe Source ............................... SUCCESS [1.231s]
[INFO] Flume JMS Source .................................. SUCCESS [2.133s]
[INFO] Flume Twitter Source .............................. SUCCESS [1.204s]
[INFO] Flume legacy Sources .............................. SUCCESS [0.043s]
[INFO] Flume legacy Avro source .......................... SUCCESS [1.234s]
[INFO] Flume legacy Thrift Source ........................ SUCCESS [1.320s]
[INFO] Flume NG Clients .................................. SUCCESS [0.038s]
[INFO] Flume NG Log4j Appender ........................... SUCCESS [6.175s]
[INFO] Flume NG Tools .................................... SUCCESS [1.559s]
[INFO] Flume NG distribution ............................. SUCCESS [1:51.993s]
[INFO] Flume NG Integration Tests ........................ SUCCESS [1.517s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:03.921s
[INFO] Finished at: Mon Jun 16 23:37:51 CST 2014
[INFO] Final Memory: 135M/745M
[INFO] ------------------------------------------------------------------------

最后将会在flume-ng-dist/target目录下产生以下的文件：

total 29208
drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-bin
-rw-r--r-- 1 wyp wyp 27307850 Jun 16 23:36 apache-flume-1.5.0-bin.tar.gz
drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 apache-flume-1.5.0-src
-rw-r--r-- 1 wyp wyp  2543174 Jun 16 23:37 apache-flume-1.5.0-src.tar.gz
drwxr-xr-x 2 wyp wyp     4096 Jun 16 23:36 archive-tmp
drwxr-xr-x 3 wyp wyp     4096 Jun 16 23:36 maven-shared-archive-resources
-rw-r--r-- 1 wyp wyp      990 Jun 16 23:37 rat.txt

apache-flume-1.5.0-bin这个文件夹里面的东西可以直接拿来用。

本文地址：《用Maven编译Apache flume-ng 1.5.0源码及问题解决》：/archives/1043，过往记忆博客，专注于hadoop、hive、spark、shark、flume的技术博客，大量的干货.过往记忆博客微信公共帐号：iteblog_hadoop