Flume集群搭建

1. 概念

集群的意思是多台机器，最少有2台机器，一台机器从数据源中获取数据，将数据传送到另一台机器上，然后输出。接下来就要实现Flume集群搭建。集群如下图所示。

2. Flume搭建

2.1 部署准备

部署主机

192.168.9.139 host14
192.168.9.128 host15
host14主机下载flume软件包

cd /opt/tools
wget http://mirrors.tuna.tsinghua.edu.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
上传解压flume

mkdir -p /apps/svr/flume/
tar -zxvf /opt/tools/apache-flume-1.7.0-bin.tar.gz -C /apps/svr/flume/

2.2 部署Flume

部署的是集群，需要在2台机安装Flume，host14作为push推送数据，host15作为pull获取数据后显示出来。

修改配置文件

cd /apps/svr/flume/apache-flume-1.7.0-bin/conf/
cp flume-env.sh.template flume-env.sh
cp flume-conf.properties.template flume-telent.conf
vim flume-env.sh
export JAVA_HOME=/apps/svr/java/jdk1.8.0_172
host15主机部署Flume

scp -r /apps/svr/flume/ host15:/apps/svr/
验证flume

/apps/svr/flume/apache-flume-1.7.0-bin/bin/flume-ng version
Flume 1.7.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
From source with checksum 0d21b3ffdc55a07e1d08875872c00523

2.3 Flume集群配置

配置push.conf

[host14]

# cd /apps/svr/flume/apache-flume-1.7.0-bin/conf
# vim push.conf

# Name the components on this agent
a2.sources= r1
a2.sinks= k1
a2.channels= c1
# Describe/configure the source
a2.sources.r1.type= spooldir
a2.sources.r1.spoolDir= /apps/svr/flume/logs
a2.sources.r1.channels= c1
# Use a channel which buffers events in memory
a2.channels.c1.type= memory
a2.channels.c1.keep-alive= 10
a2.channels.c1.capacity= 100000
a2.channels.c1.transactionCapacity= 100000
# Describe/configure the source
a2.sinks.k1.type= avro
a2.sinks.k1.channel= c1
a2.sinks.k1.hostname= host15
a2.sinks.k1.port= 8899

配置pull.conf

[host15]
# cd /apps/svr/flume/apache-flume-1.7.0-bin/conf
# vim pull.conf

# Name the components on this agent
a1.sources= r1
a1.sinks= k1
a1.channels= c1
# Describe/configure the source
a1.sources.r1.type= avro
a1.sources.r1.channels= c1
a1.sources.r1.bind= host15
a1.sources.r1.port= 8899
# Describe the sink
a1.sinks.k1.type= logger
a1.sinks.k1.channel = c1
# Use a channel which buffers events in memory
a1.channels.c1.type= memory
a1.channels.c1.keep-alive= 10
a1.channels.c1.capacity= 100000
a1.channels.c1.transactionCapacity= 100000