Flume 案例实操 - 单数据源多出口案例(一)

逃离我推掉我的手 2022-01-30 11:57 280阅读 0赞

案例实操 - 单数据源多出口案例(一)

## 拓扑结构 ##

![10][]

## 案例需求 ##

1.  使用flume-1监控文件变动，
2.  flume-1将变动内容传递给flume-2，flume-2负责存储到HDFS
3.  flume-1将变动内容传递给flume-3，flume-3负责输出到local filesystem。

![单数据源多出口案例（一）][5ce0e03ce4b06c0492f5d69d.png]

## 实现步骤 ##

### 准备工作 ###

在`/opt/module/flume-1.7.0/job`目录下创建`group1`文件夹

mkdir group1
    cd group1/

在`/opt/module/datas/`目录下创建`flume_sink`文件夹

mkdir flume_sink

### 创建配置文件 ###

#### 创建flume-file-flume.conf ####

vim flume-file-flume.conf

添加如下内容

# Name the components on this agent
    a1.sources = r1
    a1.sinks = k1 k2
    a1.channels = c1 c2
    # 将数据流复制给多个channel
    a1.sources.r1.selector.type = replicating
    
    # Describe/configure the source
    a1.sources.r1.type = exec
    a1.sources.r1.command = tail -F /opt/module/datas/flume_tmp.log
    a1.sources.r1.shell = /bin/bash -c
    
    # Describe the sink
    a1.sinks.k1.type = avro
    a1.sinks.k1.hostname = hadoop102
    a1.sinks.k1.port = 4141
    
    a1.sinks.k2.type = avro
    a1.sinks.k2.hostname = hadoop102
    a1.sinks.k2.port = 4142
    
    # Describe the channel
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    
    a1.channels.c2.type = memory
    a1.channels.c2.capacity = 1000
    a1.channels.c2.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1 c2
    a1.sinks.k1.channel = c1
    a1.sinks.k2.channel = c2

#### 创建flume-flume-hdfs.conf ####

vim flume-flume-hdfs.conf

添加如下内容

# Name the components on this agent
    a2.sources = r1
    a2.sinks = k1
    a2.channels = c1
    
    # Describe/configure the source
    a2.sources.r1.type = avro
    a2.sources.r1.bind = hadoop102
    a2.sources.r1.port = 4141
    
    # Describe the sink
    a2.sinks.k1.type = hdfs
    a2.sinks.k1.hdfs.path = hdfs://hadoop102:9000/flume2/%Y%m%d/%H
    #上传文件的前缀
    a2.sinks.k1.hdfs.filePrefix = flume2-
    #是否按照时间滚动文件夹
    a2.sinks.k1.hdfs.round = true
    #多少时间单位创建一个新的文件夹
    a2.sinks.k1.hdfs.roundValue = 1
    #重新定义时间单位
    a2.sinks.k1.hdfs.roundUnit = hour
    #是否使用本地时间戳
    a2.sinks.k1.hdfs.useLocalTimeStamp = true
    #积攒多少个Event才flush到HDFS一次
    a2.sinks.k1.hdfs.batchSize = 100
    #设置文件类型，可支持压缩
    a2.sinks.k1.hdfs.fileType = DataStream
    #多久生成一个新的文件
    a2.sinks.k1.hdfs.rollInterval = 600
    #设置每个文件的滚动大小大概是128M
    a2.sinks.k1.hdfs.rollSize = 134217700
    #文件的滚动与Event数量无关
    a2.sinks.k1.hdfs.rollCount = 0
    #最小冗余数
    a2.sinks.k1.hdfs.minBlockReplicas = 1
    
    # Describe the channel
    a2.channels.c1.type = memory
    a2.channels.c1.capacity = 1000
    a2.channels.c1.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a2.sources.r1.channels = c1
    a2.sinks.k1.channel = c1

#### 创建flume-flume-dir.conf ####

vim flume-flume-dir.conf

添加如下内容

# Name the components on this agent
    a3.sources = r1
    a3.sinks = k1
    a3.channels = c2
    
    # Describe/configure the source
    a3.sources.r1.type = avro
    a3.sources.r1.bind = hadoop102
    a3.sources.r1.port = 4142
    
    # Describe the sink
    a3.sinks.k1.type = file_roll
    a3.sinks.k1.sink.directory = /opt/module/datas/flume_sink
    
    # Describe the channel
    a3.channels.c2.type = memory
    a3.channels.c2.capacity = 1000
    a3.channels.c2.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a3.sources.r1.channels = c2
    a3.sinks.k1.channel = c2

注意：输出的本地目录必须是已经存在的目录，如果该目录不存在，并不会创建新的目录。

### 执行配置文件 ###

bin/flume-ng agent --conf conf/ --name a3 --conf-file job/group1/flume-flume-dir.conf
    bin/flume-ng agent --conf conf/ --name a2 --conf-file job/group1/flume-flume-hdfs.conf
    bin/flume-ng agent --conf conf/ --name a1 --conf-file job/group1/flume-file-flume.conf

## 结果 ##

![\-w1189][-w1189]

![\-w453][-w453]

[10]: /images/20220130/451f4eb91a4d41dea841e2983404d029.png
[5ce0e03ce4b06c0492f5d69d.png]: /images/20220130/e31b3932404049eb9e07e69b778bd72b.png
[-w1189]: /images/20220130/a6305bc3c97f4649b11e8ee688d2a92a.png
[-w453]: /images/20220130/3351775f886c438b98451b4963c41829.png