轻松入门大数据第13章进阶实战：如何基于Flume实战日志数据收集 13.1 日志收集框架Flume进阶 13.1.6 avrosink和avrosource配对使用

我会带着你远行 2024-03-16 15:19 18阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第13章 进阶实战：如何基于Flume实战日志数据收集
         *   *  13.1 日志收集框架Flume进阶
             *   *  13.1.6 avrosink和avrosource配对使用

![在这里插入图片描述][1576126c72f941dcad44f35a9609fb2c.png_pic_center]

#### 第13章 进阶实战：如何基于Flume实战日志数据收集 ####

##### 13.1 日志收集框架Flume进阶 #####

> 1.  文件数据收集到HDFS
> 2.  文件夹数据收集到HDFS
> 3.  **TAILDIR方式收集**
> 4.  **avrosink配合avrosource使用**
> 5.  **Channel Selector实战**
> 6.  **Sink Processor实战**

###### 13.1.6 avrosink和avrosource配对使用 ######

官网文档：[https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html][https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html]

![在这里插入图片描述][6fcf480e514b494091bf4cac8fc2809a.png_pic_center]

往地下滑[https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html\#setting-multi-agent-flow][https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html_setting-multi-agent-flow]

![在这里插入图片描述][34fe911b9a7e4d439869cb37c5bf3edd.png_pic_center]

![在这里插入图片描述][29fd58dffbcf498e873097078e6d4c79.png_pic_center]

为了在多个代理或跃点之间传输数据，前一个代理的接收器和当前跃点的源需要是 avro 类型，接收器指向源的主机名（或 IP 地址）和端口。

这里就有俩Agent

模拟一下：

![在这里插入图片描述][8f1f647ac2364d72896ee87ee563fba4.png_pic_center]

当然我们这里就直接把两个Agent 启动在一台上进行实验了

修改配置文件 ①

官网文档：[https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html\#avro-sink][https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html_avro-sink]

![在这里插入图片描述][7608fda9333544e0ad8fa84231a1eb33.png_pic_center]

`exec-memory-avro.conf`

exec-memory-avro.sources = exec-source
    exec-memory-avro.sinks = avro-sink
    exec-memory-avro.channels = memory-channel
    
    exec-memory-avro.sources.exec-source.type = exec
    exec-memory-avro.sources.exec-source.command = tail -F /home/hadoop/data/avro.data
    
    exec-memory-avro.channels.memory-channel.type = memory
    
    exec-memory-avro.sinks.avro-sink.type = avro
    exec-memory-avro.sinks.avro-sink.hostname = 0.0.0.0
    exec-memory-avro.sinks.avro-sink.port = 44445
    
    exec-memory-avro.sources.exec-source.channels = memory-channel
    exec-memory-avro.sinks.avro-sink.channel = memory-channel

![在这里插入图片描述][38d5c888447f45519840f48142877618.png_pic_center]

把文件也创建出来

![在这里插入图片描述][3dd7cd3cc13d4c39a24e3f742b7d1d1d.png_pic_center]

配置文件 ② 【接】

官网文档：[https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html\#avro-source][https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html_avro-source]

![在这里插入图片描述][566aa7bcd17e4cf388b06985e4c3434b.png_pic_center]

`avro-memory-logger.conf`

avro-memory-logger.sources = avro-source
    avro-memory-logger.sinks = logger-sink
    avro-memory-logger.channels = memory-channel
    
    avro-memory-logger.sources.avro-source.type = avro
    avro-memory-logger.sources.avro-source.bind = 0.0.0.0
    avro-memory-logger.sources.avro-source.port = 44445
    
    avro-memory-logger.channels.memory-channel.type = memory
    
    avro-memory-logger.sinks.logger-sink.type = logger
    
    avro-memory-logger.sources.avro-source.channels = memory-channel
    avro-memory-logger.sinks.logger-sink.channel = memory-channel

创建出来

![在这里插入图片描述][780fa1d743de4192be8a46901221d62e.png_pic_center]

注意启动时，要先启动下游

运行：

flume-ng agent \
    --name avro-memory-logger \
    --conf $FLUME_HOME/conf \
    --conf-file /home/hadoop/config/avro-memory-logger.conf \
    -Dflume.root.logger=INFO,console

![在这里插入图片描述][1778e76373184824a891bef63ba45d6e.png_pic_center]

还有一个

flume-ng agent \
    --name exec-memory-avro \
    --conf $FLUME_HOME/conf \
    --conf-file /home/hadoop/config/exec-memory-avro.conf \
    -Dflume.root.logger=INFO,console

再开一个窗口

![在这里插入图片描述][8dc08e244a0b42bb814a0b66c379afd7.png_pic_center]

OK，没问题

来个脚本往`avro.data` 中写入数据

for i in {1..100}; do echo "dingjiaxiong $i" >> /home/hadoop/data/avro.data; sleep 0.1; done

![在这里插入图片描述][d12c0707b50047bca8f2db0890c18884.png_pic_center]

OK，看下游日志

![在这里插入图片描述][b14d0de366884ee3bc3b7701bf0fc06d.png_pic_center]

没问题，都拿到了

【注意】两端的hostname和port 一定要对应上，【当然我们这里是一台机器了】，而且一定要先启动下游Agent，再启动上游Agent。

[1576126c72f941dcad44f35a9609fb2c.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/ac8c03868c894bb49bcb6a9ddd36c581.png
[https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html]: https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
[6fcf480e514b494091bf4cac8fc2809a.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/1765185663354dc8a0e6a22a1848d168.png
[https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html_setting-multi-agent-flow]: https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#setting-multi-agent-flow
[34fe911b9a7e4d439869cb37c5bf3edd.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/7f3d185ddd5a4ed59390ade0899b2f27.png
[29fd58dffbcf498e873097078e6d4c79.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/0e2b0913802d42e8842ae0dd88893764.png
[8f1f647ac2364d72896ee87ee563fba4.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/d7f7e141032e484ca4e28d91b0479d3a.png
[https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html_avro-sink]: https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#avro-sink
[7608fda9333544e0ad8fa84231a1eb33.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/04a4e07e5f204a93865498156899d91c.png
[38d5c888447f45519840f48142877618.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/21d441ff96884a299b0c5d62f89254d6.png
[3dd7cd3cc13d4c39a24e3f742b7d1d1d.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/a61dd401adc4487aa79e8f702da15590.png
[https_flume.apache.org_releases_content_1.9.0_FlumeUserGuide.html_avro-source]: https://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#avro-source
[566aa7bcd17e4cf388b06985e4c3434b.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/ced21c745c8f4ab89d8213950464f7e4.png
[780fa1d743de4192be8a46901221d62e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/9bb679f31cd14ce889bfa5c6eb37f0fd.png
[1778e76373184824a891bef63ba45d6e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/e2fa97ed36aa400ca4eac42bbcddb187.png
[8dc08e244a0b42bb814a0b66c379afd7.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/e8ed3ef66ff0478a8f2832cca1e6215f.png
[d12c0707b50047bca8f2db0890c18884.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/dc7d6f7df1814a49b15ec0c0b69f5fb2.png
[b14d0de366884ee3bc3b7701bf0fc06d.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/16/408fae04f1c848cbaa12e2b8f77383b3.png