impala&hive大数据平台数据血缘与数据地图(二)-实时采集impala血缘日志推送到kafka

红太狼 2023-07-24 03:06 3阅读 0赞

**impala数据血缘与数据地图系列：**

[1. 解析impala与hive的血缘日志][1. _impala_hive]

[2. 实时采集impala血缘日志推送到kafka][2. _impala_kafka]

\-----------------------------------------**实时采集impala血缘日志推送到kafka**\-----------------------------------------------------

# 使用filebeat采集impala的血缘日志并推送到kafka #

采用filebeat的主要原因是因为轻量，对impala的血缘日志采集不需要进行数据过滤和格式转换，因此不需要使用flume或logstash这样占用资源较大的工具。

filebeat的安装及使用请参考官方手册:

[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html][https_www.elastic.co_guide_en_beats_filebeat_current_filebeat-overview.html]

参数配置：

*vim conf/filebeat\_impala\_lineage.yml*

#=========================== Filebeat inputs =============================
    
    filebeat.inputs:
    
    - type: log
    
      # Change to true to enable this input configuration.
      enabled: true
    
      # Paths that should be crawled and fetched. Glob based paths.
      paths:
        #这里指定impala血缘目录，会读取该目录下所有日志
        - /var/log/impalad/lineage/*
    #============================= Filebeat modules ===============================
    
    filebeat.config.modules:
      # Glob pattern for configuration loading
      path: ${path.config}/modules.d/*.yml
    
      # Set to true to enable config reloading
      reload.enabled: false
    
      # Period on which files under path should be checked for changes
      #reload.period: 10s
    
    #===========kafka output===============
    output.kafka:
        #指定kafka的节点和topic
      hosts: ["uatka01:9092","uatka02:9092","uatka03:9092"]
    
      topic: wyk_filebeat_impala_lineage_new_demo
    
      required_acks: 1
    #output.console:
    #  pretty: true

# DEMO： #

启动**filebeat**，注意每个机器上只能启动一个filebeat进程，因此上面的读取日志不要指定文件名。

*$FILEBEAT\_HOME/filebeat --c $FILEBEAT\_HOME/conf/filebeat\_impala\_lineage.yml -e*

启动**kafka consumer**:

*./kafka-console-consumer.sh --bootstrap-server uatka01:9092,uatka02:9092,uatka03:9092 --topic wyk\_filebeat\_impala\_lineage\_new\_demo --zookeeper uatka01:2181,uatka02:2181,uatka03:2181*

启动**impala-shell**:

*impala-shell -i uathd03*

1. 在impala-shell内建一个视图:vw\_lineage\_test11

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70][]

2.查看impala lineage 日志文件，血缘已记录日志:

![20200409111910767.png][]

3. 查看filebeat控制台，已监听日志文件并写入kafka topic内:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1][]

4. 查看kafka consumer是否消费到该血缘记录:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 2][]

# 流程结束： #

**impalaSQL--> impala血缘日志-->Filebeat-->Kafka**

完成监控impala脚本并将血缘日志推送到kafka内。

后续只需要实时消费kafka里的信息即可。

[1. _impala_hive]: https://blog.csdn.net/wsdc0521/article/details/105403734
[2. _impala_kafka]: https://blog.csdn.net/wsdc0521/article/details/105404576
[https_www.elastic.co_guide_en_beats_filebeat_current_filebeat-overview.html]: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70]: /images/20230528/ddbea2810b654d589e20b3b8057ace9b.png
[20200409111910767.png]: /images/20230528/85ee2049f2ca4e6d9b9ef5cd70ebcde4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1]: /images/20230528/8fbd2d55e2054595b557b127558da914.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 2]: /images/20230528/147ca78235ae4fefb0cb71cce2e71ccd.png