Kafka三种可视化监控管理工具monitor，manager，eagle

清疚 2023-02-19 14:29 10阅读 0赞

### 文章目录 ###

*   *  1.Kafka Monitor
     *  2.Kafka Manager
     *   *  2.1 部署
         *  2.2 总结
     *  3.Kafka Eagle

## 1.Kafka Monitor ##

> 下载链接： https://github.com/Morningstar/kafka-offset-monitor/releases/tag/0.4.6

1.  上传jar包KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar到集群
2.  在/opt/module/下创建kafka-offset-console文件夹
3.  将上传的jar包放入刚创建的目录下
4.  在/opt/module/kafka-offset-console目录下创建启动脚本start\_en.sh，内容如下：
    
        #!/bin/bash
        # 默认port为8086，由于我的环境8086端口已被占用，所有改为8087
        java -Xms512M -Xmx512M -Xss1024K -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m -cp KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar \
        com.quantifind.kafka.offsetapp.OffsetGetterWeb \
        --offsetStorage kafka \
        --kafkaBrokers cm1:9092,cm2:9092,cm3:9092 \
        --kafkaSecurityProtocol PLAINTEXT \
        --zk cm1:2181,cm2:2181,cm3:2181 \
        --port 8087 \
        --refresh 10.seconds \
        --retain 2.days \
        --dbName offsetapp_kafka 1>mobile-logs/stdout.log 2>mobile-logs/stderr.log &
5.  在/opt/module/kafka-offset-console目录下创建mobile-logs文件夹
    
        mkdir /opt/module/kafka-offset-console/mobile-logs
6.  启动KafkaMonitor
    
        ./start_en.sh
7.  登录页面cm1:8087端口查看详情

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vImBnkt0-1592707347924)(img/image-20200620180444179.png)\]][img-vImBnkt0-1592707347924_img_image-20200620180444179.png]

启动一个消费者和一个生产者

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lHIe0gpE-1592707347926)(img/image-20200620191335707.png)\]][img-lHIe0gpE-1592707347926_img_image-20200620191335707.png]  
![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Y5tg9lrt-1592707347928)(img/image-20200620191318329.png)\]][img-Y5tg9lrt-1592707347928_img_image-20200620191318329.png]

## 2.Kafka Manager ##

### 2.1 部署 ###

> https://github.com/yahoo/kafka-manager/archive/1.3.3.15.zip

1.  上传压缩包kafka-manager-1.3.3.15.zip到集群
2.  解压kafka-manager-1.3.3.15.zip
    
        $ unzip kafka-manager-1.3.3.15.zip
3.  修改kafka-manager-1.3.3.21/conf/application.conf文件启用Kafka-Manager使用账号登录
    
        # 修改为：
        kafka-manager.zkhosts="cm1:2181,cm2:2181,cm3:2181"
        basicAuthentication.enabled=true
        basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED}
        basicAuthentication.username="admin"
        basicAuthentication.username=${?KAFKA_MANAGER_USERNAME}
        basicAuthentication.password="admin"
        basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}
        basicAuthentication.realm="Kafka-Manager"
    
    将登录Kafka-Manager的账号密码设置为admin。
4.  使用如下命令启动Kafka-Manager
    
    修改文件的权限
    
        [root@cm1 bin]$ chmod 777 kafka-manager
        # 默认为9000，但是我的集群有端口冲突，改为8090
        [root@cm1 kafka-manager-1.3.3.15]$ nohup bin/kafka-manager -Dhttp.port=8090 > nohup.out 2>&1 &
    
    端口8090监听成功。
    
        [root@cm1 kafka-manager-1.3.3.15]$ netstat -apn | grep 8090
        tcp6       0      0 :::8090                 :::*                    LISTEN      32934/java
5.  在浏览器输入http://cm1:8090访问即可
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-EOZEHEmE-1592707347930)(img/image-20200620220037789.png)\]][img-EOZEHEmE-1592707347930_img_image-20200620220037789.png]
    
    到此为止Kafka-Manager就部署成功了。
6.  登录Kafka-Manager管理界面
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-5Zr3meeZ-1592707347931)(img/image-20200620220157431.png)\]][img-5Zr3meeZ-1592707347931_img_image-20200620220157431.png]
    
    点击Cluster菜单，添加一个Kafka集群
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ckBsCXpI-1592707347933)(img/image-20200620220225791.png)\]][img-ckBsCXpI-1592707347933_img_image-20200620220225791.png]
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-aQioap9q-1592707347935)(img/image-20200620220333948.png)\]][img-aQioap9q-1592707347935_img_image-20200620220333948.png]
    
    根据自己当前Kafka集群的环境进行相应的配置，点击“Save”完成集群添加
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-PU1LDib3-1592707347936)(img/image-20200620220420934.png)\]][img-PU1LDib3-1592707347936_img_image-20200620220420934.png]
    
    点击“Kafka-CDH”连接，进入Kafka集群的详细界面，下图为Kafka的统计信息
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-VynGAnBQ-1592707347937)(img/image-20200620220544687.png)\]][img-VynGAnBQ-1592707347937_img_image-20200620220544687.png]
    
    Brokers界面，可以看到当前Kafka集群的所有Broker节点及综合指标
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-R8JQ0idR-1592707347938)(img/image-20200620220608397.png)\]][img-R8JQ0idR-1592707347938_img_image-20200620220608397.png]
    
    点击相应的Broker ID可以看到每个Broker的统计信息及详细指标等
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-0Qwl5LYa-1592707347939)(img/image-20200620220644499.png)\]][img-0Qwl5LYa-1592707347939_img_image-20200620220644499.png]
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-K5Q5Ym0U-1592707347940)(img/image-20200620220708946.png)\]][img-K5Q5Ym0U-1592707347940_img_image-20200620220708946.png]
    
    在Topic菜单下，可以创建和查看集群的所有Topic
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HDg6qMcY-1592707347941)(img/image-20200620220809254.png)\]][img-HDg6qMcY-1592707347941_img_image-20200620220809254.png]
    
    创建一个Topic
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-YHOAg1RE-1592707347942)(img/image-20200620220938142.png)\]][img-YHOAg1RE-1592707347942_img_image-20200620220938142.png]

### 2.2 总结 ###

1.  Kafka-Manager支持Kafka的0.8.x，0.9.x，0.10.x，0.11.x版本
2.  Kafka需要JDK8的环境运行
3.  Kafka-Manager支持管理多个Kafka集群。
4.  可以便捷的查看Kafka集群的状态（Brokers、Topics、副本分布及分区分布情况）
5.  通过界面的方式创建Topic及为Topic重新分配分区等功能。
6.  Kafka-Manager启动默认端口号为9000，可以通过-Dhttp.port来指定端口号

## 3.Kafka Eagle ##

> 参考
> 
> https://blog.csdn.net/qq\_19524879/article/details/82848797

1.  从git上下载:https://github.com/smartloli/kafka-eagle-bin/archive/v1.4.8.tar.gz（本例选择[1.2.9版本][1.2.9]）
2.  解压（需要解压两次）
    
        cd /opt/module/
        tar -zxf kafka-eagle-bin-1.4.8.tar.gz
        cd kafka-eagle-bin-1.4.8
        tar -zxf kafka-eagle-web-1.4.8-bin.tar.gz
3.  设置全局变量，设置JDK的全局变量JAVA\_HOME和本应用的KE\_HOME
    
        export JAVA_HOME=/usr/java/jdk1.8.0_131-cloudera
        export KE_HOME=/opt/module/kafka-eagle-bin-1.4.8/kafka-eagle-web-1.4.8
        export PATH=$JAVA_HOME/bin:$KE_HOME/bin:$PATH
4.  进入kafka-eagle的conf目录下修改配置文件 system-config.properties ，配置基础配置
    
        cd /opt/module/kafka-eagle-bin-1.4.8/kafka-eagle-web-1.4.8
        
        # 修改为以下内容
        
        #设置kafka集群（可以多个）
        kafka.eagle.zk.cluster.alias=cluster1
        cluster1.zk.list=cm1:2181,cm2:2181,cm3:2181
        
        
        ######################################
        # kafka eagle 的端口
        ######################################
        kafka.eagle.webui.port=8048
        
        
        ######################################
        # kafka jdbc 地址注意可以自己安装数据mysql也可以自带的
        ######################################
        kafka.eagle.driver=org.sqlite.JDBC
        kafka.eagle.url=jdbc:sqlite:/opt/module/kafka-eagle-bin-1.4.8/kafka-eagle-web-1.4.8/db/ke.db
        kafka.eagle.username=root
        kafka.eagle.password=123456
5.  文件配置完毕后开始启动 ，启动要进入bin目录下
    
        #进入bin目录后会看到 ke.sh 文件先修改文件的权限
        chomd 777 ke.sh
        
        ke.sh start
        # 停止
        ke.sh stop
    
    启动成功后的控制台输出
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-3dIvDh25-1592707347942)(img/image-20200620224812438.png)\]][img-3dIvDh25-1592707347942_img_image-20200620224812438.png]
    
    部署完成结束
6.  下面介绍下他的页面功能
    
    功能界面
    
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-qJqTHx1R-1592707347943)(img/image-20200620225915362.png)\]][img-qJqTHx1R-1592707347943_img_image-20200620225915362.png]
    
    使用下 kafkasql查询的功能  
    ![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-G6YGVzYX-1592707347944)(img/image-20200620230530975.png)\]][img-G6YGVzYX-1592707347944_img_image-20200620230530975.png]

[img-vImBnkt0-1592707347924_img_image-20200620180444179.png]: https://img-blog.csdnimg.cn/20200621104258303.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-lHIe0gpE-1592707347926_img_image-20200620191335707.png]: https://img-blog.csdnimg.cn/20200621104330664.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-Y5tg9lrt-1592707347928_img_image-20200620191318329.png]: https://img-blog.csdnimg.cn/20200621104344112.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-EOZEHEmE-1592707347930_img_image-20200620220037789.png]: https://img-blog.csdnimg.cn/20200621104414999.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-5Zr3meeZ-1592707347931_img_image-20200620220157431.png]: https://img-blog.csdnimg.cn/20200621104425731.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-ckBsCXpI-1592707347933_img_image-20200620220225791.png]: https://img-blog.csdnimg.cn/20200621104438468.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-aQioap9q-1592707347935_img_image-20200620220333948.png]: https://img-blog.csdnimg.cn/20200621104453525.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-PU1LDib3-1592707347936_img_image-20200620220420934.png]: https://img-blog.csdnimg.cn/20200621104505870.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-VynGAnBQ-1592707347937_img_image-20200620220544687.png]: https://img-blog.csdnimg.cn/20200621104517546.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-R8JQ0idR-1592707347938_img_image-20200620220608397.png]: https://img-blog.csdnimg.cn/20200621104527553.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-0Qwl5LYa-1592707347939_img_image-20200620220644499.png]: https://img-blog.csdnimg.cn/20200621104537923.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-K5Q5Ym0U-1592707347940_img_image-20200620220708946.png]: https://img-blog.csdnimg.cn/20200621104548438.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-HDg6qMcY-1592707347941_img_image-20200620220809254.png]: https://img-blog.csdnimg.cn/20200621104600862.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-YHOAg1RE-1592707347942_img_image-20200620220938142.png]: https://img-blog.csdnimg.cn/20200621104613356.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[1.2.9]: https://github.com/smartloli/kafka-eagle-bin/archive/v1.2.9.tar.gz
[img-3dIvDh25-1592707347942_img_image-20200620224812438.png]: https://img-blog.csdnimg.cn/20200621104634669.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-qJqTHx1R-1592707347943_img_image-20200620225915362.png]: https://img-blog.csdnimg.cn/2020062110474084.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70
[img-G6YGVzYX-1592707347944_img_image-20200620230530975.png]: https://img-blog.csdnimg.cn/2020062110480643.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2ODM4MzE1,size_16,color_FFFFFF,t_70