Grafana离线安装Cloudera Manager插件并监控CDH集群

超、凢脫俗 2023-03-13 11:41 225阅读 0赞

[一、Grafana6.7.3安装及使用][Grafana6.7.3]

[二、InfluxDB1.1.0和1.8.0版本安装并开启web界面][InfluxDB1.1.0_1.8.0_web]

[三、使用Python操作InfluxDB时序数据库][Python_InfluxDB]

[四、使用Python监控本机资源情况写入InfluxDB并使用Grafana监控][Python_InfluxDB_Grafana]

\-----------------------------Grafana离线安装Cloudera Manager插件监控CDH集群-----------------------------

本篇介绍如何安装CM插件并通过Grafana监控CDH集群健康及资源使用情况，该插件直接调用CM里的接口。

Github：[https://github.com/foursquare/datasource-plugin-clouderamanager][https_github.com_foursquare_datasource-plugin-clouderamanager]

# 下载 #

下载：[https://grafana.com/grafana/plugins/foursquare-clouderamanager-datasource/installation][https_grafana.com_grafana_plugins_foursquare-clouderamanager-datasource_installation]

如果访问不了也可以使用附件里我下载好的插件。[点击下载][Link 1]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70][]

# 配置Grafana插件路径 #

Grafana的配置文件在/etc/grafana/grafana.ini，默认插件路径/var/lib/grafana/plugins，如果需要修改默认的插件路径可以编辑此文件，修改下面的参数。

[paths]
    # Directory where grafana will automatically scan and look for plugins
    ;plugins = /var/lib/grafana/plugins

然后将上面下载的zip包解压至该参数路径下:

*unzip foursquare-datasource-plugin-clouderamanager-v0.9.2-0-g9ac3d25.zip*

![2020051311290229.png][]

# 重启Grafana #

使用下面的命令重启Grafana:

CentOS7:
    systemctl restart grafana-server
    systemctl enable grafana-server
     
    CentOS6:
    service grafana-server restart
    chkconfig grafana-server on

重新登录Grafana，可以在datasource最下面的others里看到Cloudera Manager已经添加进来了。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1][]

# 配置CM插件数据源 #

### 1.在CM中添加只读账户： ###

使用管理员账户登录CM，点击 **管理**\-->**用户**\-->**添加用户**\-->输入账户密码选择'**只读**'角色

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 2][]

### 2.配置CM插件数据源  ###

如图配置，输入CM的地址以及上面配置的只读账户:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 3][]

![2020051311461622.png][]

# 使用CM插件监控CDH #

### 例1: 查看Impala里正在执行的查询数量监控 ###

SELECT total_impala_num_queries_registered_across_impalads as "总查询" WHERE entityName = "impala" AND category = SERVICE

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 4][]

### 例2: 组件健康状态 ###

注意这里的Spark的entityName每个集群的都不一样，有些其他的指标如Impala Daemon,hive server2等等也都有各自的entityName，怎么找到它们所对应的entityName请看下面小节。

select  (health_good_rate+health_concerning_rate) * 100 as "Hdfs" where entityName="hdfs:nameservice1"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "Yarn" where entityName="yarn"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "Hive" where entityName="hive"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "Impala" where entityName="impala"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "Hbase" where entityName="hbase"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "Kudu" where entityName="kudu"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "Spark" where entityName="spar40365358-SPARK_YARN_HISTORY_SERVER-84731ef5cb40c534e51774a39"
    
    select  (health_good_rate+health_concerning_rate) * 100 as "ZK" where entityName="zookeeper"

### ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 5][] ###

### 查找服务对应的entityName ###

在CM中有很多的图表，我们只需要把对应的图表展开即可看到它这个图表里的查询命令，那个查询命令我们完全可以直接拿到grafana里使用CM插件查询。

比如，我直接查看集群当前的CPU使用率：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 7][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 8][]

同样的方式去找上面提到的spark history server的entityName,其他服务的entityName找的方式一样，不多演示了。

点击 Spark-->实例-->HistoryServer:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 9][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 10][]

[Grafana6.7.3]: https://blog.csdn.net/wsdc0521/article/details/106072350
[InfluxDB1.1.0_1.8.0_web]: https://blog.csdn.net/wsdc0521/article/details/106064914
[Python_InfluxDB]: https://blog.csdn.net/wsdc0521/article/details/106082274
[Python_InfluxDB_Grafana]: https://blog.csdn.net/wsdc0521/article/details/106082411
[https_github.com_foursquare_datasource-plugin-clouderamanager]: https://github.com/foursquare/datasource-plugin-clouderamanager
[https_grafana.com_grafana_plugins_foursquare-clouderamanager-datasource_installation]: https://grafana.com/grafana/plugins/foursquare-clouderamanager-datasource/installation
[Link 1]: https://download.csdn.net/download/wsdc0521/12414558
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70]: /images/20230312/d8af61c882e04677b91fe8fd497f7725.png
[2020051311290229.png]: /images/20230312/f637089cb83f4c61b1b2c2f7b23a8e87.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200513113652529.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20200513114319301.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/202005131145493.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[2020051311461622.png]: https://img-blog.csdnimg.cn/2020051311461622.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200513134031456.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200513135130223.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200513135649585.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200513140042685.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200513140136359.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200513140336476.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200513140441392.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dzZGMwNTIx,size_16,color_FFFFFF,t_70