【ClickHouse系列】基于clickhouse-exporter的ClickHouse监控搭建

拼搏现实的明天。 2023-02-17 14:30 410阅读 0赞

本文是想让快速搭建这一套监控组件，让大家看到最终效果，所以各个组件均采用docker的方式启动，其实使用docker-compose会更快，单考虑到熟悉成本，以及个组件的关系，本文分组件一一进行说明并启动，最后显示效果。

## 编译构建clickhouse-exporter镜像 ##

`clickhouse_exporter`组件是`Percona-Lab`提供的，原理是依赖clickhouse自身完善的指标系统表实现的，本质上就是去查询了clickhouse的`system.metrics`等系统表获取到所需的信息，所以源码很简单，一部分是查询系统表构，另一部分就是构造prometheus数据格式，有兴趣可以自行看下源码。

言归正传，首先就是要下载源码，通过自带的Dockerfile构建即可，

[root@k8s-master test]# git clone https://github.com/Percona-Lab/clickhouse_exporter.git
    [root@k8s-master clickhouse_exporter]# cd clickhouse_exporter
    [root@k8s-master clickhouse_exporter]# docker build -t xiedeyantu/clickhouse_exporter .

这里注意下，如果你的系统centos，可能会由于docker版本比较低出现以下错误

[root@k8s-master clickhouse_exporter]# docker build -t xiedeyantu/clickhouse_exporter .
    Sending build context to Docker daemon 6.638 MB
    Step 1/10 : FROM golang:1.11 AS BUILD
    Error parsing reference: "golang:1.11 AS BUILD" is not a valid repository/tag: invalid reference format

具体升级方法参考：[docker版本升级][docker]

[root@k8s-master clickhouse_exporter]# docker -v
    Docker version 19.03.11, build 42e35e61f3

由于go编译需要依赖一些`golang.org`的包，所以网络受限的可以直接使用`f1yegor/clickhouse-exporter`镜像

## 启动clickhouse-server ##

docker run -d \
    --name clickhouse \
    --hostname clickhouse \
    -p 8123:8123 \
    yandex/clickhouse-server:20.4.4.18

确认clickhouse正常启动即可

## 启动clickhouse-exporter ##

clickhouse-exporter需要配置clickhouse的ip和port，对应参数为`scrape_uri`，使用docker命令查询

[root@k8s-master clickhouse_exporter]# docker inspect -f='{
       {.NetworkSettings.IPAddress}} {
       {.Config.Hostname}}' $(sudo docker ps -a -q)
    172.17.0.2 clickhouse

创建clickhouse-exporter docker实例

docker run -d \
    --name clickhouse-exporter \
    --hostname clickhouse-exporter \
    -p 9116:9116 \
    f1yegor/clickhouse-exporter \
    -scrape_uri=http://172.17.0.2:8123/

查看metrics

curl http://192.168.80.139:9116/metrics
    
    # HELP clickhouse_arena_alloc_bytes_total Number of ArenaAllocBytes total processed
    # TYPE clickhouse_arena_alloc_bytes_total counter
    clickhouse_arena_alloc_bytes_total 4096
    # HELP clickhouse_arena_alloc_chunks_total Number of ArenaAllocChunks total processed
    # TYPE clickhouse_arena_alloc_chunks_total counter
    clickhouse_arena_alloc_chunks_total 1
    # HELP clickhouse_background_buffer_flush_schedule_pool_task Number of BackgroundBufferFlushSchedulePoolTask currently processed
    # TYPE clickhouse_background_buffer_flush_schedule_pool_task gauge
    ......

## 启动prometheus ##

首先要配置prometheus的targets，让其能知道在哪个target拉取，配置文件如下，其中`172.17.0.3`是同样方法取到的clickhouse-exporter的ip

global:
      scrape_interval:     15s
      evaluation_interval: 15s
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']
      - job_name: 'clickhouse_exporter'
        scrape_interval: 5s
        static_configs:
        - targets: ['172.17.0.3:9116']

启动prometheus docker实例

docker run -d \
      --name prometheus \
      -p 9090:9090 \
      -v /opt/monitor/prometheus.yml:/etc/prometheus/prometheus.yml \
      prom/prometheus

浏览器打开[http://192.168.80.139:9090/targets][http_192.168.80.139_9090_targets]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center]

可见clickhouse\_exporter可正常拉取到

## 启动grafana ##

docker run -d \
      -p 3000:3000 \
      --name=grafana \
      grafana/grafana

登陆grafana，账号密码是admin/admin

配置grafana数据源，查询prometheus的ip地址（同上）

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 1]

点击Data & Test，显示working即为成功

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 2]

添加dashboard，通过import方式，直接使用882模板

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 3]

选则Prometheus数据源

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 4]

最后即可看到监控dashboard

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 5]

[docker]: https://github.com/xej520/Record-Share-Progress/blob/master/003---docker/007---%E5%A6%82%E4%BD%95%E5%8D%87%E7%BA%A7docker%E7%9A%84%E7%89%88%E6%9C%AC.md
[http_192.168.80.139_9090_targets]: http://192.168.80.139:9090/targets
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center]: https://img-blog.csdnimg.cn/2020061416244080.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20200614162456746.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/2020061416252544.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 3]: https://img-blog.csdnimg.cn/20200614162536500.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 4]: https://img-blog.csdnimg.cn/20200614162555862.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA_size_16_color_FFFFFF_t_70_pic_center 5]: https://img-blog.csdnimg.cn/20200614162607886.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zOTk5MjQ4MA==,size_16,color_FFFFFF,t_70#pic_center