Prometheus+Grafana监控部署实践

你的名字 2021-11-27 02:50 403阅读 0赞

参考文档：

Prometheus github：[https://github.com/prometheus][https_github.com_prometheus]

grafana github：[https://github.com/grafana/grafana][https_github.com_grafana_grafana]

Prometheus getting\_started：[https://prometheus.io/docs/prometheus/latest/getting\_started/][https_prometheus.io_docs_prometheus_latest_getting_started]

Prometheus node\_exporter：[https://github.com/prometheus/node\_exporter][https_github.com_prometheus_node_exporter]

Prometheus部署(docker,optional)：[https://prometheus.io/docs/prometheus/latest/installation/][https_prometheus.io_docs_prometheus_latest_installation]

Prometheus配置文件详解：

[https://prometheus.io/docs/prometheus/latest/configuration/configuration/][https_prometheus.io_docs_prometheus_latest_configuration_configuration]

grafana 部署：[http://docs.grafana.org/installation/rpm/][http_docs.grafana.org_installation_rpm]

# **一．环境** #

## **1. 拓扑** ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70][]

## **2. 节点** ##

<table style="margin-left:.15pt;"> 
 <tbody> 
 <tr> 
 <td style="width:86.4pt;"> Node </td> 
 <td style="width:86.4pt;"> OS </td> 
 <td style="width:86.4pt;"> Hostname </td> 
 <td style="width:86.4pt;"> IP </td> 
 <td style="width:86.4pt;"> Remark </td> 
 </tr> 
 <tr> 
 <td style="width:86.4pt;"> prometheus&amp; grafana server </td> 
 <td style="width:86.4pt;"> centos 7.4 </td> 
 <td style="width:86.4pt;"> prometheus </td> 
 <td style="width:86.4pt;"> 172.20.1.211 </td> 
 <td style="width:86.4pt;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="width:86.4pt;"> prometheus node </td> 
 <td style="width:86.4pt;"> centos 7.4 </td> 
 <td style="width:86.4pt;"> node1 </td> 
 <td style="width:86.4pt;"> 172.20.1.212 </td> 
 <td style="width:86.4pt;"> &nbsp; </td> 
 </tr> 
 </tbody> 
</table>

## **3. 版本(截止20171130)** ##

<table style="margin-left:.15pt;"> 
 <tbody> 
 <tr> 
 <td style="width:144pt;"> Soft/Node </td> 
 <td style="width:144pt;"> Version </td> 
 <td style="width:144pt;"> Download </td> 
 </tr> 
 <tr> 
 <td style="width:144pt;"> prometheus </td> 
 <td style="width:144pt;"> 2.0.0 </td> 
 <td style="width:144pt;"> <a href="https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz" rel="nofollow">https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz</a> </td> 
 </tr> 
 <tr> 
 <td style="width:144pt;"> node_exporter </td> 
 <td style="width:144pt;"> v0.15.1 </td> 
 <td style="width:144pt;"> <a href="https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz" rel="nofollow">https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz</a> </td> 
 </tr> 
 <tr> 
 <td style="width:144pt;"> grafana </td> 
 <td style="width:144pt;"> 4.6.2-1 </td> 
 <td style="width:144pt;"> <a href="https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86_64.rpm" rel="nofollow">https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86_64.rpm</a> </td> 
 </tr> 
 </tbody> 
</table>

# **二．部署prometheus ** #

在prometheus& grafana server节点部署prometheus服务。

## **1. 下载&部署** ##

\# 下载

\[root@prometheus src\]\# cd /usr/local/src/

\[root@prometheus src\]\# wget [https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz][https_github.com_prometheus_prometheus_releases_download_v2.0.0_prometheus-2.0.0.linux-amd64.tar.gz]

\# 部署到/usr/local/目录

\# promethus不用编译安装，解压目录中有配置文件与启动文件

\[root@prometheus src\]\# tar -zxvf prometheus-2.0.0.linux-amd64.tar.gz -C /usr/local/

\[root@prometheus src\]\# cd /usr/local/

\[root@prometheus local\]\# mv prometheus-2.0.0.linux-amd64/ prometheus/

\# 验证

\[root@prometheus local\]\# cd prometheus/

\[root@prometheus prometheus\]\# ./prometheus --version

![20190723160506412.png][]

## **2. 配置文件** ##

\# 解压目录中的prometheus.yml

\# 简单验证，主要配置采用默认文件配置，有修改/新增处用红色标示

\[root@prometheus prometheus\]\# vim prometheus.yml

\# 全局配置global:

scrape\_interval:     15s \# 设置抓取(pull)时间间隔，默认是1m

evaluation\_interval: 15s \# 设置rules评估时间间隔，默认是1m

\# scrape\_timeout is set to the global default (10s).

\# 告警管理配置，暂未使用，默认配置alerting:

alertmanagers:

- static\_configs:

- targets:

\# - alertmanager:9093

\# 加载rules，并根据设置的时间间隔定期评估，暂未使用，默认配置rule\_files:

\# - "first\_rules.yml"

\# - "second\_rules.yml"

\# 抓取(pull)，即监控目标配置

\# 默认只有主机本身的监控配置scrape\_configs:

\# 监控目标的label（这里的监控目标只是一个metric，而不是指某特定主机，可以在特定主机取多个监控目标），在抓取的每条时间序列表中都会添加此label

- job\_name: 'prometheus'

\# metrics\_path defaults to '/metrics'

\# scheme defaults to 'http'.

\# 可覆盖全局配置设置的抓取间隔，由15秒重写成5秒。

scrape\_interval: 5s

\# 静态指定监控目标，暂不涉及使用一些服务发现机制发现目标static\_configs:

- targets: \['localhost:9090'\]

\# (opentional)再添加一个label，标识了监控目标的主机

labels:

instance: prometheus

- job\_name: 'linux'

scrape\_interval: 10s

static\_configs:

\# 采用node\_exporter默认开放的端口

- targets: \['172.20.1.212:9100'\]

labels:

instance: node1

## **3. 设置用户** ##

\# 添加用户，后期用此账号启动服务

\[root@prometheus prom etheus\]\# groupadd prometheus

\[root@prometheus prometheus\]\# useradd -g prometheus -s /sbin/nologin prometheus

\# 赋权

\[root@prometheus prometheus\]\# cd ~

\[root@prometheus ~\]\# chown -R prometheus:prometheus /usr/local/prometheus/

\# 创建prometheus运行数据目录

\[root@prometheus ~\]\# mkdir -p /var/lib/prometheus

\[root@prometheus ~\]\# chown -R prometheus:prometheus /var/lib/prometheus/

## **4. 设置开机启动** ##

\[root@prometheus ~\]\# touch /usr/lib/systemd/system/prometheus.service

\[root@prometheus ~\]\# chown prometheus:prometheus /usr/lib/systemd/system/prometheus.service

\[root@prometheus ~\]\# vim /usr/lib/systemd/system/prometheus.service\[Unit\]

Description=Prometheus

Documentation=https://prometheus.io/

After=network.target

\[Service\]\# Type设置为notify时，服务会不断重启

Type=simple

User=prometheus\# --storage.tsdb.path是可选项，默认数据目录在运行目录的./dada目录中

ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus

Restart=on-failure

\[Install\]

WantedBy=multi-user.target

\# 设置开机启动

\[root@prometheus ~\]\# systemctl enable Prometheus

\[root@prometheus ~\]\# systemctl start prometheus

## **5. 设置iptables** ##

\[root@prometheus ~\]\# vim /etc/sysconfig/iptables

\-A INPUT -p tcp -m state --state NEW -m tcp --dport 9090 -j ACCEPT

\[root@prometheus ~\]\# service iptables restart

## **6. 启动并验证** ##

### **1）查看服务状态** ###

\[root@prometheus ~\]\# systemctl status prometheus![20190723160538536.png][]

\[root@prometheus ~\]\# netstat -tunlp | grep 9090

![20190723160549408.png][]

### **2）web ui** ###

Prometheus自带有简单的UI，[http://172.20.1.211:9090][http_172.20.1.211_9090]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 1][]

在Status菜单下，Configuration，Rule，Targets等，

Statu-->Configuration展示prometheus.yml的配置，如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 2][]

Statu-->Targets展示监控具体的监控目标，这里监控目标"linux"暂未设置node\_exporter，未scrape数据，如下：

![20190723160641149.png][]

## **7. 绘图** ##

访问：[http://172.20.1.211:9090/metrics][http_172.20.1.211_9090_metrics]，查看从exporter具体能抓到的数据，如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 3][]

访问：[http://172.20.1.211:9090][http_172.20.1.211_9090]，在输入框中任意输入1个exporter能抓取得值，点击"Execute"与"Execute"按钮，即可见相应抓取数据的图形，同时可对时间与unit做调整，如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 4][]

# **三．部署node\_exporter** #

Node\_exporter收集机器的系统数据，这里采用prometheus官方提供的exporter，除node\_exporter外，官方还提供consul，memcached，haproxy，mysqld等exporter，具体可查看官网。

这里在prometheus node节点部署相关服务。

## **1. 下载&部署** ##

\# 下载

\[root@node1 ~\]\# cd /usr/local/src/

\[root@node1 src\]\# wget [https://github.com/prometheus/node\_exporter/releases/download/v0.15.1/node\_exporter-0.15.1.linux-amd64.tar.gz][https_github.com_prometheus_node_exporter_releases_download_v0.15.1_node_exporter-0.15.1.linux-amd64.tar.gz]

\# 部署

\[root@node1 src\]\# tar -zxvf node\_exporter-0.15.1.linux-amd64.tar.gz -C /usr/local/

\[root@node1 src\]\# cd /usr/local/

\[root@node1 local\]\# mv node\_exporter-0.15.1.linux-amd64/ node\_exporter/

## **2. 设置用户** ##

\[root@node1 ~\]\# groupadd prometheus

\[root@node1 ~\]\# useradd -g prometheus -s /sbin/nologin prometheus

\[root@node1 ~\]\# chown -R prometheus:prometheus /usr/local/node\_exporter/

## **3. 设置开机启动** ##

\[root@node1 ~\]\# vim /usr/lib/systemd/system/node\_exporter.service\[Unit\]

Description=node\_exporter

Documentation=https://prometheus.io/

After=network.target

\[Service\]

Type=simple

User=prometheus

ExecStart=/usr/local/node\_exporter/node\_exporter

Restart=on-failure

\[Install\]

WantedBy=multi-user.target

\[root@node1 ~\]\# systemctl enable node\_exporter

\[root@node1 ~\]\# systemctl start node\_exporter

## **4. 设置iptables** ##

\# 官方node\_exporter默认使用9100端口

\[root@node1 ~\]\# vim /etc/sysconfig/iptables

\-A INPUT -p tcp -m state --state NEW -m tcp --dport 9100 -j ACCEPT

\[root@node1 ~\]\# service iptables restart

## **5. 验证** ##

访问：[http://172.20.1.211:9090][http_172.20.1.211_9090]，可见node1主机已经可被监控，如下：

![20190723160721360.png][]

# **四．部署grafana** #

在prometheus& grafana server节点部署grafana服务。

## **1. 下载&安装** ##

\# 下载

\[root@prometheus ~\]\# cd /usr/local/src/

\[root@prometheus src\]\# wget [https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86\_64.rpm][https_s3-us-west-2.amazonaws.com_grafana-releases_release_grafana-4.6.2-1.x86_64.rpm]

\# 安装

\[root@prometheus src\]\# yum localinstall grafana-4.6.2-1.x86\_64.rpm

## **2. 配置文件** ##

配置文件位于/etc/grafana/grafana.ini，这里暂时保持默认配置即可。

## **3. 设置开机启动** ##

\[root@prometheus src\]\# systemctl enable grafana-server

\[root@prometheus src\]\# systemctl start grafana-server

## **4. 设置iptables** ##

\# grafana-server默认使用3000端口

\[root@prometheus src\]\# vim /etc/sysconfig/iptables

\-A INPUT -p tcp -m state --state NEW -m tcp --dport 3000 -j ACCEPT

\[root@prometheus src\]\# service iptables restart

## **5. 添加数据源 ** ##

### **1）登陆** ###

访问：[http://172.20.1.211:3000][http_172.20.1.211_3000]，默认账号/密码：admin/admin

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 5][]

### **2）添加数据源** ###

在登陆首页，点击"Add data source"按钮，跳转到添加数据源页面，配置如下：

Name: prometheus

Type: prometheus

URL: http://localhost:9090/

Access: proxy

取消Default的勾选，其余默认，点击"Add"，如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 6][]

在"Dashboards"页签下"import"自带的模版，如下：

![20190723160801217.png][]

## **6. 导入dashboard** ##

从grafana官网下载相关dashboaed到本地，如：[https://grafana.com/dashboards/405][https_grafana.com_dashboards_405]

Grafana首页-->左上角图标-->Dashboard-->import

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 7][]

Upload已下载至本地的json文件（或者使用dashboard id，如这里的405），如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 8][]

数据源选择"prometheus"，即添加的数据源name，点击"Import"按钮，如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 9][]

## **7. 查看dashboard** ##

Grafana首页-->左上角图标-->Dashboard-->Home，Home下拉列表中可见有已添加的两个dashboard，"Prometheus Stats"与"Node Exporter Server Metrics"，选择1个即可，如下：![20190723160911378.png][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 10][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 11][]

[https_github.com_prometheus]: https://github.com/prometheus
[https_github.com_grafana_grafana]: https://github.com/grafana/grafana
[https_prometheus.io_docs_prometheus_latest_getting_started]: https://prometheus.io/docs/prometheus/latest/getting_started/
[https_github.com_prometheus_node_exporter]: https://github.com/prometheus/node_exporter
[https_prometheus.io_docs_prometheus_latest_installation]: https://prometheus.io/docs/prometheus/latest/installation/
[https_prometheus.io_docs_prometheus_latest_configuration_configuration]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/
[http_docs.grafana.org_installation_rpm]: http://docs.grafana.org/installation/rpm/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70]: /images/20211126/d52363fd8a1145119f9d7f480768355e.png
[https_github.com_prometheus_prometheus_releases_download_v2.0.0_prometheus-2.0.0.linux-amd64.tar.gz]: https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz
[20190723160506412.png]: /images/20211126/cb90f493e00044899d53fa220035b4b4.png
[20190723160538536.png]: /images/20211126/6c583ec3e3684bcd8a1f144f89982b5d.png
[20190723160549408.png]: /images/20211126/7bddb52271734ec3a97e55634f94560b.png
[http_172.20.1.211_9090]: http://172.20.1.211:9090/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 1]: /images/20211126/e3409817a8ae42d198c7ad7ee5469a9a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 2]: /images/20211126/111f8cd18cad42a3ae239ed1adfe4532.png
[20190723160641149.png]: /images/20211126/2eb2633339194c568dbbd8db9532d80c.png
[http_172.20.1.211_9090_metrics]: http://172.20.1.211:9090/metrics
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 3]: /images/20211126/ba515739596d4f0f99ec1b3ae9a1f6fd.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 4]: /images/20211126/3db080a17edc464c95fc674b1dd130f0.png
[https_github.com_prometheus_node_exporter_releases_download_v0.15.1_node_exporter-0.15.1.linux-amd64.tar.gz]: https://github.com/prometheus/node_exporter/releases/download/v0.15.1/node_exporter-0.15.1.linux-amd64.tar.gz
[20190723160721360.png]: /images/20211126/3811b45c829849fe8917cf04bd620566.png
[https_s3-us-west-2.amazonaws.com_grafana-releases_release_grafana-4.6.2-1.x86_64.rpm]: https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.2-1.x86_64.rpm
[http_172.20.1.211_3000]: http://172.20.1.211:3000/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 5]: /images/20211126/a9696ec7687745da81f0c6f8e92d7242.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 6]: /images/20211126/cc5e441cdc294da08a6306bb7793d83d.png
[20190723160801217.png]: /images/20211126/7141e8949abf425c907da969edf94cc5.png
[https_grafana.com_dashboards_405]: https://grafana.com/dashboards/405
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 7]: /images/20211126/13284523c1394890bee927a41675c699.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 8]: /images/20211126/508860751cf84727af40c6d49e5984f9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 9]: /images/20211126/81b7c532de6641ea86c38f28ac637104.png
[20190723160911378.png]: https://img-blog.csdnimg.cn/20190723160911378.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 10]: /images/20211126/005bad1d27ea4373b100d885982bbf51.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2JhaWR1XzM4NDMyNzMy_size_16_color_FFFFFF_t_70 11]: /images/20211126/aabcddae39fa423bb9f3e06f579cbaed.png