基于k8s Prometheus+Grafana+Altermanager钉钉报警

约定不等于承诺〃 2022-01-17 04:51 610阅读 0赞

## 相关推荐 ##

[1. 使用Prometheus Operator监控kubetnetes集群][1. _Prometheus Operator_kubetnetes]

[2. 使用Prometheus Operator实现应用自定义监控][2. _Prometheus Operator]

## 一、概述 ##

Alertmanager与Prometheus是相互分离的两个组件。Prometheus服务器根据报警规则将警报发送给Alertmanager，然后Alertmanager将silencing、inhibition、aggregation等消息通过电子邮件、dingtalk和HipChat发送通知。

Alertmanager处理由例如Prometheus服务器等客户端发来的警报。它负责删除重复数据、分组，并将警报通过路由发送到正确的接收器，比如电子邮件、Slack、dingtalk等。Alertmanager还支持groups,silencing和警报抑制的机制。

钉钉作为内部通讯工具，基本上大家在电脑和手机上都能用，消息可以第一时间查看，报警消息的即时性要求比较高，所以适合用钉钉通知。

## 二、添加钉钉机器人 ##

请参考官方文档：[自定义机器人][Link 1]

添加机器人后获取机器人的hook（机器人好像只能在钉钉群里面添加）,在后面部署会用到。

机器人hook：[https://oapi.dingtalk.com/robot/send?access\_token=xxxxxx][https_oapi.dingtalk.com_robot_send_access_token_xxxxxx]

## 三、配置Alertmanager ##

Alertmanager官方文档：[https://github.com/prometheus/docs/blob/db2a09a8a7e193d6e474f37055908a6d432b88b5/content/docs/alerting/configuration.md\#webhook\_config][https_github.com_prometheus_docs_blob_db2a09a8a7e193d6e474f37055908a6d432b88b5_content_docs_alerting_configuration.md_webhook_config]

修改Alertmanager报警配置，因上面的官方文档已经给出来每个参数的详细信息，就不再一一解释了。

[root@node-01 prometheus]# vim prometheus-operator/values.yaml 
    
      config:
        global:
          resolve_timeout: 2m
        route:
          group_by: ['job']
          group_wait: 30s
          group_interval: 2m
          repeat_interval: 12h
          receiver: 'webhook'
          routes:
          - match:
              alertname: DeadMansSwitch
            receiver: 'webhook'
        receivers:
        - name: 'webhook'
          webhook_configs:
          - url: http://webhook-dingtalk/dingtalk/send/
            send_resolved: true

更新prometheus-operator

[root@node-01 prometheus]# helm upgrade p ./prometheus-operator

修改成功后可以在alertmanager的status页面看到相关配置  
![config][]

## 四、转换alertmanager数据格式 ##

Alertmanager会以下列JSON格式的数据通过HTTP POST请求发送到端点：

{
      "version": "4",
      "groupKey": <string>,    // key identifying the group of alerts (e.g. to deduplicate)
      "status": "<resolved|firing>",
      "receiver": <string>,
      "groupLabels": <object>,
      "commonLabels": <object>,
      "commonAnnotations": <object>,
      "externalURL": <string>,  // backlink to the Alertmanager.
      "alerts": [
        {
          "labels": <object>,
          "annotations": <object>,
          "startsAt": "<rfc3339>",
          "endsAt": "<rfc3339>"
        },
        ...
      ]
    }

这是测试报警数据的示例：

b'{
    "receiver":"webhook",
    "status":"firing",
    "alerts":[{
        "status":"firing",
        "labels":{
            "alertname":"DeadMansSwitch",
            "prometheus":"monitoring/p-prometheus",
            "severity":"none"
    
        },
        "annotations":{
            "message":"This is a DeadMansSwitch meant to ensure that the entire alerting pipeline is functional."
    
        },
        "startsAt":"2019-03-08T10:02:28.680317737Z",
        "endsAt":"0001-01-01T00:00:00Z",
        "generatorURL":"http://prom.cnlinux.club/graph?g0.expr=vector%281%29\\u0026g0.tab=1"
    
    }],
    "groupLabels":{},
    "commonLabels":{
        "alertname":"DeadMansSwitch",
        "prometheus":"monitoring/p-prometheus",
        "severity":"none"
    
    },
    "commonAnnotations":{
    "message":"This is a DeadMansSwitch meant to ensure that the entire alerting pipeline is functional."
    
    },
    "externalURL":"http://alert.cnlinux.club","version":"4",
    "groupKey":"{}/{alertname=\\"DeadMansSwitch\\"}:{}"}\n'

钉钉对数据的格式是有要求的（具体要求在上面钉钉官方文档），所以需要将Alertmanager传过来的数据进行格式转化。

以下我们用自己写的python脚本来转换。

**脚本说明：**

*  alertmanager传过来的数据中，重要的是labels\{\}的数据，但是里面数据太多，很多信息在报警的信息中是不需要的，所以在脚本中添加了一个EXCLUDE\_LIST列表，用于排除不需要的数据。

[root@node-01 prometheus]# cat app.py
    #!/usr/bin/env python
    import io, sys
    
    sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='utf-8')
    sys.stderr = io.TextIOWrapper(sys.stderr.detach(), encoding='utf-8')
    
    from flask import Flask, Response
    from flask import request
    import requests
    import logging
    import json
    import locale
    #locale.setlocale(locale.LC_ALL,"en_US.UTF-8")
    
    app = Flask(__name__)
    
    console = logging.StreamHandler()
    fmt = '%(asctime)s - %(filename)s:%(lineno)s - %(name)s - %(message)s'
    formatter = logging.Formatter(fmt)
    console.setFormatter(formatter)
    log = logging.getLogger("flask_webhook_dingtalk")
    log.addHandler(console)
    log.setLevel(logging.DEBUG)
    
    EXCLUDE_LIST = ['prometheus', 'endpoint']
    
    @app.route('/')
    def index():
        return 'Webhook Dingtalk by Billy https://blog.51cto.com/billy98'
    
    @app.route('/dingtalk/send/',methods=['POST'])
    
    def hander_session():
    
        profile_url = sys.argv[1]
        post_data = request.get_data()
        post_data = json.loads(post_data.decode("utf-8"))['alerts']
        post_data = post_data[0]
        messa_list = []
        messa_list.append('### 报警类型: %s' % post_data['status'].upper())
        messa_list.append('**startsAt:** %s' % post_data['startsAt'])
        for i in post_data['labels'].keys():
            if i in EXCLUDE_LIST:
                continue
            else:
                messa_list.append("**%s:** %s" % (i, post_data['labels'][i]))
        messa_list.append('**Describe:** %s' % post_data['annotations']['message'])
    
        messa = (' \\n\\n > '.join(messa_list))
        status = alert_data(messa, post_data['labels']['alertname'], profile_url )
        log.info(status)
        return status
    
    def alert_data(data,title,profile_url):
        headers = {'Content-Type':'application/json'}
        send_data = '{"msgtype": "markdown","markdown": {"title": \"%s\" ,"text": \"%s\" }}' %(title,data)  # type: str
        send_data = send_data.encode('utf-8')
        reps = requests.post(url=profile_url, data=send_data, headers=headers)
        return reps.text
    
    if __name__ == '__main__':
        app.debug = False
        app.run(host='0.0.0.0', port='8080')

## 五、制作Docker镜像 ##

将上面的python脚本做成镜像，然后把他们以服务的形式运行在k8s集群中，保证高可用。

大家也可以用我已经制作成功的镜像：`docker pull billy98/webhook-dingtalk:latest`,直接pull即可。

[root@node-01 prometheus]# cat Dockerfile
    
    FROM centos:7 as build
    MAINTAINER billy98 5884625@qq.com
    
    RUN curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo && yum install -y python36 python36-pip && pip3.6 install flask requests werkzeug
    ADD app.py /usr/local/alert-dingtalk.py
    
    FROM gcr.io/distroless/python3
    COPY --from=build /usr/local/alert-dingtalk.py /usr/local/alert-dingtalk.py
    COPY --from=build usr/local/lib64/python3.6/site-packages usr/local/lib64/python3.6/site-packages
    COPY --from=build usr/local/lib/python3.6/site-packages usr/local/lib/python3.6/site-packages
    ENV PYTHONPATH=usr/local/lib/python3.6/site-packages:usr/local/lib64/python3.6/site-packages
    EXPOSE 8080
    ENTRYPOINT ["python","/usr/local/alert-dingtalk.py"]

[root@node-01 prometheus]# docker build -t billy98/webhook-dingtalk:latest .

我这样build出来的镜像只有50多M，具体的使用方法参考：

distroless：[https://github.com/GoogleContainerTools/distroless][https_github.com_GoogleContainerTools_distroless]

## 六、部署webhook-dingtalk ##

[root@node-01 prometheus]# cat webhook-dingtalk.yaml 
    apiVersion: apps/v1beta2
    kind: Deployment
    metadata:
      labels:
        app: webhook-dingtalk
      name: webhook-dingtalk
      namespace: monitoring
      #需要和alertmanager在同一个namespace
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: webhook-dingtalk
      template:
        metadata:
          labels:
            app: webhook-dingtalk
        spec:
          containers:
          - image: billy98/webhook-dingtalk:latest
            name: webhook-dingtalk
            args:
            - "https://oapi.dingtalk.com/robot/send?access_token=xxxxxx"
            #上面创建的钉钉机器人hook
            ports:
            - containerPort: 8080
              protocol: TCP
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
              limits:
                cpu: 500m
                memory: 500Mi
            livenessProbe:
              failureThreshold: 3
              initialDelaySeconds: 30
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
              tcpSocket:
                port: 8080
            readinessProbe:
              failureThreshold: 3
              initialDelaySeconds: 30
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
              httpGet:
                port: 8080
                path: /
          imagePullSecrets:
            - name: IfNotPresent
    ---
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: webhook-dingtalk
      name: webhook-dingtalk
      namespace: monitoring
      #需要和alertmanager在同一个namespace
    spec:
      ports:
      - name: http
        port: 80
        protocol: TCP
        targetPort: 8080
      selector:
        app: webhook-dingtalk
      type: ClusterIP

钉钉中报警信息如下：

![基于k8s Prometheus+Grafana+Altermanager钉钉报警][k8s Prometheus_Grafana_Altermanager]

![基于k8s Prometheus+Grafana+Altermanager钉钉报警][k8s Prometheus_Grafana_Altermanager 1]

报警恢复的消息  
![基于k8s Prometheus+Grafana+Altermanager钉钉报警][k8s Prometheus_Grafana_Altermanager 2]

至此所有的操作已完成。

**如有问题欢迎在下面留言交流。希望大家多多关注和点赞，谢谢！**

转载于:https://blog.51cto.com/billy98/2362369

[1. _Prometheus Operator_kubetnetes]: https://blog.51cto.com/billy98/2355587
[2. _Prometheus Operator]: https://blog.51cto.com/billy98/2357919
[Link 1]: https://open-doc.dingtalk.com/docs/doc.htm?treeId=257&articleId=105735&docType=1
[https_oapi.dingtalk.com_robot_send_access_token_xxxxxx]: https://oapi.dingtalk.com/robot/send?access_token=xxxxxx
[https_github.com_prometheus_docs_blob_db2a09a8a7e193d6e474f37055908a6d432b88b5_content_docs_alerting_configuration.md_webhook_config]: https://github.com/prometheus/docs/blob/db2a09a8a7e193d6e474f37055908a6d432b88b5/content/docs/alerting/configuration.md#webhook_config
[config]: https://s1.51cto.com/images/blog/201903/13/1df1196a6d9181c62330081f2f1a0245.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[https_github.com_GoogleContainerTools_distroless]: https://github.com/GoogleContainerTools/distroless
[k8s Prometheus_Grafana_Altermanager]: https://s1.51cto.com/images/blog/201903/13/364bb47e6b14c8ec63c0c68bb02c0f03.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[k8s Prometheus_Grafana_Altermanager 1]: https://s1.51cto.com/images/blog/201903/13/810450833d568a8dc9727218b4604378.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=
[k8s Prometheus_Grafana_Altermanager 2]: https://s1.51cto.com/images/blog/201903/13/9dcebac6d6f4ed993c197863e0900c4c.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=