python Django 实现自定义prometheus export

一时失言乱红尘 2023-07-01 15:00 5阅读 0赞

Prometheus 是一个 Metrics 监控系统，与 Kubernetes 同属 CNCF（Cloud Native Computing Foundation），它已经成为炙手可热的 Kubernetes 生态圈中的核心监控系统。

Prometheus 所有的Metrics 都是通过组件**export**主动pull获取到的。

Prometheus提供4种类型Metrics：`Counter`, `Gauge`, `Summary`和`Histogram`

Counter可以增长，并且在程序重启的时候会被重设为0，常被用于任务个数，总处理时间，错误个数等只增不减的指标。

Gauge与Counter类似，唯一不同的是Gauge数值可以减少，常被用于温度、利用率等指标。

Summary/Histogram概念比较复杂，对于我来说目前没有使用场景，暂无了解。本文主要使用Gauge实现获取es性能指标。

备注：本文主要使用Prometheus-operater部署形式

views.py

from django.shortcuts import render
    import prometheus_client
    from prometheus_client import Counter,Gauge
    from prometheus_client.core import CollectorRegistry
    from django.views.generic import View
    from django.http import HttpResponse
    import random
    from .util import esApi
    
    REGISTRY = CollectorRegistry(auto_describe=False)
    esStatus = Gauge("elasticsearch", "elasticsearch status is:", ["node", "class"],
                     registry=REGISTRY)  # 数值可大可小
    
    class ApiResponse(View):
        def get(self,request):
            es_obj = esApi.esApi()
            es_result = es_obj.es_status()
            for node, values in es_result.items():
                for key,value in values.items():
                    esStatus.labels(node,key).set(value)
            return HttpResponse(prometheus_client.generate_latest(REGISTRY),content_type="text/plain")

esApi.py

from elasticsearch import Elasticsearch
    
    class esApi():
        def __init__(self):
            self.es = Elasticsearch([{'host': '10.30.30.25', 'port': 9200}], timeout=3600)
            self.result = dict()
            self.Value = dict()
            self.esStatus = dict()
        def es_status(self):
           for key,value in self.es.nodes.stats()['nodes'].items():
               self.Value['es_heap_used_percent'] = value['jvm']['mem']['heap_used_percent']           #75的时候进行GC   节点总是大约75%，那你节点正在承受内存方面的压力，这是一个告警，预示着你不久就会出现慢GC heap使用率一直在85%
               self.Value['es_heap_used_in_bytes'] = value['jvm']['mem']['heap_used_in_bytes']
               self.Value['es_young_collection_count'] = value['jvm']['gc']['collectors']['young']['collection_count']
               self.Value['es_young_collection_millis'] = value['jvm']['gc']['collectors']['young']['collection_time_in_millis']
               self.Value['es_old_collection_count'] = value['jvm']['gc']['collectors']['old']['collection_count']
               self.Value['es_old_collection_millis'] = value['jvm']['gc']['collectors']['old']['collection_time_in_millis']
               self.Value['es_index_total'] = value['indices']['indexing']['index_total']
               self.Value['index_time_in_millis'] = value['indices']['indexing']['index_time_in_millis']
               self.result[key] = self.Value
           return self.result
        def cluster_status(self):
            esStatus = self.es.cluster.health()
            self.esStatus['es_status'] = esStatus['status']
            self.esStatus['es_unassigned_shards'] = esStatus['unassigned_shards']
            return self.esStatus

在使用prometheus-operator的情况下需要编写servicemonitor来配置target

apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      labels:
         app: autoexport
      name: autoexport
      namespace: monitoring
    spec:
      selector:
        matchLabels:
          app: autoexport
      endpoints:
      - port: metrics
        scheme: http
        interval: 30s
        path: '/api/autoMetric/Apimetric'

k8s-deployment

apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: autoexport
      namespace: monitoring
    spec:
      replicas: 1
      strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 0
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: autoexport
        spec:
          containers:
            - name: autoexport
              image: autoexport:v0.0.1
              ports:
                - containerPort: 8000
              livenessProbe:
                tcpSocket:
                  port: 8000
                initialDelaySeconds: 600
                periodSeconds: 10
                timeoutSeconds: 1
              readinessProbe:
                tcpSocket:
                  port: 8000
                initialDelaySeconds: 10
                periodSeconds: 5
                timeoutSeconds: 1
              resources:
                limits:
                  memory: "1Gi"
                requests:
                  memory: "256Mi"
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: autoexport
      namespace: monitoring
      labels:
        app: autoexport
    spec:
      ports:
      - name: autometrics
        port: 8000
        targetPort: 8000
      selector:
        app: autoexport

部署完成之后 到web层查看效果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIyNTQzOTkx_size_16_color_FFFFFF_t_70][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIyNTQzOTkx_size_16_color_FFFFFF_t_70 1][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIyNTQzOTkx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200123140046944.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIyNTQzOTkx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIyNTQzOTkx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200123140111780.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzIyNTQzOTkx,size_16,color_FFFFFF,t_70