Kubernetes安装kube-prometheus及配置持久化数据

川长思鸟来 2024-04-03 10:21 50阅读 0赞

![image.png][]

![image.png][image.png 1]

[参考信息][Link 1]

### 二、准备清单文件 ###

*  从官方的地址获取最新的main分支，或者直接打包下载main

git clone https://github.com/prometheus-operator/kube-prometheus.git
    git checkout main

*  或者直接下载打包好的包

wget https://github.com/prometheus-operator/kube-prometheus/archive/refs/tags/main.tar.gz
    tar -xvf main.tar.gz
    mv kube-prometheus-main kube-prometheus
    默认下载下来的文件较多，建议把文件进行归类处理，将相关yaml文件移动到对应目录下

*  新建相关目录

cd kube-prometheus/manifests
    mkdir -p adapter  alertmanager  blackbox  grafana  kube-control-plane  kube-state-metrics  node-exporter  prometheus serviceMonitor

#### 2.1、修改yaml,增加持久化存储 ####

> 修改Prometheus部署文件

`vim manifests/prometheus/prometheus-prometheus.yaml`

apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
      labels:
        app.kubernetes.io/component: prometheus
        app.kubernetes.io/instance: k8s
        app.kubernetes.io/name: prometheus
        app.kubernetes.io/part-of: kube-prometheus
        app.kubernetes.io/version: 2.32.1
      name: k8s
      namespace: monitoring
    spec:
      alerting:
        alertmanagers:
        - apiVersion: v2
          name: alertmanager-main
          namespace: monitoring
          port: web
      enableFeatures: []
      externalLabels: {}
      image: quay.io/prometheus/prometheus:v2.32.1
      nodeSelector:
        kubernetes.io/os: linux
      podMetadata:
        labels:
          app.kubernetes.io/component: prometheus
          app.kubernetes.io/instance: k8s
          app.kubernetes.io/name: prometheus
          app.kubernetes.io/part-of: kube-prometheus
          app.kubernetes.io/version: 2.32.1
      podMonitorNamespaceSelector: {}
      podMonitorSelector: {}
      probeNamespaceSelector: {}
      probeSelector: {}
      replicas: 2
      resources:
        requests:
          memory: 400Mi
      ruleNamespaceSelector: {}
      ruleSelector: {}
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 1000
      serviceAccountName: prometheus-k8s
      serviceMonitorNamespaceSelector: {}
      serviceMonitorSelector: {}
      version: 2.32.1
    
    # 新增持久化存储,yaml 末尾添加
      retention: 3d
      storage:
        volumeClaimTemplate:
          spec:
            storageClassName: managed-nfs-storage
            resources:
              requests:
                storage: 50Gi

![image.png][image.png 2]

> 修改Grafana部署文件

`vim manifests/grafana/grafana-deployment.yaml`

volumes:
    #      - emptyDir: {}           # 注释此两行，新增下三行
    #        name: grafana-storage
          - name: grafana-storage
            persistentVolumeClaim:
              claimName: grafana-pvc
          - name: grafana-datasources
            secret:
              secretName: grafana-datasources

![image.png][image.png 3]

#### 2.2、准备 持久化配置(nfs-storage) ####

### 三、启动服务 ###

`[root@Master02 manifests]# kubectl apply -f setup/`

> 使用apply时会报错，`The CustomResourceDefinition "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes`

最好使用`kubectl create -f setup/`

![image.png][image.png 4]

#### 创建PVC ####

`kubectl create -f grafana/grafana-pvc.yaml`![image.png][image.png 5]

#### 启动服务 ####

> 先把pvc的文件拿出来

`[root@Master02 manifests]# mv grafana/grafana-pvc.yaml ./`![image.png][image.png 6]

创建文件`[root@Master02 manifests]# kubectl create -f adapter/ -f alertmanager/ -f blackbox/ -f grafana/ -f kube-state-metrics/ -f node-exporter/ -f prometheus/ -f kube-control-plane/ -f serviceMonitor/`![image.png][image.png 7]

创建完成后有一个报错。![image.png][image.png 8]

查看日志信息

`err="opening storage failed: lock DB directory: resource temporarily unavailable"`

![image.png][image.png 9]

前往NFS共享服器目录删除`lock`文件

![image.png][image.png 10]

然后回到Master 删除报错的Pod后再次查看启动正常

![image.png][image.png 11]

##### 启动完成如下图 #####

![image.png][image.png 12]![image.png][image.png 13]

##### 配置Ingress-Nginx #####

> 持久化存储日志

---
    apiVersion: v1
    kind: Namespace
    metadata:
      name: ingress-nginx
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
    
    ---
    # 添加 ingress的 日志持久化 pvc
    # add ingress logs volume-pvc
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: ingress-logs-pvc
      namespace: ingress-nginx
    spec:
      # 指向我们创建的 nfs-storage
      storageClassName: prometheus-nfs
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
    ---
    ...
    ---
    # Source: ingress-nginx/templates/controller-configmap.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      labels:
        helm.sh/chart: ingress-nginx-3.30.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.46.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: controller
      name: ingress-nginx-controller
      namespace: ingress-nginx
    data:
      # 设置configmap  定义日志输出目录
      access-log-path: "/var/log/nginx/access.log"
      error-log-path: "/var/log/nginx/error.log"
    
    ...
              args:
                - /nginx-ingress-controller
                - --election-id=ingress-controller-leader
                - --ingress-class=nginx
                - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
                - --validating-webhook=:8443
                - --validating-webhook-certificate=/usr/local/certificates/cert
                - --validating-webhook-key=/usr/local/certificates/key
                - --v=2
                # 添加下面2两行，日志输出路径
                - --log_dir=/var/log/nginx/
                - --logtostderr=false
    ...
              volumeMounts:
                - name: webhook-cert
                  mountPath: /usr/local/certificates/
                  readOnly: true
                # 添加挂载目录
                - name: ingress-logs
                  mountPath: /var/log/nginx/
              resources:
                requests:
                  cpu: 100m
                  memory: 90Mi
          nodeSelector:
            kubernetes.io/os: linux
          serviceAccountName: ingress-nginx
          terminationGracePeriodSeconds: 300
          volumes:
            - name: webhook-cert
              secret:
                secretName: ingress-nginx-admission
            #持久化日志
            - name: ingress-logs
              persistentVolumeClaim:
                claimName: ingress-logs-pvc
    ...

> 编写ingress 代理规则

---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: prom-ingress
      namespace: monitoring
      annotations:
        kubernetes.io/ingress.class: "nginx"
        prometheus.io/http_probe: "true"
    spec:
      rules:
      - host: alert.nercoa.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: alertmanager-main
                port:
                  number: 9093
      - host: grafana.nercoa.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 3000
      - host: prom.nercoa.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prometheus-k8s
                port:
                  number: 9090

但是在执行时报错：`Error from server (InternalError): error when creating "prom-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": x509: certificate signed by unknown authority`

![image.png][image.png 14]

删除指定资源后再创建正常：`kubectl delete -A validatingwebhookconfigurations.admissionregistration.k8s.io ingress-nginx-admission`

![image.png][image.png 15]

#### 监测WindowsNode ####

安装提供的Exporter组件即可。[下载地址][Link 2]

![image.png][image.png 16]

安装完成后需要在K8S环境中进行配置Secrets,[参考链接][Link 3]

![image.png][image.png 17]

[参考示例][Link 4]

--------------------

`find . -name *-serviceMon* -exec mv {} ./serviceMonitor/ \;`

![image.png][image.png 18]

--------------------

### 增加Zabbix数据源 ###

[https://grafana.com/grafana/plugins/alexanderzobnin-zabbix-app/?tab=installation][https_grafana.com_grafana_plugins_alexanderzobnin-zabbix-app_tab_installation][https://grafana.com/api/plugins/alexanderzobnin-zabbix-app/versions/4.2.4/download][https_grafana.com_api_plugins_alexanderzobnin-zabbix-app_versions_4.2.4_download]

`[root@Master02 grafana-zabbix-plugins]# kubectl cp alexanderzobnin-zabbix-app-4.2.4.zip -n monitoring grafana-59d5b5c887-s2jcn:/var/lib/grafana/plugins/`

> 最好是将这个Grafana的插件目录给挂载出来，这样新增插件时直接解压拷贝过去就好了。

[root@Master02 grafana]# kubectl exec -it -n monitoring grafana-59d5b5c887-s2jcn sh
    kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
    /usr/share/grafana $ pwd
    /usr/share/grafana
    /usr/share/grafana $ cd /var/lib/grafana/plugins/
    /var/lib/grafana/plugins $ ls
    alexanderzobnin-zabbix-app-4.2.4.zip
    /var/lib/grafana/plugins $ unzip alexanderzobnin-zabbix-app-4.2.4.zip 
    /var/lib/grafana/plugins $ rm -rf alexanderzobnin-zabbix-app-4.2.4.zip

![image.png][image.png 19]

#### 新增Granfana不支持的数据源 ####

[https://grafana.com/grafana/plugins/grafana-simple-json-datasource/?tab=installation][https_grafana.com_grafana_plugins_grafana-simple-json-datasource_tab_installation]

![image.png][image.png 20]

[image.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/ce4d6ae955054ba28d45b0f06df696eb.png
[image.png 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/09c3ced7988b4f25a35e95dad491e526.png
[Link 1]: https://www.icode9.com/content-4-1150007.html
[image.png 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/504fa0a78a91406b8ee176fa8294626e.png
[image.png 3]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/3f2eb339c05c4fc1938fb43e5784fd25.png
[image.png 4]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/3ff12e88fd9c435098e402a4b39e4083.png
[image.png 5]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/6433c3d862214552bd267f09b5ed2662.png
[image.png 6]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/a1cc9e18a01342e58f05155e67a4c2ea.png
[image.png 7]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/bbdd72db9eb049e093d97b39f67c7705.png
[image.png 8]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/644595e2d6794a8bac1a03607be87a73.png
[image.png 9]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/e3e1fa9604e04f10a5889222ca66ee7c.png
[image.png 10]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/15699033000d45d28e3cab0e24655fad.png
[image.png 11]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8fcc419bdde54ed381e3d5cd5a652803.png
[image.png 12]: https://img-blog.csdnimg.cn/img_convert/d7cdfe7464544b4069c92782b3a5a38f.png
[image.png 13]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/dd71aa8e8bac4f179a140059470e44b5.png
[image.png 14]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/1b2182b258b046efa47f5736bf11d151.png
[image.png 15]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/6eef5b6f10b141868f8b0bbc56895773.png
[Link 2]: https://objects.githubusercontent.com/github-production-release-asset-2e65be/66566232/750593c1-2881-4baf-94c0-4af9f41312ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220122%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220122T012731Z&X-Amz-Expires=300&X-Amz-Signature=afd5bee4e90fdc842b2bc8a96d12efc670636a4a77626d692a7fddf2cf96e9bf&X-Amz-SignedHeaders=host&actor_id=9602672&key_id=0&repo_id=66566232&response-content-disposition=attachment%3B%20filename%3Dwindows_exporter-0.17.1-amd64.msi&response-content-type=application%2Foctet-stream
[image.png 16]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/f97e6c6140724f30a51eb905522a425c.png
[Link 3]: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/additional-scrape-config.md
[image.png 17]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/58b48773961c47a19303089b8df6d47f.png
[Link 4]: https://github.com/prometheus-operator/prometheus-operator/tree/main/example/additional-scrape-configs
[image.png 18]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2dd9b0f7047c46b6ad3f3a81f07d28b7.png
[https_grafana.com_grafana_plugins_alexanderzobnin-zabbix-app_tab_installation]: https://grafana.com/grafana/plugins/alexanderzobnin-zabbix-app/?tab=installation
[https_grafana.com_api_plugins_alexanderzobnin-zabbix-app_versions_4.2.4_download]: https://grafana.com/api/plugins/alexanderzobnin-zabbix-app/versions/4.2.4/download
[image.png 19]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/c43045b74f3643f59d8ee16f1f3c8dc3.png
[https_grafana.com_grafana_plugins_grafana-simple-json-datasource_tab_installation]: https://grafana.com/grafana/plugins/grafana-simple-json-datasource/?tab=installation
[image.png 20]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/72dc4fdcf7c34eaa94aaff1ada70e93d.png