用Prometheus和Grafana监控Kubenetes集群

傷城~ 2022-12-19 04:00 274阅读 0赞

### 文章目录 ###

*  用Prometheus和Grafana监控Kubenetes集群
 *   *  前言
     *  官方文档
     *  部署kube-prometheus
     *   *  安装Prometheus组件
         *  安装Grafana和其他组件
     *  Kubernetes集群监控
     *  其他内容
     *   *  （可选）通过Nginx反向代理Grafana
     *  参考文档

# 用Prometheus和Grafana监控Kubenetes集群 #

## 前言 ##

本文描述了使用[kube-prometheus][] 来集成Promeheus和Grafana监控Kubernetes集群。

环境说明：

*  Kubernetes 集群 v.1.19.3
 *  kube-prometheus v0.6

## 官方文档 ##

**Kubernetes:**

*  [资源监控工具][Link 1]
 *  [Tools for Monitoring Resources][]
 *  [Extend the Kubernetes API with CustomResourceDefinitions][]

**Prometheus:**

*  [https://prometheus.io/][https_prometheus.io]
 *  [https://github.com/prometheus-operator/kube-prometheus][kube-prometheus]

**Grafana:**

*  [https://grafana.com/docs/grafana/latest/getting-started/getting-started/][https_grafana.com_docs_grafana_latest_getting-started_getting-started]

## 部署kube-prometheus ##

当前kube-prometheus的最新版本为v0.6，官方建议直接用Master分支。

> Note: This release is more for pinning to a specific commit to preserve compatibility with kubernetes versions, master is always stable, and we encourage using master.

克隆kube-prometheus：

git clone https://github.com/coreos/kube-prometheus.git

kube-prometheus v0.6需要用到的镜像：

# quay.io images
    quay.io/prometheus-operator/prometheus-operator:v0.43.0
    quay.io/brancz/kube-rbac-proxy:v0.6.0
    quay.io/prometheus/alertmanager:v0.21.0
    quay.io/coreos/kube-state-metrics:v1.9.7
    quay.io/prometheus/node-exporter:v1.0.1
    quay.io/prometheus/prometheus:v2.20.0
    
    # Docker Hub images
    grafana/grafana:7.1.0
    directxman12/k8s-prometheus-adapter:v0.8.0

> 如果集群内不能连接外网，可以提前通过代理和阿里云镜像加速器拉取好这些镜像。

### 安装Prometheus组件 ###

安装Promeheus组件：

cd kube-prometheus
    kubectl create -f manifests/setup

创建了以下资源：

# namespace
    namespace/monitoring created
    
    
    # CRD(CustomResourceDefinition)
    customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
    customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
    
    # RBAC
    clusterrole.rbac.authorization.k8s.io/prometheus-operator created
    clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
    
    # deloyment
    deployment.apps/prometheus-operator created
    
    # service
    service/prometheus-operator created
    
    # serviceaccount
    serviceaccount/prometheus-operator created

验证：

kubectl get namespace monitoring
    
    # crd is CustomResurceDefinition
    kubectl get crd | grep monitoring
    kubectl get clusterrole.rbac.authorization.k8s.io | grep prometheus
    kubectl get clusterrolebinding.rbac.authorization.k8s.io | grep prometheus
    
    kubectl get deployment -n monitoring
    kubectl get service -n monitoring
    kubectl get serviceaccount -n monitoring

验证pod：

kubectl get pods -n monitoring

说明：

*  对Pod状态不是Running或没有全部Reday，通过`kubectl describe pod`查看详细错误信息。
 *  对`ImagePullBackOff`的问题，可通过代理和阿里云镜像加速器来拉取。
 *  如果集群节点CPU资源不足，也会导致Pod一直处在Pending状态。

### 安装Grafana和其他组件 ###

安装Grafana和其他组件：

kubectl apply -f manifests/

创建了以下资源：

alertmanager.monitoring.coreos.com/main created
    secret/alertmanager-main created
    service/alertmanager-main created
    serviceaccount/alertmanager-main created
    servicemonitor.monitoring.coreos.com/alertmanager created
    secret/grafana-datasources created
    configmap/grafana-dashboard-apiserver created
    configmap/grafana-dashboard-cluster-total created
    configmap/grafana-dashboard-controller-manager created
    configmap/grafana-dashboard-k8s-resources-cluster created
    configmap/grafana-dashboard-k8s-resources-namespace created
    configmap/grafana-dashboard-k8s-resources-node created
    configmap/grafana-dashboard-k8s-resources-pod created
    configmap/grafana-dashboard-k8s-resources-workload created
    configmap/grafana-dashboard-k8s-resources-workloads-namespace created
    configmap/grafana-dashboard-kubelet created
    configmap/grafana-dashboard-namespace-by-pod created
    configmap/grafana-dashboard-namespace-by-workload created
    configmap/grafana-dashboard-node-cluster-rsrc-use created
    configmap/grafana-dashboard-node-rsrc-use created
    configmap/grafana-dashboard-nodes created
    configmap/grafana-dashboard-persistentvolumesusage created
    configmap/grafana-dashboard-pod-total created
    configmap/grafana-dashboard-prometheus-remote-write created
    configmap/grafana-dashboard-prometheus created
    configmap/grafana-dashboard-proxy created
    configmap/grafana-dashboard-scheduler created
    configmap/grafana-dashboard-statefulset created
    configmap/grafana-dashboard-workload-total created
    configmap/grafana-dashboards created
    deployment.apps/grafana created
    service/grafana created
    serviceaccount/grafana created
    servicemonitor.monitoring.coreos.com/grafana created
    clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
    clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
    deployment.apps/kube-state-metrics created
    service/kube-state-metrics created
    serviceaccount/kube-state-metrics created
    servicemonitor.monitoring.coreos.com/kube-state-metrics created
    clusterrole.rbac.authorization.k8s.io/node-exporter created
    clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
    daemonset.apps/node-exporter created
    service/node-exporter created
    serviceaccount/node-exporter created
    servicemonitor.monitoring.coreos.com/node-exporter created
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
    clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
    clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
    clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
    clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
    configmap/adapter-config created
    deployment.apps/prometheus-adapter created
    rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
    service/prometheus-adapter created
    serviceaccount/prometheus-adapter created
    servicemonitor.monitoring.coreos.com/prometheus-adapter created
    clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
    clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
    servicemonitor.monitoring.coreos.com/prometheus-operator created
    prometheus.monitoring.coreos.com/k8s created
    rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
    rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
    rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
    rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
    role.rbac.authorization.k8s.io/prometheus-k8s-config created
    role.rbac.authorization.k8s.io/prometheus-k8s created
    role.rbac.authorization.k8s.io/prometheus-k8s created
    role.rbac.authorization.k8s.io/prometheus-k8s created
    prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
    service/prometheus-k8s created
    serviceaccount/prometheus-k8s created
    servicemonitor.monitoring.coreos.com/prometheus created
    servicemonitor.monitoring.coreos.com/kube-apiserver created
    servicemonitor.monitoring.coreos.com/coredns created
    servicemonitor.monitoring.coreos.com/kube-controller-manager created
    servicemonitor.monitoring.coreos.com/kube-scheduler created
    servicemonitor.monitoring.coreos.com/kubelet created

验证：

kubectl get ds -n monitoring
    kubectl get deployment -n monitoring
    kubectl get svc -n monitoring
    kubectl get ss -n monitoring
    kubectl get pods -n monitoring

其中最重要的两个服务：

prometheus-k8s          ClusterIP   10.96.6.247     <none>        9090/TCP                     100m
    grafana                 NodePort    10.107.3.85     <none>        3000:31333/TCP               100m

在kube-prometheus v0.6中，Grafana的服务类型为NodePort，端口为31333。

可以通过`http://<node_port>:31333`方式访问Grafana，缺省密码为`admin/admin`，首次登录时系统将提示修改密码。

## Kubernetes集群监控 ##

安装好的Grafana中已内置很多有用的Kubernetes监控dashboard。

常用的Kubernetes监控 dashboard包括：

*  集群监控：Kubernetes / Compute Resource / Cluster
 *  按节点监控Pod：Kubernetes / Compute Resources / Node (Pods)
 *  按Namespace监控Pods：Kubernetes / Compute Resources / Namespace (Pods)
 *  工作负载监控：Kubernetes / Compute Resources / Namespace (Workloads)
 *  网络监控：Kubernetes / Networking / Cluster

## 其他内容 ##

### （可选）通过Nginx反向代理Grafana ###

如果不想直接通过NodePort方式访问Grafana，可以通过Nginx反向代理Grafana。

以[http://grafana.xdevops.cn/][http_grafana.xdevops.cn]为例：

在`/etc/nginx/conf.d`下创建`grafana.xdevops.cn.conf`，内容为：

upstream grafana_server { 
        server 127.0.0.1:31333;
    }
    
    server { 
    
        listen 80;
        listen [::]:80;
    
        server_name grafana.xdevops.cn;
    
        location / { 
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Host $http_host;
    
            proxy_pass http://grafana_server;
            proxy_redirect off;
            proxy_http_version 1.1;
            proxy_buffering off;
    
            chunked_transfer_encoding off;
        }
    }

说明：

*  `server 127.0.0.1:31333;` 为Grafana服务的NodePort IP和端口。

验证Nginx配置，并让Nginx配置生效：

nginx -t
    nginx -s reload

在域名供应商的控制台中添加一条`grafana.xdevops.cn`的DNS 解析A记录。

然后就可以通过[http://grafana.xdevops.cn/][http_grafana.xdevops.cn]来访问Grafana了。

## 参考文档 ##

*  [kubernetes系列教程(二十)prometheus提供完备监控系统][kubernetes_prometheus]

[kube-prometheus]: https://github.com/prometheus-operator/kube-prometheus
[Link 1]: https://kubernetes.io/zh/docs/tasks/debug-application-cluster/resource-usage-monitoring/
[Tools for Monitoring Resources]: https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/
[Extend the Kubernetes API with CustomResourceDefinitions]: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
[https_prometheus.io]: https://prometheus.io/
[https_grafana.com_docs_grafana_latest_getting-started_getting-started]: https://grafana.com/docs/grafana/latest/getting-started/getting-started/
[http_grafana.xdevops.cn]: http://grafana.xdevops.cn/
[kubernetes_prometheus]: https://cloud.tencent.com/developer/article/1579651