K8S集群监控—cAdvisor+Heapster+InfluxDB+Grafana

港控/mmm° 2021-12-11 05:37 527阅读 0赞

一、集群监控原理  
cAdvisor：容器数据收集。  
Heapster：集群监控数据收集，汇总所有节点监控数据。  
InfluxDB：时序数据库，存储监控数据。  
Grafana：可视化展示。  
二、搭建cAdvisor+Heapster+InfluxDB+Grafana  
1.从官网上拉取安装包  
获取v1.5.2heapster+influxdb+grafana安装yaml文件到 heapster release 页面下载最新版本的 heapster：  
wget https://github.com/kubernetes/heapster/archive/v1.5.2.zip  
unzip v1.5.2.zip  
解压后   
cd heapster-1.5.2/deploy/kube-config/influxdb/  
里面有3个配置文件，我们依次修改一下就可以使用了。  
具体代码如下：  
vi influxdb.yaml  
apiVersion: extensions/v1beta1  
kind: Deployment  
metadata:  
  name: monitoring-influxdb  
  namespace: kube-system  
spec:  
  replicas: 1  
  template:  
    metadata:  
      labels:  
        task: monitoring  
        k8s-app: influxdb  
    spec:  
      containers:  
      - name: influxdb  
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-influxdb-amd64:v1.1.1  
        volumeMounts:  
        - mountPath: /data  
          name: influxdb-storage  
      volumes:  
      - name: influxdb-storage  
        emptyDir: \{\}

\---  
apiVersion: v1  
kind: Service  
metadata:  
  labels:  
    task: monitoring  
    \# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)  
    \# If you are NOT using this as an addon, you should comment out this line.  
    kubernetes.io/cluster-service: 'true'  
    kubernetes.io/name: monitoring-influxdb  
  name: monitoring-influxdb  
  namespace: kube-system  
spec:  
  type: NodePort  
  ports:  
  - nodePort: 31001  
    port: 8086  
    targetPort: 8086  
  selector:  
    k8s-app: influxdb  
      
执行：kubectl create -f influxdb.yaml

vi heapster.yaml  
apiVersion: v1  
kind: ServiceAccount  
metadata:  
  name: heapster  
  namespace: kube-system

\---

kind: ClusterRoleBinding  
apiVersion: rbac.authorization.k8s.io/v1beta1  
metadata:  
  name: heapster  
subjects:  
  - kind: ServiceAccount  
    name: heapster  
    namespace: kube-system  
roleRef:  
  kind: ClusterRole  
  name: cluster-admin  
  apiGroup: rbac.authorization.k8s.io

\---

apiVersion: extensions/v1beta1  
kind: Deployment  
metadata:  
  name: heapster  
  namespace: kube-system  
spec:  
  replicas: 1  
  template:  
    metadata:  
      labels:  
        task: monitoring  
        k8s-app: heapster  
    spec:  
      serviceAccountName: heapster  
      containers:  
      - name: heapster  
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-amd64:v1.4.2  
        imagePullPolicy: IfNotPresent  
        command:  
        - /heapster  
        - --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true   
        - --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086

\---

apiVersion: v1  
kind: Service  
metadata:  
  labels:  
    task: monitoring  
    \# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)  
    \# If you are NOT using this as an addon, you should comment out this line.  
    kubernetes.io/cluster-service: 'true'  
    kubernetes.io/name: Heapster  
  name: heapster  
  namespace: kube-system  
spec:  
  ports:  
  - port: 80  
    targetPort: 8082  
  selector:  
    k8s-app: heapster  
执行： kubectl create -f heapster.yaml

vi grafana.yaml  
apiVersion: extensions/v1beta1  
kind: Deployment  
metadata:  
  name: monitoring-grafana  
  namespace: kube-system  
spec:  
  replicas: 1  
  template:  
    metadata:  
      labels:  
        task: monitoring  
        k8s-app: grafana  
    spec:  
      containers:  
      - name: grafana  
        image: registry.cn-hangzhou.aliyuncs.com/google-containers/heapster-grafana-amd64:v4.4.1  
        ports:  
        - containerPort: 3000  
          protocol: TCP  
        volumeMounts:  
        - mountPath: /etc/ssl/certs  
          name: ca-certificates  
          readOnly: true  
        - mountPath: /var  
          name: grafana-storage  
        env:  
        - name: INFLUXDB\_HOST  
          value: monitoring-influxdb  
        - name: GF\_SERVER\_HTTP\_PORT  
          value: "3000"  
          \# The following env variables are required to make Grafana accessible via  
          \# the kubernetes api-server proxy. On production clusters, we recommend  
          \# removing these env variables, setup auth for grafana, and expose the grafana  
          \# service using a LoadBalancer or a public IP.  
        - name: GF\_AUTH\_BASIC\_ENABLED  
          value: "false"  
        - name: GF\_AUTH\_ANONYMOUS\_ENABLED  
          value: "true"  
        - name: GF\_AUTH\_ANONYMOUS\_ORG\_ROLE  
          value: Admin  
        - name: GF\_SERVER\_ROOT\_URL  
          \# If you're only using the API Server proxy, set this value instead:  
          \# value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy  
          value: /  
      volumes:  
      - name: ca-certificates  
        hostPath:  
          path: /etc/ssl/certs  
      - name: grafana-storage  
        emptyDir: \{\}  
\---  
apiVersion: v1  
kind: Service  
metadata:  
  labels:  
    \# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)  
    \# If you are NOT using this as an addon, you should comment out this line.  
    kubernetes.io/cluster-service: 'true'  
    kubernetes.io/name: monitoring-grafana  
  name: monitoring-grafana  
  namespace: kube-system  
spec:  
  \# In a production setup, we recommend accessing Grafana through an external Loadbalancer  
  \# or through a public IP.  
  \# type: LoadBalancer  
  \# You could also use NodePort to expose the service at a randomly-generated port  
  \# type: NodePort  
  type: NodePort  
  ports:  
  - nodePort: 30108  
    port: 80  
    targetPort: 3000  
  selector:  
    k8s-app: grafana  
      
执行：kubectl create -f grafana.yaml

查看grafana 所在的节点：  
kubectl get pod -n kube-system -o wide|grep grafana  
经查看在node3上 ip为：172.18.0.25  
在浏览器中访问grafana地址 http://172.18.0.25:30108  
登录用户名:admin 密码：admin

报错以及解决方案：  
一、如遇到Error from server (AlreadyExists): error when creating "influxdb.yaml": deployments.extensions "monitoring-influxdb" already exists  
解决方案：  
kubectl delete -f influxdb.yaml  即删除yaml文件

二、如在Cluster里看不到节点信息，检查heapster的日志  
kubectl get pods -n kube-system  \#查找pod  
kubectl logs heapster-77fb88dbfc-k4tr9   \#查看日志  
报错信息如下：  
http://172.17.0.26:10255/stats/container/": Post http://172.17.0.26:10255/stats/container/: dial tcp 172.17.0.26:10255: getsockopt: connection refused  
E0710 06:04:05.031893       1 kubelet.go:231\] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.18.0.25:10255/stats/container/": Post http://172.18.0.25:10255/stats/container/: dial tcp 172.18.0.25:10255: getsockopt: connection refused  
E0710 06:04:05.037431       1 kubelet.go:231\] error while getting containers from Kubelet: failed to get all container stats from Kubelet URL "http://172.19.0.22:10255/stats/container/": Post http://172.19.0.22:10255/stats/container/: dial tcp 172.19.0.22:10255: getsockopt: connection refused

通过kubectl top 命令也获取不到结果：  
kubectl top pod  
输出：error: Metrics not available for pod default/nginx-64f497f8fd-4c6h2, age: 1148h17m14.701943925s  
kubectl top node  
输出：error: metrics not available yet

解决方法：  
在heapster.yaml 清单文件中进行如下修改  
\- --source=kubernetes:https://kubernetes.default?kubeletHttps=true&kubeletPort=10250&insecure=true   
\- --sink=influxdb:http://monitoring-influxdb.kube-system.svc.cluster.local:8086  
然后删除heapster重建  
kubectl delete -f heapster.yaml   
kubectl apply -f  heapster.yaml

heapster默认30秒检查一次，因此需要等上30s才会收集到数据  
再次登录到dashboard，可以单独内存、CPU信息