kubernetes node DiskPressure 压力

小灰灰 2022-05-09 03:20 337阅读 0赞

问题描述：  
k8s slave节点没有启动任何容器

[root@dev-slave-107 log]# docker ps 
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    [root@dev-slave-107 log]# docker ps 
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

查看磁盘使用量

[root@dev-slave-107 ~]# df -h 
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/vda1        50G   42G  4.7G  91% /
    devtmpfs        7.8G     0  7.8G   0% /dev
    tmpfs           7.8G     0  7.8G   0% /dev/shm
    tmpfs           7.8G  637M  7.2G   9% /run
    tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
    tmpfs           770M     0  770M   0% /run/user/0

磁盘使用量达到了 91%

kubelet 上报了DiskPressure 状态为true

- lastHeartbeatTime: 2018-09-30T01:41:32Z
        lastTransitionTime: 2018-09-30T01:34:01Z
        message: kubelet has disk pressure
        reason: KubeletHasDiskPressure
        status: "True"
        type: DiskPressure

查看哪个目录使用磁盘的大小`du -sh ./*`

进到某个目录使用

[root@dev-slave-107 log]# du -sh .
    38G	.

发现日志message的日志多

[root@dev-slave-107 log]# ll -h 
    total 38G
    -rw-r--r-- 1 root root 9.4K Sep 29 17:46 boot.log
    -rw------- 1 root utmp    0 Sep  1 03:26 btmp
    -rw------- 1 root utmp    0 Aug  1 03:50 btmp-20180901
    drwxr-xr-x 2 root root  12K Sep 29 17:17 containers
    -rw------- 1 root root  17M Sep 30 09:51 cron
    -rw------- 1 root root  74K Sep 30 03:06 dhclient
    -rw-r--r-- 1 root root  39K Sep 29 17:46 dmesg
    -rw-r--r-- 1 root root  39K Sep 29 15:42 dmesg.old
    drwxr-xr-x 2 root root 4.0K Sep 29 17:12 filebeat
    drwxr-xr-x 2 root root 4.0K Jun  7 21:11 filelog
    -rw-r--r-- 1 root root 560K Sep 29 17:46 firewall
    -rw------- 1 root root 183M Sep 30 09:51 maillog
    -rw------- 1 root root  37G Sep 30 09:51 messages
    -rw------- 1 root root 695K Sep 30 03:06 NetworkManager
    drwxr-xr-x 2 root root 4.0K Sep 29 17:17 pods
    drwxr-x--- 2 root root  20K Sep 29 03:55 qingcloud-flex-volume
    -rw------- 1 root root  53K Sep 30 09:33 secure
    -rw-r--r-- 1 root root  120 Sep 29 17:46 wpa_supplicant.log
    -rw-rw-r-- 1 root utmp  49K Sep 30 09:31 wtmp
    You have mail in /var/spool/mail/root

解决办法

[root@dev-slave-107 log]# echo "" >messages

查看磁盘使用量

[root@dev-slave-107 log]# df -h 
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/vda1        50G  5.0G   42G  11% /
    devtmpfs        7.8G     0  7.8G   0% /dev
    tmpfs           7.8G     0  7.8G   0% /dev/shm
    tmpfs           7.8G  637M  7.2G   9% /run
    tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
    tmpfs           770M     0  770M   0% /run/user/0

重启docker kubelet

systemctl restart docker kubelet

问题解决

root@dev-slave-107 log]# docker ps 
    CONTAINER ID        IMAGE                                                                                                     COMMAND                  CREATED             STATUS              PORTS               NAMES
    6283e10f3465        2ae65d218c78                                                                                              "/bin/bash /daemon.sh"   7 seconds ago       Up 6 seconds                            k8s_stateful-pod_stateful-new-pod-0_lijiaob_251a07dd-c3d6-11e8-98ff-52549da43ad9_0
    1008fba8085d        harbor.enncloud.cn/paas/pause-amd64:3.1                                                                   "/pause"                 8 seconds ago       Up 7 seconds                            k8s_POD_stateful-new-pod-0_lijiaob_251a07dd-c3d6-11e8-98ff-52549da43ad9_0
    925aef84f293        20282b8475a8                                                                                              "/install-cni.sh"        9 seconds ago       Up 7 seconds                            k8s_install-cni_calico-node-gd62v_kube-system_911a77c1-c453-11e8-98ff-52549da43ad9_0
    b318f9048e50        c75dc27cbfd7                                                                                              "/blockplugin --nodei"   10 seconds ago      Up 8 seconds                            k8s_csi-qingcloud_csi-qingcloud-node-2vp78_kube-system_90bfc29c-c453-11e8-98ff-52549da43ad9_0
    99c84abbeba3        0c4dc3ee8c07                                                                                              "/proxy --cluster-cid"   12 seconds ago      Up 9 seconds                            k8s_kube-proxy_kube-proxy-zzcgg_kube-system_90810274-c453-11e8-98ff-52549da43ad9_0