kubernetes启动Pod遇到CrashLoopBackOff的解决思路

小灰灰 2022-06-01 12:00 424阅读 0赞

#### 1.问题现状 ####

通过命令行工具kubectl 获取异常容器

[root@master ~]# kubectl get pods -n kube-system | grep -v Running
    NAME                                        READY     STATUS             RESTARTS   AGE
    prometheus-tim-3864503240-rwpq5             0/1       CrashLoopBackOff   2516       8d

#### 2.查找问题 ####

[root@master ~]# kubectl describe pod prometheus-tim-3864503240-rwpq5 -n kube-system 
    ...
    spec.containers{prometheus} Warning     BackOff     Back-off restarting failed container
      8d        4s      59160   kubelet, test-95                    Warning     FailedSync  Error syncing pod

然后通过查看异常pod的日志,就知道是什么问题了

[root@master-64 ~]# kubectl logs prometheus-tim-3864503240-rwpq5 -n kube-system 
    level=info ts=2018-01-29T03:11:57.888337607Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0, branch=HEAD, revision=0a74f98628a0463dddc90528220c94de5032d1a0)"
    level=info ts=2018-01-29T03:11:57.888422278Z caller=main.go:216 build_context="(go=go1.9.2, user=root@615b82cb36b6, date=20171108-07:11:59)"
    level=info ts=2018-01-29T03:11:57.888440579Z caller=main.go:217 host_details="(Linux 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 prometheus-tim-3864503240-rwpq5 (none))"
    level=info ts=2018-01-29T03:11:57.890277632Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
    level=info ts=2018-01-29T03:11:57.890325752Z caller=targetmanager.go:71 component="target manager" msg="Starting target manager..."
    level=info ts=2018-01-29T03:11:57.8908479Z caller=main.go:314 msg="Starting TSDB"
    level=info ts=2018-01-29T03:11:57.903340116Z caller=main.go:326 msg="TSDB started"
    level=info ts=2018-01-29T03:11:57.9034449Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yaml level=error ts=2018-01-29T03:11:57.903503914Z caller=main.go:356 msg="Error loading config" err="couldn't load configuration (--config.file=/etc/prometheus/prometheus.yaml): open /etc/prometheus/prometheus.yaml: no such file or directory"

查看相关的configmap ,存不存在,因为配置文件是通过configmap挂载的

[root@test config-volume]# pwd
    /var/lib/kubelet/pods/5d2b9835-fd9c-11e7-9d36-5254b24cbf5e/volumes/kubernetes.io~configmap/config-volume
    [root@test config-volume]# 
    [root@test config-volume]# 
    [root@test config-volume]# 
    [root@test config-volume]# ls -l
    total 0

果然不存在,所以程序是起不来的

#### 3.解决问题 ####

直接删除掉异常的pod就解决了

kubectl delete pod prometheus-tim-3864503240-rwpq5 -n kube-system

END