k8s 搭建HPA和metric-server 问题排查

客官°小女子只卖身不卖艺 2022-12-24 11:51 120阅读 0赞

## 背景 ##

在k8s里面搭建HPA时。HPA需要metric-server进行采集数据，在过程中猜到两个坑，经过最后的折腾总算是解决了问题。所以记录一下方便其他人在遇到问题时可以快速解决。

### metric-server 修改日志级别查看问题 ###

在metric-server搭建完成后，可以通过 kubectl logs -f podName 查看metric-server输出的日志。但是有的时候查看日志看不出来问题的话。可以调整一下日志级别。这样可以输出更多有信息的日志。如下图  
–v 表示日志级别。数字越大，输出的日志越多。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MjEwNzY3_size_16_color_FFFFFF_t_70]

### HPA not able to find CPU 问题排查 ###

当HPA的event里面有如下输出日志的时候，这种问题是HPA的target的Deployment 管理的Pod没有设置CPU的request。设置就好了。

kubectl describe hpa hpa-demo
    Name:                                                  hpa-demo
    Namespace:                                             default
    Labels:                                                <none>
    Annotations:                                           <none>
    CreationTimestamp:                                     Tue, 19 Nov 2019 17:14:56 +0800
    Reference:                                             Deployment/hpa-demo
    Metrics:                                               ( current / target )
      resource cpu on pods  (as a percentage of request):  <unknown> / 10%
    Min replicas:                                          1
    Max replicas:                                          10
    Deployment pods:                                       1 current / 0 desired
    Conditions:
      Type           Status  Reason                   Message
      ----           ------  ------                   -------
      AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
      ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: missing request for cpu
    Events:
      Type     Reason                        Age                From                       Message
      ----     ------                        ----               ----                       -------
      Warning  FailedGetResourceMetric       14s (x4 over 60s)  horizontal-pod-autoscaler  missing request for cpu
      Warning  FailedComputeMetricsReplicas  14s (x4 over 60s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MjEwNzY3_size_16_color_FFFFFF_t_70 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MjEwNzY3_size_16_color_FFFFFF_t_70]: /images/20221120/dfbc2cfab4614fde8ef800eb3b5d72f3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI0MjEwNzY3_size_16_color_FFFFFF_t_70 1]: /images/20221120/74fa73d9267144869dd16a85a6eee0b6.png