kubernetes基于k8s-prometheus-adapter实现 pod 的hpa(自动伸缩)

灰太狼 2022-05-13 11:20 446阅读 0赞

**系统**

[root@k8s-build ~]# uname -a 
    Linux k8s-build 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

**kubernetes的版本**

[root@master-47-35 custom-metrics-api]# kubectl version 
    Client Version: version.Info{ Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{ Major:"1", Minor:"11+", GitVersion:"v1.11.0-1+feebbe2b16f546", GitCommit:"feebbe2b16f546ac289f7afdae40f1721e0656fe", GitTreeState:"clean", BuildDate:"2018-08-23T02:06:55Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

**获取代码k8s-prometheus-adapter**

yum install git -y 
    cd /root/gopath/src/
    git clone https://github.com/DirectXMan12/k8s-prometheus-adapter.git

**切换分支到`v0.2.1`**

[root@k8s-build k8s-prometheus-adapter]# git checkout v0.2.1
    Note: checking out 'v0.2.1'.
    
    You are in 'detached HEAD' state. You can look around, make experimental
    changes and commit them, and you can discard any commits you make in this
    state without impacting any branches by performing another checkout.
    
    If you want to create a new branch to retain commits you create, you may
    do so (now or later) by using -b with the checkout command again. Example:
    
      git checkout -b new_branch_name
    
    HEAD 目前位于 3fc2dbd... Explicitly build on tags in Travis
    [root@k8s-build k8s-prometheus-adapter]# git status 
    # 头指针分离于 v0.2.1
    无文件要提交，干净的工作区

**构建k8s-prometheus-adapter镜像**

[root@k8s-build k8s-prometheus-adapter]# make docker-build
    cp deploy/Dockerfile /tmp/tmp.FbNVQfpHCg
    cd /tmp/tmp.FbNVQfpHCg && sed -i "s|BASEIMAGE|busybox|g" Dockerfile
    docker run -it -v /tmp/tmp.FbNVQfpHCg:/build -v /root/gopath/src/k8s-prometheus-adapter:/go/src/github.com/directxman12/k8s-prometheus-adapter -e GOARCH=amd64 golang:1.10 /bin/bash -c "\ CGO_ENABLED=0 go build -tags netgo -o /build/adapter github.com/directxman12/k8s-prometheus-adapter/cmd/adapter"
    Unable to find image 'golang:1.10' locally
    Trying to pull repository docker.io/library/golang ... 
    1.10: Pulling from docker.io/library/golang
    05d1a5232b46: Pull complete 
    5cee356eda6b: Pull complete 
    89d3385f0fd3: Pull complete 
    80ae6b477848: Pull complete 
    94ebfeaaddf3: Pull complete 
    e132030a369d: Pull complete 
    c67c5750c788: Pull complete 
    Digest: sha256:322a64a16553516761f8e71eb94136140031e6a7fcfe794e579de14a7da21ee7
    Status: Downloaded newer image for docker.io/golang:1.10
    docker build -t directxman12/k8s-prometheus-adapter-amd64:latest /tmp/tmp.FbNVQfpHCg
    Sending build context to Docker daemon 55.61 MB
    Step 1/4 : FROM busybox
    Trying to pull repository docker.io/library/busybox ... 
    latest: Pulling from docker.io/library/busybox
    8c5a7da1afbc: Already exists 
    Digest: sha256:032ddd66f10483436e8a252e69fdfd20d0164e9953585c10d378183a0924db34
    Status: Downloaded newer image for docker.io/busybox:latest
     ---> e1ddd7948a1c
    Step 2/4 : COPY adapter /
     ---> a39120e24ddb
    Removing intermediate container 6358600308cb
    Step 3/4 : USER 1001:1001
     ---> Running in 9755e39bed35
     ---> 718d846e413b
    Removing intermediate container 9755e39bed35
    Step 4/4 : ENTRYPOINT /adapter
     ---> Running in 68554cb44316
     ---> cb309c2eb2f4
    Removing intermediate container 68554cb44316
    Successfully built cb309c2eb2f4
    rm -rf /tmp/tmp.FbNVQfpHCg

**查看k8s-prometheus-adapter镜像**

[root@k8s-build k8s-prometheus-adapter]# docker images | grep k8s-prometheus-adapter-amd64 directxman12/k8s-prometheus-adapter-amd64 latest cb309c2eb2f4 About a minute ago 56.8 MB

**修改镜像tag,指定到私有的镜像仓库**

[root@k8s-build k8s-prometheus-adapter]# docker tag cb309c2eb2f4 harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1 [root@k8s-build k8s-prometheus-adapter]# docker push harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1 The push refers to a repository [harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64] d4821deae84a: Pushed f9d9e4e6e2f0: Pushed v0.2.1: digest: sha256:32ead17195963566630d9a8286773ddff92f9ccc126cf9a6e2cdcf686d163aef size: 739

**生成名字叫cm-adapter-serving-certs的secret,值为serving.crt和 serving.key，实际上就是证书**

证书生成请参考[requestheader-authentication][]

这里有生成证书的脚本,供参考，这个证书生成脚本需要gcc golang环境，安装gcc go

yum install go gcc -y

**脚本内容**

[root@k8s-build certs]# cat gencerts.sh 
    #!/usr/bin/env bash 
    # Detect if we are on mac or should use GNU base64 options
    case $(uname) in
            Darwin)
                b64_opts='-b=0'
                ;; 
            *)
                b64_opts='--wrap=0'
    esac
    
    go get -v -u github.com/cloudflare/cfssl/cmd/...
    export PATH=$GOPATH/bin:$PATH
    export PURPOSE=metrics
    openssl req -x509 -sha256 -new -nodes -days 365 -newkey rsa:2048 -keyout ${PURPOSE}-ca.key -out ${PURPOSE}-ca.crt -subj "/CN=ca"
    echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "${PURPOSE}-ca-config.json"
    
    export SERVICE_NAME=custom-metrics-apiserver
    export ALT_NAMES='"custom-metrics-apiserver.custom-metrics","custom-metrics-apiserver.custom-metrics.svc"'
    echo "{\"CN\":\"${SERVICE_NAME}\", \"hosts\": [${ALT_NAMES}], \"key\": {\"algo\": \"rsa\",\"size\": 2048}}" | \
            cfssl gencert -ca=metrics-ca.crt -ca-key=metrics-ca.key -config=metrics-ca-config.json - | cfssljson -bare apiserver
    
    cat <<-EOF > cm-adapter-serving-certs.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: cm-adapter-serving-certs
      namespace: custom-metrics
    data:
      serving.crt: $(base64 ${b64_opts} < apiserver.pem)
      serving.key: $(base64 ${b64_opts} < apiserver-key.pem)
    EOF

**生成证书**

[root@k8s-build certs]# ./gencerts.sh 
    github.com/cloudflare/cfssl (download)
    Generating a 2048 bit RSA private key
    .+++
    ......+++
    writing new private key to 'metrics-ca.key'
    -----
    2018/08/25 20:36:58 [INFO] generate received request
    2018/08/25 20:36:58 [INFO] received CSR
    2018/08/25 20:36:58 [INFO] generating key: rsa-2048
    2018/08/25 20:36:59 [INFO] encoded CSR
    2018/08/25 20:36:59 [INFO] signed certificate with serial number 573231977150840198182111747492918974864906787988

**查看脚本生成的内容**

[root@k8s-build certs]# ls -l 
    总用量 32
    -rw-r--r-- 1 root root 1066 8月 25 20:36 apiserver.csr -rw------- 1 root root 1679 8月 25 20:36 apiserver-key.pem -rw-r--r-- 1 root root 1277 8月 25 20:36 apiserver.pem -rw-r--r-- 1 root root 4053 8月 25 20:36 cm-adapter-serving-certs.yaml -rwxr-xr-x 1 root root 1198 8月 25 20:36 gencerts.sh -rw-r--r-- 1 root root 94 8月 25 20:36 metrics-ca-config.json -rw-r--r-- 1 root root 1074 8月 25 20:36 metrics-ca.crt -rw-r--r-- 1 root root 1704 8月 25 20:36 metrics-ca.key

**查看证书内容**

[root@master-47-35 custom-metrics]# cat cm-adapter-serving-certs.yaml
    apiVersion: v1
    data:
      serving.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHRENDQWdDZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFmTVIwd0d3WURWUVFEREJSc2IyTmgKYkdodmMzUkFNVFV4TXpjME5qZzBNREFlRncweE56RXlNakF3TlRFME1EQmFGdzB4T0RFeU1qQXdOVEUwTURCYQpNQjh4SFRBYkJnTlZCQU1NRkd4dlkyRnNhRzl6ZEVBeE5URXpOelEyT0RRd01JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW9mZE4zd1RqVldlZFo2YlBRN3ZrMTNPeUkwMEVwOGcvYzlTTCtneWMKQ3RHK3hYSkhrVlN4eW1zVlhsNnF4T2tWTWkrL2JqRFVqc1VsSnY5a2JjZkhHemYwYmxiQWdocVhvQ20ycjkvRwp0TVNlczlTL0NER1FNQUhPbkxsemdjVkxVQ1kyUVVXdS9HZDVwQU96dkJSUVBWczNBNUNZcDMzS0grTFVTTExOCnVNMjcvSkVnclUwczdBQ29ZaDk3KzQrSU1qb1pmbHRiK0JOQzRaU3BvemhQZ0EwQWhXRXl0SXkrRDdVWWdIR2cKNm8ybWJpTkV1QUNrQmk5TmFITm8zdGdieVN3OGQzYU9vd3JQQ3VlNEV0UTNIZGlNMHB5WUtuSUhsTFhyNjA1Sgpra25ENWs0OWFoaktQUFJhTlpjNlZpZlEvTzNiekVnNUwvVld6K2lneFJrSyt3SURBUUFCbzE4d1hUQU9CZ05WCkhROEJBZjhFQkFNQ0FxUXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBbEJnTlZIUkVFSGpBY2dnbHNiMk5oYkdodmMzU0NDV3h2WTJGc2FHOXpkSWNFZndBQUFUQU5CZ2txaGtpRwo5dzBCQVFzRkFBT0NBUUVBWERVV3ozVW0zVFg4VFJCVmQ1VVF5Z1ZhKzloMlRtVWJMU0JpdlBURDVRTWFBZUJzCkRYQU0vbTZrWXZIcFlTTi9MOWhraWd1QzhTazdlVFZqeWZabjlsYmNoSVV1NytoYys1UThJTDZVb0VaQ2cxVHQKNDNlYzBhZlc1YUU2bm5lejVhMWs3UDFhY0xhL1ZFUzVWM25wUmg2YUl4a0xVVHNMaUFsdGRGcURpeTF4aVlORgorL2pWMGFHRXhWWWJMbFVQMkIxaTlQODZneGFjNkJRTCtSQTNZZjFISWl5ZkliemMrbjV4Z2U1T0NIbC9XVjV1CkIzTHdZdkh1TGdpTDNZbyswN0gycWZ3MlNIUmZxdlJHdlpmS2tnWmcrTVQrZElYR3ZTTE0ybUhhdElxQnNXZEwKS1VFZnorREdYbzBIam1LTmhuWThRM0kwN1IxY29ta05KUFMxL3c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==  
      serving.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBb2ZkTjN3VGpWV2VkWjZiUFE3dmsxM095STAwRXA4Zy9jOVNMK2d5Y0N0Ryt4WEpICmtWU3h5bXNWWGw2cXhPa1ZNaSsvYmpEVWpzVWxKdjlrYmNmSEd6ZjBibGJBZ2hxWG9DbTJyOS9HdE1TZXM5Uy8KQ0RHUU1BSE9uTGx6Z2NWTFVDWTJRVVd1L0dkNXBBT3p2QlJRUFZzM0E1Q1lwMzNLSCtMVVNMTE51TTI3L0pFZwpyVTBzN0FDb1loOTcrNCtJTWpvWmZsdGIrQk5DNFpTcG96aFBnQTBBaFdFeXRJeStEN1VZZ0hHZzZvMm1iaU5FCnVBQ2tCaTlOYUhObzN0Z2J5U3c4ZDNhT293clBDdWU0RXRRM0hkaU0wcHlZS25JSGxMWHI2MDVKa2tuRDVrNDkKYWhqS1BQUmFOWmM2VmlmUS9PM2J6RWc1TC9WV3oraWd4UmtLK3dJREFRQUJBb0lCQURUci82L05neEtXNXFIRgpxWmV5UDFYKzRjZ1RGR0JXV0JWYTc5bm1ZTjVRSm91Umliejg5WkwwWm1rTmViTVpqZWwySEpDK3Q5Vm1xWHNTCm5CZ2EzUDRSekhaNU5qSE15cHAweHRLMHhkS3ZKRVFTVWRFMWtLMS9DYW5BRnZQaUtYZW81TFZ6TlhnWHYzRk8KSEY1cUxNVy9sanc5OVFOVllNUUdjK0xGT2Q4V0pRRm9KZVh3bFBKdC9vK3NWUCtHK3dtYmp3VUNtc3ZqTFc5NAoxbS9XeWtwVlMrMCt3QVRJTGJCeTJqUUVtK3hwZk1IT0xhU01Hem9oUnEySHFvY1d2NE10OWI4bmFPNURyQ3dCCmZIQnIxd3VtL3duWGVESUFQay8yVVYxSllzdDQveVFjSENUS1V0Qzh3WFd6V3RabEE3dFNmM1huNUM0MXRhakoKaTIxMzNuRUNnWUVBd1BUOVRUS3R4NzZHYVovZE5WWStLY1ExYnplQlA5NlI0SFp0M1d4NllUbEJZZ2oxQWg4dQpZVGJCYmhYdDExcTQ0b0h4eXk0eThCVC9xWU55dWhIUGJjMExCYW4zamtwRVZEZ3VaempMSHc3YUUwTlVkNFNDCmFzd0FjTGlnMEdiTU1RMVJnZndHdVpOc3ZXZ2wrVXZWdVB3SVRKUzBUYkdQeVVGTnpZSmlsUjBDZ1lFQTF1STMKaFB1VjBoOGUyZEVHT3g5L0FXb1hCU3lhajNVZ2xTV2VnNTNBOFJ1OU9POTl5OGxHSXZxUDdsVXpub01xZzlwMwpEdXF3TFdCSnQwYU11SG1yZVF6RzhiME9ieG1UTmwreTdUaTIwOU9nUTBaZ0ZhSFhtbng2UStaWlovNFd0UXlMCmxzUUxPMDNqcjUwb3dQV3FJSnRmdEVXckZsYXBCd01IdFBxY0hQY0NnWUJVT1J0dlhOV1p3RUh2VE1aRGg3ZHUKaDZSRFBWai9SZWtxcnNiWnhZWE5kaTVoMmtNSGdCREFnaVFBSnpZL3IrdlBncVRGNlE1MjJOT1hiRGJoblBDTApudHBXeUkrejNocmhIL3l4TllzRHJYbm11cGJkN0VyNHpyN0tkT3hNeXZJSjN5UmF0bVZCRDJhQmJqcFZESkxuClFhdGxzZXJabHMyS0xTVUd0UWpPeVFLQmdDTUVpbmVwYUNqdDJXRTFwd1pMZzY3L1V2RU02OE1sRzBJeTl6b0wKb3oybmY1YWZzS2hkRElqMnZDM2l4SjY0VVYxYlpMdVJrMVJSdExHaFZaMWFkbzhyZmVBM0p2c0Q5L0dNZHVRaApzZW1qT3lLN2ZUMEpPdEY2SjVLWkgwakxKM2dwcXpYT2pPWUZNMjJTOXVrN0g1Wmd1NS90R294eFl1bzVuQy9xCkJ6YlJBb0dBVHUwc2Rnb2dxL2RHdDFIVjNmZnhyVGhVazBMdVZxM0x4Ni9mN0tXWGN4b21qUWlQc0RZUTVkU3IKb21Kb2wvVUxWUnRQbGJtcmZKN1VpcHBlNXh1eUFNQUVocldIeG15SlR2TWZQSks3UGs4eHgrcUlSUGd0YklobwpWcjN5ZFoyYnRka2h4OXo5dGY4WXJ5RkxqNnNIWTY4V1Z2T3hETkcwYTZPYUhWWFJJMUE9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
    kind: Secret
    metadata:
      name: cm-adapter-serving-certs
      namespace: custom-metrics

创建k8s-prometheus-adapter configmap,可以使用这个方式

kubectl create  secret generic  cm-adapter-serving-certs --from-file=serving.crt=apiserver.crt --from-file=serving.key=apiserver.key -n custom-metrics

也可以使用这种方式

kubectl create -f cm-adapter-serving-certs.yaml

**创建`custom-metrics namespace`**

kubectl create namespace custom-metrics

**部署**

修改镜像`custom-metrics-apiserver-deployment.yaml`的镜像

cd /opt/ssl/custom-metrics/k8s-prometheus-adapter/deploy/manifests
    
    修改
        custom-metrics-apiserver-deployment.yaml
    
       image: directxman12/k8s-prometheus-adapter-amd64
        to 
       image: harbor.enncloud.cn/enncloud/k8s-prometheus-adapter-amd64:v0.2.1
    
       - --prometheus-url=http://prometheus.prom.svc:9090/
       to  
       - --prometheus-url=http://prometheus-monitor.kube-system:9090/
    
      - --config=/default-config.yaml 
      to 
      - --config=/etc/adapter/config.yaml
    添加这个
    - --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

修改之后

apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: custom-metrics-apiserver
      name: custom-metrics-apiserver
      namespace: custom-metrics
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: custom-metrics-apiserver
      template:
        metadata:
          labels:
            app: custom-metrics-apiserver
          name: custom-metrics-apiserver
        spec:
          serviceAccountName: custom-metrics-apiserver
          containers:
          - name: custom-metrics-apiserver
            image: directxman12/k8s-prometheus-adapter-amd64
            args:
            - /adapter
            - --secure-port=6443
            - --tls-cert-file=/var/run/serving-cert/serving.crt
            - --tls-private-key-file=/var/run/serving-cert/serving.key
            - --logtostderr=true
            - --prometheus-url=http://prometheus-monitor.kube-system:9090/
            - --metrics-relist-interval=1m
            - --requestheader-client-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            - --v=10
            - --config=/etc/adapter/config.yaml
            ports:
            - containerPort: 6443
            volumeMounts:
            - mountPath: /var/run/serving-cert
              name: volume-serving-cert
              readOnly: true
            - mountPath: /etc/adapter/
              name: config
              readOnly: true
            - mountPath: /tmp
              name: tmp-vol
          volumes:
          - name: volume-serving-cert
            secret:
              secretName: cm-adapter-serving-certs
          - name: config
            configMap:
              name: adapter-config
          - name: tmp-vol
            emptyDir: {}

**添加kube-apiserver的启动参数**

--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \
             --requestheader-allowed-names=aggregator \
             --requestheader-extra-headers-prefix=X-Remote-Extra- \
             --requestheader-group-headers=X-Remote-Group \
             --requestheader-username-headers=X-Remote-User \
             --proxy-client-cert-file=/etc/kubernetes/ssl/aggregator-proxy.pem \
             --proxy-client-key-file=/etc/kubernetes/ssl/aggregator-proxy-key.pem \
             --enable-aggregator-routing=true \

添加kube-controller-manager的启动参数

"--horizontal-pod-autoscaler-use-rest-clients",
     "--horizontal-pod-autoscaler-sync-period=60s",

cd /opt/ssl/custom-metrics/k8s-prometheus-adapter/deploy/manifests
    kubectl create -f .

查看部署结果

[root@master-47-35 custom-metrics]# kubectl get all -n custom-metrics 
    NAME                                           READY     STATUS    RESTARTS   AGE
    pod/custom-metrics-apiserver-5d8dd96b5-8s9rq   1/1       Running   0          16h
    
    NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
    service/custom-metrics-apiserver   ClusterIP   10.254.40.72   <none>        443/TCP   16h
    
    NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/custom-metrics-apiserver   1         1         1            1           16h
    
    NAME                                                 DESIRED   CURRENT   READY     AGE
    replicaset.apps/custom-metrics-apiserver-5d8dd96b5   1         1         1         16h

查看api-versions

[root@master-47-35 k8s-prometheus-adapter]# kubectl api-versions 
    admissionregistration.k8s.io/v1beta1
    apiextensions.k8s.io/v1beta1
    apiregistration.k8s.io/v1
    apiregistration.k8s.io/v1beta1
    apps/v1
    apps/v1beta1
    apps/v1beta2
    authentication.k8s.io/v1
    authentication.k8s.io/v1beta1
    authorization.k8s.io/v1
    authorization.k8s.io/v1beta1
    autoscaling/v1
    autoscaling/v2beta1
    batch/v1
    batch/v1beta1
    certificates.k8s.io/v1beta1
    custom.metrics.k8s.io/v1beta1
    events.k8s.io/v1beta1
    extensions/v1beta1
    monitoring.coreos.com/v1
    networking.k8s.io/v1
    policy/v1beta1
    rbac.authorization.k8s.io/v1
    rbac.authorization.k8s.io/v1beta1
    scheduling.k8s.io/v1beta1
    storage.k8s.io/v1
    storage.k8s.io/v1beta1
    v1

确认这个`custom.metrics.k8s.io/v1beta1` api存在

root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
    {
       "kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{
       "name":"pods/fs_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/spec_memory_reservation_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_io_time","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_reads_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_writes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/network_tcp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_sector_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_writes_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/spec_memory_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/start_time_seconds","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/spec_cpu_shares","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/spec_memory_reservation_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_read","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_cfs_throttled_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_load_average_10s","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_swap","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_failures","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/start_time_seconds","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_max_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/spec_cpu_period","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_schedstat_run","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_io_time_weighted","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/network_tcp_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_io_current","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/spec_cpu_quota","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_cfs_throttled","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_cfs_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_io_current","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_reads_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_reads","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_writes_merged","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_failures","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_user","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_write","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_cfs_throttled_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_schedstat_run_periods","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_load_average_10s","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/last_seen","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/spec_cpu_period","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_system","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_read","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_schedstat_run_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_sector_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_rss","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_rss","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_working_set_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_inodes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_reads_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_writes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_cache","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/spec_cpu_shares","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_reads_merged","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_sector_reads","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_failcnt","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/spec_memory_swap_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/spec_memory_swap_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_schedstat_run","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/last_seen","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_cache","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_writes_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_max_usage_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_working_set_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_schedstat_runqueue","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_system","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_cfs_periods","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_sector_writes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_writes_merged","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/network_udp_usage","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_swap","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/cpu_cfs_throttled","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_inodes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/network_udp_usage","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/spec_memory_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/tasks_state","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_io_time","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_write","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/memory_usage_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/spec_cpu_quota","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_inodes_free","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_limit_bytes","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/memory_failcnt","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/tasks_state","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/cpu_schedstat_runqueue","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_io_time_weighted","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{
       "name":"pods/fs_reads","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{
       "name":"namespaces/fs_inodes_free","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]}]}

**接下来就是验证了**

获取CPU cpu\_usage

[root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/cpu_usage"
    {
       "kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{
       "selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/cpu_usage"},"items":[{
       "describedObject":{
       "kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-qg2gj","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"},{
       "describedObject":{
       "kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-t4xf2","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"},{
       "describedObject":{
       "kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-w26gb","apiVersion":"/__internal"},"metricName":"cpu_usage","timestamp":"2018-09-12T14:49:30Z","value":"0"}]}
    
    获取内存 memory_usage_bytes
    [root@master-47-35 ~]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/memory_usage_bytes"
    {
       "kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{
       "selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/memory_usage_bytes"},"items":[{
       "describedObject":{
       "kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-qg2gj","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"3547136"},{
       "describedObject":{
       "kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-t4xf2","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"4435968"},{
       "describedObject":{
       "kind":"Pod","namespace":"default","name":"nginx-dm-fff68d674-w26gb","apiVersion":"/__internal"},"metricName":"memory_usage_bytes","timestamp":"2018-09-12T14:52:46Z","value":"6172672"}]}

查看`v1beta1.custom.metrics.k8s.io`是否存在

[root@master-47-35 ~]# kubectl get apiservice v1beta1.custom.metrics.k8s.io -o yaml
    apiVersion: apiregistration.k8s.io/v1
    kind: APIService
    metadata:
      creationTimestamp: 2018-09-12T08:57:08Z
      deletionGracePeriodSeconds: 0
      deletionTimestamp: 2018-09-12T10:34:21Z
      finalizers:
      - foregroundDeletion
      name: v1beta1.custom.metrics.k8s.io
      resourceVersion: "3189364"
      selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
      uid: d470a606-b669-11e8-aec0-5254e98192ae
    spec:
      group: custom.metrics.k8s.io
      groupPriorityMinimum: 100
      insecureSkipTLSVerify: true
      service:
        name: custom-metrics-apiserver
        namespace: custom-metrics
      version: v1beta1
      versionPriority: 100
    status:
      conditions:
      - lastTransitionTime: 2018-09-12T10:14:40Z
        message: all checks passed
        reason: Passed
        status: "True"
        type: Available

基于内存

apiVersion: autoscaling/v1
    kind: HorizontalPodAutoscaler
    metadata:
      annotations:
        autoscaling.alpha.kubernetes.io/conditions: '[{
       "type":"AbleToScale","status":"True","lastTransitionTime":"2018-09-13T02:23:13Z","reason":"SucceededGetScale","message":"the HPA controller was able to get the target''s current scale"},{
       "type":"ScalingActive","status":"False","lastTransitionTime":"2018-09-13T02:24:13Z","reason":"ScalingDisabled","message":"scaling is disabled since the replica count of the target is zero"},{
       "type":"ScalingLimited","status":"True","lastTransitionTime":"2018-09-13T02:03:10Z","reason":"TooManyReplicas","message":"the desired replica count is more than the maximum replica count"}]'
        autoscaling.alpha.kubernetes.io/metrics: '[{
       "type":"Pods","pods":{
       "metricName":"memory_usage_bytes","targetAverageValue":"1M"}}]'
      creationTimestamp: 2018-09-13T02:01:10Z
      name: example2
      namespace: demo
      resourceVersion: "3299536"
      selfLink: /apis/autoscaling/v1/namespaces/demo/horizontalpodautoscalers/example2
      uid: e2be2af9-b6f8-11e8-bf72-5254e98192ae
    spec:
      maxReplicas: 2
      minReplicas: 1
      scaleTargetRef:
        apiVersion: apps/v1beta1
        kind: Deployment
        name: example2
    status:
      currentReplicas: 0
      desiredReplicas: 0
      lastScaleTime: 2018-09-13T02:17:13Z

**基于cpu cpu\_usage的使用**

apiVersion: autoscaling/v1
    kind: HorizontalPodAutoscaler
    metadata:
      annotations:
        autoscaling.alpha.kubernetes.io/conditions: '[{
       "type":"AbleToScale","status":"True","lastTransitionTime":"2018-09-13T02:32:34Z","reason":"ReadyForNewScale","message":"the last scale time was sufficiently old as to warrant a new scale"},{
       "type":"ScalingActive","status":"True","lastTransitionTime":"2018-09-13T02:23:33Z","reason":"ValidMetricFound","message":"the HPA was able to successfully calculate a replica count from pods metric cpu_usage"},{
       "type":"ScalingLimited","status":"True","lastTransitionTime":"2018-09-13T02:23:33Z","reason":"TooManyReplicas","message":"the desired replica count is more than the maximum replica count"}]'
        autoscaling.alpha.kubernetes.io/current-metrics: '[{
       "type":"Pods","pods":{
       "metricName":"cpu_usage","currentAverageValue":"322m"}}]'
        autoscaling.alpha.kubernetes.io/metrics: '[{
       "type":"Pods","pods":{
       "metricName":"cpu_usage","targetAverageValue":"30m"}}]'
      creationTimestamp: 2018-09-13T02:21:33Z
      name: example
      namespace: demo
      resourceVersion: "3301437"
      selfLink: /apis/autoscaling/v1/namespaces/demo/horizontalpodautoscalers/example
      uid: bbc859b6-b6fb-11e8-bf72-5254e98192ae
    spec:
      maxReplicas: 4
      minReplicas: 2
      scaleTargetRef:
        apiVersion: apps/v1beta1
        kind: Deployment
        name: example
    status:
      currentReplicas: 4
      desiredReplicas: 4
      lastScaleTime: 2018-09-13T02:26:34Z

不过有个问题[issues][]，但是不影响使用

end

参考

[部署方式][Link 1]

[https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/v0.2.1/docs/walkthrough.md][https_github.com_DirectXMan12_k8s-prometheus-adapter_blob_v0.2.1_docs_walkthrough.md]

[requestheader-authentication]: https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md#requestheader-authentication
[issues]: https://github.com/kubernetes-incubator/metrics-server/issues/27
[Link 1]: https://github.com/DirectXMan12/k8s-prometheus-adapter/tree/v0.2.1/deploy
[https_github.com_DirectXMan12_k8s-prometheus-adapter_blob_v0.2.1_docs_walkthrough.md]: https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/v0.2.1/docs/walkthrough.md