Kubernetes高可用集群二进制部署（四）部署kubectl和kube-controller-manager、kube-scheduler

清疚 2023-10-14 13:13 0阅读 0赞

[Kubernetes概述][Kubernetes]  
[使用kubeadm快速部署一个k8s集群][kubeadm_k8s]  
[Kubernetes高可用集群二进制部署（一）主机准备和负载均衡器安装][Kubernetes 1]  
[Kubernetes高可用集群二进制部署（二）ETCD集群部署][Kubernetes_ETCD]  
[Kubernetes高可用集群二进制部署（三）部署api-server][Kubernetes_api-server]  
[Kubernetes高可用集群二进制部署（四）部署kubectl和kube-controller-manager、kube-scheduler][Kubernetes_kubectl_kube-controller-manager_kube-scheduler]  
[Kubernetes高可用集群二进制部署（五）kubelet、kube-proxy、Calico、CoreDNS][Kubernetes_kubelet_kube-proxy_Calico_CoreDNS]  
[Kubernetes高可用集群二进制部署（六）Kubernetes集群节点添加][Kubernetes_Kubernetes]

### 1. 部署kubectl ###

#### 1.1 创建kubectl证书请求文件 ####

在master1执行

[root@k8s-master1 /]#cd /data/k8s-work
    cat > admin-csr.json << "EOF"
    {
        
      "CN": "admin",
      "hosts": [], #通过conf文件对集群进行访问，所以这里不指定主机
      "key": {
        
        "algo": "rsa",
        "size": 2048
      },
      "names": [
        {
        
          "C": "CN",
          "ST": "Beijing",
          "L": "Beijing",
          "O": "system:masters",             
          "OU": "system"
        }
      ]
    }
    EOF

说明：
    
    后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；
    kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；
    O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；
    注：
    这个admin 证书，是将来生成管理员用的kubeconfig 配置文件用的，现在我们一般建议使用RBAC 来对kubernetes 进行角色权限控制， kubernetes 将证书中的CN 字段 作为User， O 字段作为 Group；
    "O": "system:masters", 必须是system:masters，否则后面kubectl create clusterrolebinding报错。

#### 1.2 生成证书文件 ####

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#### 1.3 复制文件到指定目录 ####

cp admin*.pem /etc/kubernetes/ssl/

#### 1.4 生成kubeconfig配置文件 ####

```kube.config`` 为```kubectl``的配置文件，包含访问 `apiserver`` 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube.config
    
    # cat kube.config
    
    #设置管理员证书
    kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config
    
    # cat kube.config
    
    #设置安全上下文
    kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config
    
    # cat kube.config
    
    kubectl config use-context kubernetes --kubeconfig=kube.config

#### 1.5 准备kubectl配置文件并进行角色绑定 ####

mkdir ~/.kube
    cp kube.config ~/.kube/config
    kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes --kubeconfig=/root/.kube/config

#### 1.6 查看集群状态 ####

export KUBECONFIG=$HOME/.kube/config

kubectl cluster-info
    kubectl get componentstatuses
    kubectl get all --all-namespaces

#### 1.7 同步kubectl配置文件到集群其它master节点 ####

k8s-master2:
    mkdir /root/.kube
    
    k8s-master3:
    mkdir /root/.kube

scp /root/.kube/config k8s-master2:/root/.kube/config
    scp /root/.kube/config k8s-master3:/root/.kube/config

#### 1.8 配置kubectl命令补全(可选) ####

yum install -y bash-completion
    source /usr/share/bash-completion/bash_completion
    source <(kubectl completion bash)
    kubectl completion bash > ~/.kube/completion.bash.inc
    source '/root/.kube/completion.bash.inc'  
    source $HOME/.bash_profile

### 2. 部署kube-controller-manager ###

#### 2.1 创建kube-controller-manager证书请求文件 ####

在master1执行

[root@k8s-master1 k8s-work]#cat > kube-controller-manager-csr.json << "EOF"
    {
        
        "CN": "system:kube-controller-manager",
        "key": {
        
            "algo": "rsa",
            "size": 2048
        },
        "hosts": [
          "127.0.0.1",
          "192.168.10.103", #三台master的ip
          "192.168.10.104",
          "192.168.10.105"
        ],
        "names": [
          {
        
            "C": "CN",
            "ST": "Beijing",
            "L": "Beijing",
            "O": "system:kube-controller-manager",
            "OU": "system"
          }
        ]
    }
    EOF

说明：
    
    hosts 列表包含所有 kube-controller-manager 节点 IP；
    CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限

#### 2.2 创建kube-controller-manager证书文件 ####

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

# ls
    
    kube-controller-manager.csr     
    kube-controller-manager-csr.json
    kube-controller-manager-key.pem
    kube-controller-manager.pem

#### 2.3 创建kube-controller-manager的kube-controller-manager.kubeconfig ####

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-controller-manager.kubeconfig
    
    kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
    
    kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
    
    kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

#### 2.4 创建kube-controller-manager配置文件 ####

cat > kube-controller-manager.conf << "EOF"
    KUBE_CONTROLLER_MANAGER_OPTS="--port=10252 \
      --secure-port=10257 \
      --bind-address=127.0.0.1 \
      --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
      --service-cluster-ip-range=10.96.0.0/16 \
      --cluster-name=kubernetes \
      --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
      --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
      --allocate-node-cidrs=true \
      --cluster-cidr=10.244.0.0/16 \
      --experimental-cluster-signing-duration=87600h \
      --root-ca-file=/etc/kubernetes/ssl/ca.pem \
      --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
      --leader-elect=true \
      --feature-gates=RotateKubeletServerCertificate=true \
      --controllers=*,bootstrapsigner,tokencleaner \
      --horizontal-pod-autoscaler-use-rest-clients=true \
      --horizontal-pod-autoscaler-sync-period=10s \
      --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
      --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
      --use-service-account-credentials=true \
      --alsologtostderr=true \
      --logtostderr=false \
      --log-dir=/var/log/kubernetes \
      --v=2"
    EOF

#### 2.5 创建服务启动文件 ####

[root@k8s-master1 k8s-work]# cat > kube-controller-manager.service << "EOF"
    [Unit]
    Description=Kubernetes Controller Manager
    Documentation=https://github.com/kubernetes/kubernetes
    
    [Service]
    EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
    ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
    Restart=on-failure
    RestartSec=5
    
    [Install]
    WantedBy=multi-user.target
    EOF

#### 2.6 同步文件到集群master节点 ####

cp kube-controller-manager*.pem /etc/kubernetes/ssl/
    cp kube-controller-manager.kubeconfig /etc/kubernetes/
    cp kube-controller-manager.conf /etc/kubernetes/
    cp kube-controller-manager.service /usr/lib/systemd/system/

scp  kube-controller-manager*.pem k8s-master2:/etc/kubernetes/ssl/
    scp  kube-controller-manager*.pem k8s-master3:/etc/kubernetes/ssl/
    scp  kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master2:/etc/kubernetes/
    scp  kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master3:/etc/kubernetes/
    scp  kube-controller-manager.service k8s-master2:/usr/lib/systemd/system/
    scp  kube-controller-manager.service k8s-master3:/usr/lib/systemd/system/

#查看证书
    openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text

#### 2.7 启动服务 ####

systemctl daemon-reload 
    systemctl enable --now kube-controller-manager
    systemctl status kube-controller-manager

kubectl get componentstatuses

### 3. 部署kube-scheduler ###

#### 3.1 创建kube-scheduler证书请求文件 ####

[root@k8s-master1 k8s-work]# cat > kube-scheduler-csr.json << "EOF"
    {
        
        "CN": "system:kube-scheduler",
        "hosts": [
          "127.0.0.1",
          "192.168.10.103",
          "192.168.10.104",
          "192.168.10.105"
        ],
        "key": {
        
            "algo": "rsa",
            "size": 2048
        },
        "names": [
          {
        
            "C": "CN",
            "ST": "Beijing",
            "L": "Beijing",
            "O": "system:kube-scheduler",
            "OU": "system"
          }
        ]
    }
    EOF

#### 3.2 生成kube-scheduler证书 ####

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

# ls
    kube-scheduler.csr
    kube-scheduler-csr.json
    kube-scheduler-key.pem
    kube-scheduler.pem

#### 3.3 创建kube-scheduler的kubeconfig ####

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.10.100:6443 --kubeconfig=kube-scheduler.kubeconfig
    
    kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
    
    kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
    
    kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

#### 3.4 创建服务配置文件 ####

cat > kube-scheduler.conf << "EOF"
    KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \
    --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
    --leader-elect=true \
    --alsologtostderr=true \
    --logtostderr=false \
    --log-dir=/var/log/kubernetes \
    --v=2"
    EOF

#### 3.5创建服务启动配置文件 ####

cat > kube-scheduler.service << "EOF"
    [Unit]
    Description=Kubernetes Scheduler
    Documentation=https://github.com/kubernetes/kubernetes
    
    [Service]
    EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
    ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
    Restart=on-failure
    RestartSec=5
    
    [Install]
    WantedBy=multi-user.target
    EOF

#### 3.6 同步文件至集群master节点 ####

cp kube-scheduler*.pem /etc/kubernetes/ssl/
    cp kube-scheduler.kubeconfig /etc/kubernetes/
    cp kube-scheduler.conf /etc/kubernetes/
    cp kube-scheduler.service /usr/lib/systemd/system/

scp  kube-scheduler*.pem k8s-master2:/etc/kubernetes/ssl/
    scp  kube-scheduler*.pem k8s-master3:/etc/kubernetes/ssl/
    scp  kube-scheduler.kubeconfig kube-scheduler.conf k8s-master2:/etc/kubernetes/
    scp  kube-scheduler.kubeconfig kube-scheduler.conf k8s-master3:/etc/kubernetes/
    scp  kube-scheduler.service k8s-master2:/usr/lib/systemd/system/
    scp  kube-scheduler.service k8s-master3:/usr/lib/systemd/system/

#### 3.7 启动服务 ####

systemctl daemon-reload
    systemctl enable --now kube-scheduler
    systemctl status kube-scheduler

kubectl get componentstatuses

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-KAnxrdBq-1691074083716)(img/2023-08-02-22-15-45.png)\]\#\# 1. 部署kubectl

#### 1.1 创建kubectl证书请求文件 ####

在master1执行

#### 1.2 生成证书文件 ####

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#### 1.3 复制文件到指定目录 ####

cp admin*.pem /etc/kubernetes/ssl/

#### 1.4 生成kubeconfig配置文件 ####

```kube.config`` 为```kubectl``的配置文件，包含访问 `apiserver`` 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书

#### 1.5 准备kubectl配置文件并进行角色绑定 ####

#### 1.6 查看集群状态 ####

export KUBECONFIG=$HOME/.kube/config

kubectl cluster-info
    kubectl get componentstatuses
    kubectl get all --all-namespaces

#### 1.7 同步kubectl配置文件到集群其它master节点 ####

k8s-master2:
    mkdir /root/.kube
    
    k8s-master3:
    mkdir /root/.kube

scp /root/.kube/config k8s-master2:/root/.kube/config
    scp /root/.kube/config k8s-master3:/root/.kube/config

#### 1.8 配置kubectl命令补全(可选) ####

### 2. 部署kube-controller-manager ###

#### 2.1 创建kube-controller-manager证书请求文件 ####

在master1执行

#### 2.2 创建kube-controller-manager证书文件 ####

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

# ls
    
    kube-controller-manager.csr     
    kube-controller-manager-csr.json
    kube-controller-manager-key.pem
    kube-controller-manager.pem

#### 2.3 创建kube-controller-manager的kube-controller-manager.kubeconfig ####

#### 2.4 创建kube-controller-manager配置文件 ####

#### 2.5 创建服务启动文件 ####

#### 2.6 同步文件到集群master节点 ####

#查看证书
    openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text

#### 2.7 启动服务 ####

systemctl daemon-reload 
    systemctl enable --now kube-controller-manager
    systemctl status kube-controller-manager

kubectl get componentstatuses

### 3. 部署kube-scheduler ###

#### 3.1 创建kube-scheduler证书请求文件 ####

#### 3.2 生成kube-scheduler证书 ####

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

# ls
    kube-scheduler.csr
    kube-scheduler-csr.json
    kube-scheduler-key.pem
    kube-scheduler.pem

#### 3.3 创建kube-scheduler的kubeconfig ####

#### 3.4 创建服务配置文件 ####

#### 3.5创建服务启动配置文件 ####

#### 3.6 同步文件至集群master节点 ####

cp kube-scheduler*.pem /etc/kubernetes/ssl/
    cp kube-scheduler.kubeconfig /etc/kubernetes/
    cp kube-scheduler.conf /etc/kubernetes/
    cp kube-scheduler.service /usr/lib/systemd/system/

#### 3.7 启动服务 ####

systemctl daemon-reload
    systemctl enable --now kube-scheduler
    systemctl status kube-scheduler

kubectl get componentstatuses

![在这里插入图片描述][a83f191577b644fc97d1f6bdff970d3d.png]

[Kubernetes]: https://xy5462.blog.csdn.net/article/details/132093671?spm=1001.2014.3001.5502
[kubeadm_k8s]: https://xy5462.blog.csdn.net/article/details/132093723?spm=1001.2014.3001.5502
[Kubernetes 1]: https://xy5462.blog.csdn.net/article/details/132093826?spm=1001.2014.3001.5502
[Kubernetes_ETCD]: https://xy5462.blog.csdn.net/article/details/132093851?spm=1001.2014.3001.5502
[Kubernetes_api-server]: https://xy5462.blog.csdn.net/article/details/132093879?spm=1001.2014.3001.5502
[Kubernetes_kubectl_kube-controller-manager_kube-scheduler]: https://xy5462.blog.csdn.net/article/details/132093931?spm=1001.2014.3001.5502
[Kubernetes_kubelet_kube-proxy_Calico_CoreDNS]: https://xy5462.blog.csdn.net/article/details/132094015?spm=1001.2014.3001.5502
[Kubernetes_Kubernetes]: https://xy5462.blog.csdn.net/article/details/132094054?spm=1001.2014.3001.5502
[a83f191577b644fc97d1f6bdff970d3d.png]: https://img-blog.csdnimg.cn/a83f191577b644fc97d1f6bdff970d3d.png