Kubernetes高可用集群二进制部署（二）ETCD集群部署

超、凢脫俗 2023-10-14 13:13 2阅读 0赞

[Kubernetes概述][Kubernetes]  
[使用kubeadm快速部署一个k8s集群][kubeadm_k8s]  
[Kubernetes高可用集群二进制部署（一）主机准备和负载均衡器安装][Kubernetes 1]  
[Kubernetes高可用集群二进制部署（二）ETCD集群部署][Kubernetes_ETCD]  
[Kubernetes高可用集群二进制部署（三）部署api-server][Kubernetes_api-server]  
[Kubernetes高可用集群二进制部署（四）部署kubectl和kube-controller-manager、kube-scheduler][Kubernetes_kubectl_kube-controller-manager_kube-scheduler]  
[Kubernetes高可用集群二进制部署（五）kubelet、kube-proxy、Calico、CoreDNS][Kubernetes_kubelet_kube-proxy_Calico_CoreDNS]  
[Kubernetes高可用集群二进制部署（六）Kubernetes集群节点添加][Kubernetes_Kubernetes]

### 1. 配置免密登录 ###

> 在k8s-master1上操作

生成密钥对，把公钥复制到远程主机

ssh-keygen
    #直接回车，不设置密码

ssh-copy-id root@k8s-master1
    ssh-copy-id root@k8s-master2
    ssh-copy-id root@k8s-master3
    ssh-copy-id root@k8s-worker1

ssh root@k8s-master1
    #依次此时是否能否正常登录

### 2 部署ETCD集群 ###

> 在k8s-master1上操作。

#### 2.1 创建工作目录 ####

mkdir -p /data/k8s-work

#### 2.2 获取cfssl工具 ####

cd /data/k8s-work
    wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
    wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
    wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

说明：
    cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有：
    
    - cfssl，是CFSSL的命令行工具
    - cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。

chmod +x cfssl*

mv cfssl_linux-amd64 /usr/local/bin/cfssl
    mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
    mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

# cfssl version
    Version: 1.2.0
    Revision: dev
    Runtime: go1.6

#### 2.3 创建CA证书 ####

在master1上创建证书

##### 2.3.1 配置ca证书请求文件 #####

在当前目录/data/k8s-work执行

cat > ca-csr.json <<"EOF"
    {
        
      "CN": "kubernetes",
      "key": {
        
          "algo": "rsa",
          "size": 2048
      },
      "names": [
        {
        
          "C": "CN",
          "ST": "Beijing",
          "L": "Beijing",
          "O": "kubemsb",
          "OU": "CN"
        }
      ],
      "ca": {
        
              "expiry": "87600h"
      }
    }
    EOF

##### 2.3.2 创建ca证书 #####

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

##### 2.3.3 配置ca证书策略 #####

cat > ca-config.json <<"EOF"
    {
        
      "signing": {
        
          "default": {
        
              "expiry": "87600h"
            },
          "profiles": {
        
              "kubernetes": {
        
                  "usages": [
                      "signing",
                      "key encipherment",
                      "server auth",
                      "client auth"
                  ],
                  "expiry": "87600h"
              }
          }
      }
    }
    EOF

server auth 表示client可以对使用该ca对server提供的证书进行验证
    
    client auth 表示server可以使用该ca对client提供的证书进行验证

#### 2.4 创建etcd证书 ####

##### 2.4.1 配置etcd请求文件 #####

> 一般可以部署三台集群，可以是3、5、7、9

cat > etcd-csr.json <<"EOF"
    {
        
      "CN": "etcd",
      "hosts": [
        "127.0.0.1",
        "192.168.10.103",
        "192.168.10.104",
        "192.168.10.105"
      ],
      "key": {
        
        "algo": "rsa",
        "size": 2048
      },
      "names": [{
        
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "kubemsb",
        "OU": "CN"
      }]
    }
    EOF

##### 2.4.2 生成etcd证书 #####

确保在当前目录有`ca-key.pem`、`ca-config.json`、`etcd-csr.json`文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd

# ls
    输出
    ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

etcd.csr：etcd证书请求文件  
etcd-key.pem：私钥  
etcd.pem：基于私钥签发的证书

#### 2.5 部署etcd集群 ####

##### 2.5.1 下载etcd软件包 #####

![在这里插入图片描述][8b94d32e34304764ba232a4ed9a77183.png]

[etcd releases][]

![在这里插入图片描述][69c5c703499f4befae273502865bd155.png]

wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz

##### 2.5.2 安装etcd软件 #####

tar -xvf etcd-v3.5.2-linux-amd64.tar.gz
    cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/
    #-p 保留源文件的权限

##### 2.5.3 分发etcd软件 #####

scp etcd-v3.5.2-linux-amd64/etcd* k8s-master2:/usr/local/bin/
    
    scp etcd-v3.5.2-linux-amd64/etcd* k8s-master3:/usr/local/bin/

##### 2.5.4 创建配置文件 #####

在三个etcd节点操作

mkdir /etc/etcd

其中地址都是当前主机的ip本身

cat >  /etc/etcd/etcd.conf <<"EOF"
    #[Member]
    ETCD_NAME="etcd1"
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://192.168.10.103:2380"
    ETCD_LISTEN_CLIENT_URLS="https://192.168.10.103:2379,http://127.0.0.1:2379"
    
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.103:2380"
    ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.103:2379"
    ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.103:2380,etcd2=https://192.168.10.104:2380,etcd3=https://192.168.10.105:2380"
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new"
    EOF

说明：
    ETCD_NAME：节点名称，集群中唯一
    ETCD_DATA_DIR：数据目录
    ETCD_LISTEN_PEER_URLS：集群通信监听地址
    ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
    ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
    ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
    ETCD_INITIAL_CLUSTER：集群节点地址
    ETCD_INITIAL_CLUSTER_TOKEN：集群Token
    ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

##### 2.5.5 创建服务配置文件 #####

mkdir -p /etc/etcd/ssl
    mkdir -p /var/lib/etcd/default.etcd

cd /data/k8s-work
    cp ca*.pem /etc/etcd/ssl
    cp etcd*.pem /etc/etcd/ssl

cat > /etc/systemd/system/etcd.service <<"EOF"
    [Unit]
    Description=Etcd Server
    After=network.target
    After=network-online.target
    Wants=network-online.target
    
    [Service]
    Type=notify
    EnvironmentFile=-/etc/etcd/etcd.conf
    WorkingDirectory=/var/lib/etcd/
    ExecStart=/usr/local/bin/etcd \
      --cert-file=/etc/etcd/ssl/etcd.pem \
      --key-file=/etc/etcd/ssl/etcd-key.pem \
      --trusted-ca-file=/etc/etcd/ssl/ca.pem \
      --peer-cert-file=/etc/etcd/ssl/etcd.pem \
      --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
      --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
      --peer-client-cert-auth \
      --client-cert-auth
    Restart=on-failure
    RestartSec=5
    LimitNOFILE=65536
    
    [Install]
    WantedBy=multi-user.target
    EOF

##### 2.5.6 同步etcd配置到集群其它master节点 #####

创建目录
    mkdir -p /etc/etcd
    mkdir -p /etc/etcd/ssl
    mkdir -p /var/lib/etcd/default.etcd

服务配置文件,需要修改etcd节点名称及IP地址
    for i in k8s-master2 k8s-master3
    do
    scp /etc/etcd/etcd.conf $i:/etc/etcd/
    done

k8s-master2:
    
    cat /etc/etcd/etcd.conf
    #[Member]
    ETCD_NAME="etcd2"
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://192.168.10.104:2380"
    ETCD_LISTEN_CLIENT_URLS="https://192.168.10.104:2379,http://127.0.0.1:2379"
    
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.104:2380"
    ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.104:2379"
    ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.103:2380,etcd2=https://192.168.10.104:2380,etcd3=https://192.168.10.105:2380"
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new"

k8s-master3:
    
    cat /etc/etcd/etcd.conf
    #[Member]
    ETCD_NAME="etcd3"
    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
    ETCD_LISTEN_PEER_URLS="https://192.168.10.105:2380"
    ETCD_LISTEN_CLIENT_URLS="https://192.168.10.105:2379,http://127.0.0.1:2379"
    
    #[Clustering]
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.105:2380"
    ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.105:2379"
    ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.103:2380,etcd2=https://192.168.10.104:2380,etcd3=https://192.168.10.105:2380"
    ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
    ETCD_INITIAL_CLUSTER_STATE="new"

证书文件
    for i in k8s-master2 k8s-master3
    do
    scp /etc/etcd/ssl/* $i:/etc/etcd/ssl
    done

服务启动配置文件
    for i in k8s-master2 k8s-master3
    do
    scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/
    done

##### 2.5.7 启动etcd集群 #####

三台主机全都执行

systemctl daemon-reload
    systemctl enable --now etcd.service  #现在开启并设置开机自启
    systemctl status etcd

##### 2.5.8 验证集群状态 #####

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.103:2379,https://192.168.10.104:2379,https://192.168.10.105:2379 endpoint health

![在这里插入图片描述][be9f4cb2c627416fbf6431b6ecaa229e.png]

耗时较长的话可以做一些优化工作

[Kubernetes]: https://xy5462.blog.csdn.net/article/details/132093671?spm=1001.2014.3001.5502
[kubeadm_k8s]: https://xy5462.blog.csdn.net/article/details/132093723?spm=1001.2014.3001.5502
[Kubernetes 1]: https://xy5462.blog.csdn.net/article/details/132093826?spm=1001.2014.3001.5502
[Kubernetes_ETCD]: https://xy5462.blog.csdn.net/article/details/132093851?spm=1001.2014.3001.5502
[Kubernetes_api-server]: https://xy5462.blog.csdn.net/article/details/132093879?spm=1001.2014.3001.5502
[Kubernetes_kubectl_kube-controller-manager_kube-scheduler]: https://xy5462.blog.csdn.net/article/details/132093931?spm=1001.2014.3001.5502
[Kubernetes_kubelet_kube-proxy_Calico_CoreDNS]: https://xy5462.blog.csdn.net/article/details/132094015?spm=1001.2014.3001.5502
[Kubernetes_Kubernetes]: https://xy5462.blog.csdn.net/article/details/132094054?spm=1001.2014.3001.5502
[8b94d32e34304764ba232a4ed9a77183.png]: https://img-blog.csdnimg.cn/8b94d32e34304764ba232a4ed9a77183.png
[etcd releases]: https://github.com/etcd-io/etcd/releases?page=2
[69c5c703499f4befae273502865bd155.png]: https://img-blog.csdnimg.cn/69c5c703499f4befae273502865bd155.png
[be9f4cb2c627416fbf6431b6ecaa229e.png]: https://img-blog.csdnimg.cn/be9f4cb2c627416fbf6431b6ecaa229e.png