在CentOS7上用kubeadm HAProxy Keepalived 安装多Master节点的高可用Kubernetes集群

ゞ浴缸里的玫瑰 2022-12-18 01:52 101阅读 0赞

### 文章目录 ###

*  在CentOS7上用kubeadm HAProxy Keepalived 安装多Master节点的高可用Kubernetes集群
 *   *  前言
     *  官方文档
     *  部署规划
     *   *  资源规划
         *  集群规划
     *  部署负载均衡服务器
     *   *  安装和配置HAProxy
         *  安装和配置Keepalived
         *  让HAProxy使用Keepalived VIP
     *  部署第一个节点
     *   *  拉取k8s-deploy项目
         *  安装Kubernetes
         *  克隆服务器作为基准镜像
     *  部署第一个Master节点
     *   *  初始化集群
         *  安装Calico网络组件
     *  部署Worker节点
     *  部署其他Master节点
     *  查看集群部署情况
     *  (可选）安装Metrics Server
     *  (可选）安装Prometheus和Grafana
     *  Troubleshooting
     *   *  用VMWare试验时怎么设置主机名和IP
         *  官方Yum源安装太慢，可使用阿里云Yum源
         *  服务器上不了网，可通过HTTP Proxy正向代理上网
         *   *   *  设置docker pull proxy
         *  从Docker Hub拉取镜像太慢，可使用阿里云镜像加速器
         *  拉取Google镜像太慢，可使用阿里云镜像仓库
         *  重启机器后，集群不可用
     *  其他内容
     *   *  Kubernetes兼容的Docker版本
         *  本例试验环境
         *  Kubernetes集群安装和部署难点
     *  参考文档

# 在CentOS7上用kubeadm HAProxy Keepalived 安装多Master节点的高可用Kubernetes集群 #

## 前言 ##

之前写过一篇文章[使用kubeadm一键部署kubernetes集群 v1.10.3 v1.11.0 v1.13.0][kubeadm_kubernetes_ v1.10.3 v1.11.0 v1.13.0] 和编写了一个一键部署单Master节点的Kubernetes集群的开源项目[k8s-deploy][]。

现在Kubernetes已更新到v1.19，所以重新更新了[k8s-deploy][]项目，支持部署多Master节点的高可用Kubernetes集群。

> 在正式安装前，推荐先阅读[k8s-deploy][]项目和Troubleshooting中的常见问题。

## 官方文档 ##

*  [使用 kubeadm 引导集群][kubeadm]
 *  [使用 kubeadm 创建集群][kubeadm 1]
 *  [高可用拓扑选项][Link 1]
 *  [利用 kubeadm 创建高可用集群][kubeadm 2]
 *  [Bootstrapping clusters with kubeadm][]
 *  [Creating a cluster with kubeadm][]
 *  [Options for Highly Available topology][]
 *  [Creating Highly Available clusters with kubeadm][]

## 部署规划 ##

### 资源规划 ###

全部节点都在一个低延迟的子网中：

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>IP</th> 
   <th>说明</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td><code>k8s-lb-01</code></td> 
   <td>192.168.0.151<br>192.168.0.170</td> 
   <td>HAProxy + Keepalived (master)</td> 
  </tr> 
  <tr> 
   <td><code>k8s-lb-02</code></td> 
   <td>192.168.0.161<br>192.168.0.170</td> 
   <td>HAProxy + Keepalived (backup)</td> 
  </tr> 
  <tr> 
   <td><code>k8s-master-01</code></td> 
   <td>192.168.0.152</td> 
   <td>Master节点1</td> 
  </tr> 
  <tr> 
   <td><code>k8s-master-02</code></td> 
   <td>192.168.0.153</td> 
   <td>Master节点2</td> 
  </tr> 
  <tr> 
   <td><code>k8s-master-03</code></td> 
   <td>192.168.0.162</td> 
   <td>Master节点3</td> 
  </tr> 
  <tr> 
   <td><code>k8s-worker-01</code></td> 
   <td>192.168.0.154</td> 
   <td>Worker节点1</td> 
  </tr> 
  <tr> 
   <td><code>k8s-worker-02</code></td> 
   <td>192.168.0.164</td> 
   <td>Worker节点2</td> 
  </tr> 
  <tr> 
   <td><code>k8s-worker-03</code></td> 
   <td>192.168.0.165</td> 
   <td>Worker节点3</td> 
  </tr> 
 </tbody> 
</table>

说明：

*  采用外部HAProxy作集群负载均衡器；
 *  采用2个 HAProxy + Keepalived 来保证负载均衡的高可用，Keepalived VIP为192.168.0.170；
 *  采用每个Master节点上伴随部署Etcd的方式([Slacked Etcd typology 堆叠Etcd拓扑][Slacked Etcd typology _Etcd]），而Etcd高可用最少需要3个节点（奇数），因此Master节点至少为3个。
 *  3个worker节点来保证worker节点高可用。
 *  服务器规格都为CentOS7 x86\_64 2vCPU/2G 。
 *  全部机器都在192.168.0.0/24网段，且相互连通。

### 集群规划 ###

计划部署“2负载均衡 + 3 Master节点 + 3 Worker节点”的高可用Kubernetes集群：

*  Docker 19.03.11
 *  Kubernetes v1.19.3
 *  Calico网络组件
 *  HAProxy负载均衡
 *  集群Control-Plane Endpoint：`192.168.0.170:6443`
 *  集群API Server(s)：
    
     *  `192.168.0.152:6443`
     *  `192.168.0.153:6443`
     *  `192.168.0.162:6443`

## 部署负载均衡服务器 ##

### 安装和配置HAProxy ###

安装HAProxy：

yum install haproxy -y

编辑`/etc/haproxy/haproxy.cfg`，删掉默认的代理设置，添加负载均衡反向代理到Kubernetes集群Master节点的设置：

#---------------------------------------------------------------------
    # apiserver frontend which proxys to the masters
    #---------------------------------------------------------------------
    frontend apiserver
        bind *:6443
        mode tcp
        option tcplog
        default_backend apiserver
    
    #---------------------------------------------------------------------
    # round robin balancing for apiserver
    #---------------------------------------------------------------------
    backend apiserver
        option httpchk GET /healthz
        http-check expect status 200
        mode tcp
        option ssl-hello-chk
        balance     roundrobin
            server k8s-master-01 192.168.0.152:6443 check
            server k8s-master-02 192.168.0.153:6443 check
            server k8s-master-03 192.168.0.162:6443 check

以服务方式运行HAProxy：

systemctl daemon-reload
    systemctl enable haproxy
    systemctl start haproxy

查看HAProxy服务状态：

systemctl status haproxy -l

> 如果HAProxy启动失败，请关闭SELinux再试。

### 安装和配置Keepalived ###

安装Keepalived：

yum install keepalived -y

配置Linux系统内核参数：

sudo cat <<EOF >  /etc/sysctl.d/keepalived.conf
    net.ipv4.ip_forward = 1
    net.ipv4.ip_nonlocal_bind = 1
    EOF
    
    sudo sysctl --system

说明：

*  `net.ipv4.ip_forward = 1` 允许IP转发
 *  `net.ipv4.ip_nonlocal_bind = 1` 允许绑定浮动IP，即VIP

在`/etc/keepalived`下新建`check_apiserver.sh`用来Keepalived作健康检查，内容为：

#!/bin/sh
    
    
    APISERVER_VIP=$1
    APISERVER_DEST_PORT=$2
    
    errorExit() { 
        echo "*** $*" 1>&2
        exit 1
    }
    
    curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"
    if ip addr | grep -q ${APISERVER_VIP}; then
        curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/"
    fi

为脚本增加可执行文件权限：

chmod +x /etc/keepalived/check_apiserver.sh

**在Keepalived master节点和backup节点的配置略有不同。**

在Keepalived master节点上的`/etc/keepalived/keepalived.conf`中的配置为：

! /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
    global_defs { 
        router_id LVS_DEVEL
    }
    vrrp_script check_apiserver { 
      script "/etc/keepalived/check_apiserver.sh 192.168.0.170 6443"
      interval 3
      weight -2
      fall 10
      rise 2
    }
    
    vrrp_instance VI_1 { 
        state MASTER
        interface ens33
        virtual_router_id 51
        priority 101
        authentication { 
            auth_type PASS
            auth_pass Keep@lived
        }
        virtual_ipaddress { 
            192.168.0.170
        }
        track_script { 
            check_apiserver
        }
    }

说明：

*  `script "/etc/keepalived/check_apiserver.sh 192.168.0.170 6443"` 声明Keepalived健康检查脚本。
 *  `state MASTER` 声明该节点为Keepalived master节点。
 *  `interface ens33` 声明该网卡名称。
 *  `priority 101` Keepalived master节点优先级，需要比Keepalived backup节点优先级大。
 *  `auth_pass Keep@lived` Keepalived master和backup节点需要设置一样的`auth_pass`
 *  `virtual_ipaddress` 为Keepalived VIP。

在Keepalived backup节点上的`/etc/keepalived/keepalived.conf`中的配置为：

! /etc/keepalived/keepalived.conf
    ! Configuration File for keepalived
    global_defs { 
        router_id LVS_DEVEL
    }
    vrrp_script check_apiserver { 
      script "/etc/keepalived/check_apiserver.sh 192.168.0.170 6443"
      interval 3
      weight -2
      fall 10
      rise 2
    }
    
    vrrp_instance VI_1 { 
        state BACKUP
        interface ens33
        virtual_router_id 51
        priority 100
        authentication { 
            auth_type PASS
            auth_pass Keep@lived
        }
        virtual_ipaddress { 
            192.168.0.170
        }
        track_script { 
            check_apiserver
        }
    }

说明：

*  `state BACKUP` 声明该节点为Keepalived backup节点。
 *  `priority 100` Keepalived backup节点优先级，需要比Keepalived master节点优先级小。

以服务方式运行Keepalived：

systemctl daemon-reload
    systemctl enable keepalived
    systemctl start keepalived

查看Keepalived服务状态：

systemctl status keepalived -l

### 让HAProxy使用Keepalived VIP ###

修改HAProxy的配置文件，绑定frontend IP为Keepalived VIP：

frontend apiserver
        bind 192.168.0.170:6443

重新启动HAProxy：

systemctl daemon-reload
    systemctl restart haproxy

## 部署第一个节点 ##

### 拉取k8s-deploy项目 ###

# 安装Git
    yum install git -y
    
    # 拉取k8s-deploy项目
    mkdir -p ~/k8s
    cd ~/k8s
    git clone https://github.com/cookcodeblog/k8s-deploy.git
    
    # 增加脚本的可执行文件权限
    cd k8s-deploy/kubeadm_v1.19.3
    find . -name '*.sh' -exec chmod u+x { } \;

### 安装Kubernetes ###

# 安装前检查和配置
    bash 01_pre_check_and_configure.sh
    
    # 安装Docker
    bash 02_install_docker.sh
    
    # 安装kubeadm, kubectl, kubelet
    bash 03_install_kubernetes.sh
    
    # 拉取Kubernetes集群要用到的镜像
    bash 04_pull_kubernetes_images_from_aliyun.sh
    
    # 拉取Calico网络组件的镜像
    bash 04_pull_calico_images.sh

### 克隆服务器作为基准镜像 ###

建议克隆这台服务器作为基准镜像，来加快后面的Kubernetes集群节点的部署。

## 部署第一个Master节点 ##

### 初始化集群 ###

cd ~/k8s
    cd k8s-deploy/kubeadm_v1.19.3
    
    # 05_kubeadm_init.sh <CONTROL_PLANE_ENDPOINT>
    bash 05_kubeadm_init.sh 192.168.0.170:6443

复制`kubeadm init`的输出日志，后面往集群中加入Master节点和Worker节点时需要用到。

### 安装Calico网络组件 ###

# support eth* and ens* networkd interfaces
    bash 06_install_calico.sh eth-ens

## 部署Worker节点 ##

运行`kubeadm init`中打印的日志中关于加入“worker node”的命令。

示例：

kubeadm join 192.168.0.170:6443 --token r7w69v.3e1nweyk81h5zj6y \
        --discovery-token-ca-cert-hash sha256:1234a2317d27f0a4c6bcf5f284416a2fb3e8f3bd61aa88bc279a4f6ef18e09a1

让Worker节点上的`kubectl`命令生效：

bash enable_kubectl_worker.sh

## 部署其他Master节点 ##

运行`kubeadm init`中打印的日志中关于加入“control-plane node"的命令。

示例：

kubeadm join 192.168.0.170:6443 --token r7w69v.3e1nweyk81h5zj6y \
        --discovery-token-ca-cert-hash sha256:1234a2317d27f0a4c6bcf5f284416a2fb3e8f3bd61aa88bc279a4f6ef18e09a1 \
        --control-plane --certificate-key 0e48107fbcd11cda60a5c2b76ae488b4ebf57223a4001acac799996740a6044e

如果忘记了`kubeadm join`命令的内容，可运行`kubeadm token create --print-join-command` 重新获取，并可运行`kubeadm init phase upload-certs --upload-certs` 获取新的`certificate-key`。

> Please note that the certificate-key gives access to cluster sensitive data, keep it secret!  
> As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use  
> “kubeadm init phase upload-certs --upload-certs” to reload certs afterward.

让Master节点上的`kubectl`命令生效：

bash enable_kubectl_master.sh

## 查看集群部署情况 ##

# Display cluster info
    kubectl cluster-info
    
    # Nodes
    kubectl get nodes
    
    # Display pds
    kubectl get pods --all-namespaces -o wide
    
    # Check pods status incase any pods are not in running status
    kubectl get pods --all-namespaces | grep -v Running

## (可选）安装Metrics Server ##

bash 07_install_metrics_server.sh

参见：

*  [在Kubernetes集群上安装Metrics Server][Kubernetes_Metrics Server]

## (可选）安装Prometheus和Grafana ##

bash 08_install_prometheus_grafana.sh

参见：

*  [用Prometheus和Grafana监控Kubenetes集群][Prometheus_Grafana_Kubenetes]

## Troubleshooting ##

### 用VMWare试验时怎么设置主机名和IP ###

参见：

*  [使用工作站和VMware搭建集群测试环境][VMware]

### 官方Yum源安装太慢，可使用阿里云Yum源 ###

参见：

*  [使用国内Yum源][Yum]

### 服务器上不了网，可通过HTTP Proxy正向代理上网 ###

如果节点机器不能上网，可以通过设置代理来上网，这样可以方便在线yum安装、下载文件和拉取镜像。

参见：

*  [在CentOS7上设置Squid HTTP Proxy正向代理][CentOS7_Squid HTTP Proxy]

> 如果使用了http proxy，在初始化Kubernetes集群前需要重置全局的`http_proxy`和`https_proxy`，并运行`systemctl restart kubelet`重启kubelet服务。

##### 设置docker pull proxy #####

使用`docker pull`通过代理拉取Docker镜像，参见：

*  [https://docs.docker.com/config/daemon/systemd/\#httphttps-proxy][https_docs.docker.com_config_daemon_systemd_httphttps-proxy]

`/etc/systemd/system/docker.service.d/http-proxy.conf`示例：

[Service]
    Environment="HTTP_PROXY=http://172.31.240.127:3128"
    Environment="HTTPS_PROXY=http://172.31.240.127:3128"
    Environment="NO_PROXY=127.0.0.1,localhost,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

注意，如果使用Squid作为proxy server，`/etc/systemd/system/docker.service.d/http-proxy.conf`中的HTTP\_PROXY和HTTPS\_PROXY应设置成一样，都为`http://proxy_server_ip:3128` （**http**)。

如果设置了`HTTPS_PROXY=https://172.31.240.127:3128` (**https**)，则会报错：

proxyconnect tcp: tls: first record does not look like a TLS handshake

参见：

*  [https://github.com/docker/for-linux/issues/415][https_github.com_docker_for-linux_issues_415]

### 从Docker Hub拉取镜像太慢，可使用阿里云镜像加速器 ###

参见：

*  [Docker国内Yum源和国内镜像仓库][Docker_Yum]

### 拉取Google镜像太慢，可使用阿里云镜像仓库 ###

*  如果无法从Google Kubernetes Yum源安装，可以通过阿里云Kubernetes Yum源来安装。
    
    参见：
    
     *  [Kubernetes国内镜像、下载安装包和拉取gcr.io镜像][Kubernetes_gcr.io]
     *  [https://github.com/cookcodeblog/docker-library][https_github.com_cookcodeblog_docker-library]

### 重启机器后，集群不可用 ###

**问题：**  
重启机器后，集群不可用，运行`kubectl cluster-info`返回错误信息，运行`kubectl get nodes`没有响应，超时后报错“Error from server (InternalError): an error on the server ("") has prevented the request from succeeding (get nodes)“。

在Master节点上，运行`systemctl status kubelet -l` 和`journalctl -xefu kubelet` 报错“No node found"。

**原因：**  
少于3个Master节点启动时，Etcd高可用条件不满足，API Server退出。

**解决方法：**

1.  启动HAProxy机器。
2.  启动全部Master节点。
3.  （可选）在Master节点上重启kubelet: `systemctl restart kubelet`。

参见：

*  [https://serverfault.com/questions/1029654/deleting-a-control-node-from-the-cluster-kills-the-apiserver][https_serverfault.com_questions_1029654_deleting-a-control-node-from-the-cluster-kills-the-apiserver]

## 其他内容 ##

### Kubernetes兼容的Docker版本 ###

<table> 
 <thead> 
  <tr> 
   <th>Kubernetes版本</th> 
   <th>Docker版本</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td><a href="https://v1-15.docs.kubernetes.io/docs/setup/release/notes/" rel="nofollow">v1.15</a></td> 
   <td><code>1.13.1</code>, <code>17.03</code>, <code>17.06</code>, <code>17.09</code>, <code>18.06</code>, <code>18.09</code></td> 
  </tr> 
  <tr> 
   <td><a href="https://v1-16.docs.kubernetes.io/docs/setup/release/notes/" rel="nofollow">v1.16</a></td> 
   <td><code>1.13.1</code>, <code>17.03</code>, <code>17.06</code>, <code>17.09</code>, <code>18.06</code>, <code>18.09</code></td> 
  </tr> 
  <tr> 
   <td><a href="https://v1-17.docs.kubernetes.io/docs/setup/release/notes/" rel="nofollow">v1.17</a></td> 
   <td><code>1.13.1</code>, <code>17.03</code>, <code>17.06</code>, <code>17.09</code>, <code>18.06</code>, <code>18.09</code>, <code>19.03</code></td> 
  </tr> 
  <tr> 
   <td><a href="https://v1-18.docs.kubernetes.io/docs/setup/release/notes/" rel="nofollow">v1.18</a></td> 
   <td><code>1.13.1</code>, <code>17.03</code>, <code>17.06</code>, <code>17.09</code>, <code>18.06</code>, <code>18.09</code>, <code>19.03</code></td> 
  </tr> 
  <tr> 
   <td><a href="https://kubernetes.io/docs/setup/release/notes/" rel="nofollow">v1.19</a></td> 
   <td><code>1.13.1</code>, <code>17.03</code>, <code>17.06</code>, <code>17.09</code>, <code>18.06</code>, <code>18.09</code>, <code>19.03</code></td> 
  </tr> 
 </tbody> 
</table>

### 本例试验环境 ###

2台笔记本电脑：

*  Windows 10，Intel Core i5 / 16G，VMWare workstation 12 pro。
 *  Macbook pro，Intel Core i5 / 16G, VMWare fusion。
 *  虚拟机网络模式为桥接模式，网段为192.168.0.0/24，子网掩码为255.255.255.0，网关为192.168.0.1，网卡名称为ens33。

### Kubernetes集群安装和部署难点 ###

**Kubernetes集群安装和部署需要掌握：**

*  Linux系统管理知识。
 *  网络知识。
 *  部署架构知识，如果有公有云的IaaS经验就比较容易上手。
 *  安全知识。

**复杂性：**

*  版本：Kubernetes版本，Linux发行版，CPU arch版本
 *  Bug：Kubernetes bug，Docker bug，Linux bug，虚拟机软件 bug
 *  多种网络模式的不同：虚拟机桥接网络模式，虚拟机NAT网络模式，不同公有云的VPC网络模式，不同Kubernetes网络组件的网络模式
 *  网络受限：不能直接拉取Kubernetes镜像、Docker Hub镜像

**解决问题思路：**

*  从原理上分析和定位问题
 *  Kubernetes 官方文档
 *  Kubernetes GitHub issues
 *  Stackoverflow

## 参考文档 ##

*  [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/][Creating a cluster with kubeadm]
 *  [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/][https_kubernetes.io_docs_setup_production-environment_tools_kubeadm_install-kubeadm]
 *  [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/][Creating Highly Available clusters with kubeadm]
 *  [https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md\#options-for-software-load-balancing][https_github.com_kubernetes_kubeadm_blob_master_docs_ha-considerations.md_options-for-software-load-balancing]
 *  [https://kubernetes.io/docs/setup/production-environment/container-runtimes/\#docker][https_kubernetes.io_docs_setup_production-environment_container-runtimes_docker]
 *  [https://docs.docker.com/engine/install/centos/][https_docs.docker.com_engine_install_centos]
 *  [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/\#pod-network][https_kubernetes.io_docs_setup_production-environment_tools_kubeadm_create-cluster-kubeadm_pod-network]
 *  [https://kubernetes.io/docs/concepts/cluster-administration/networking/\#how-to-implement-the-kubernetes-networking-model][https_kubernetes.io_docs_concepts_cluster-administration_networking_how-to-implement-the-kubernetes-networking-model]
 *  [https://kubernetes.io/docs/concepts/cluster-administration/addons/][https_kubernetes.io_docs_concepts_cluster-administration_addons]
 *  [https://docs.projectcalico.org/getting-started/kubernetes/][https_docs.projectcalico.org_getting-started_kubernetes]
 *  [https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/][https_kubernetes.io_docs_reference_setup-tools_kubeadm_kubeadm-init]
 *  [Unable to update cni config: No networks found in /etc/cni/net.d][Unable to update cni config_ No networks found in _etc_cni_net.d]
 *  [https://docs.projectcalico.org/releases][https_docs.projectcalico.org_releases]
 *  [https://github.com/kubernetes/kubernetes/issues/93472][https_github.com_kubernetes_kubernetes_issues_93472]
 *  [https://octetz.com/docs/2019/2019-03-26-ha-control-plane-kubeadm/][https_octetz.com_docs_2019_2019-03-26-ha-control-plane-kubeadm]
 *  [https://blog.scottlowe.org/2019/08/12/converting-kubernetes-to-ha-control-plane/][https_blog.scottlowe.org_2019_08_12_converting-kubernetes-to-ha-control-plane]
 *  [https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/][https_kubernetes.io_docs_reference_setup-tools_kubeadm_kubeadm-join]
 *  [https://blog.inkubate.io/install-and-configure-a-multi-master-kubernetes-cluster-with-kubeadm/][https_blog.inkubate.io_install-and-configure-a-multi-master-kubernetes-cluster-with-kubeadm]
 *  [https://dockerlabs.collabnix.com/kubernetes/beginners/Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.html][https_dockerlabs.collabnix.com_kubernetes_beginners_Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.html]
 *  [https://www.unixmen.com/configure-high-available-load-balancer-haproxy-keepalived/][https_www.unixmen.com_configure-high-available-load-balancer-haproxy-keepalived]
 *  [https://snapdev.net/2015/09/08/install-haproxy-and-keepalived-on-centos-7-for-mariadb-cluster/][https_snapdev.net_2015_09_08_install-haproxy-and-keepalived-on-centos-7-for-mariadb-cluster]
 *  [https://www.unixmen.com/installing-haproxy-for-load-balancing-on-centos-7][https_www.unixmen.com_installing-haproxy-for-load-balancing-on-centos-7]
 *  [https://blog.inkubate.io/install-and-configure-a-multi-master-kubernetes-cluster-with-kubeadm/][https_blog.inkubate.io_install-and-configure-a-multi-master-kubernetes-cluster-with-kubeadm]
 *  [https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md\#options-for-software-load-balancing][https_github.com_kubernetes_kubeadm_blob_master_docs_ha-considerations.md_options-for-software-load-balancing]
 *  [kubelet服务启动失败，错误代码255][kubelet_255]
 *  [https://kifarunix.com/configure-highly-available-haproxy-with-keepalived-on-ubuntu-20-04/][https_kifarunix.com_configure-highly-available-haproxy-with-keepalived-on-ubuntu-20-04]

[kubeadm_kubernetes_ v1.10.3 v1.11.0 v1.13.0]: https://blog.csdn.net/nklinsirui/article/details/80602724
[k8s-deploy]: https://github.com/cookcodeblog/k8s-deploy
[kubeadm]: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/
[kubeadm 1]: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
[Link 1]: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/ha-topology/
[kubeadm 2]: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/high-availability/
[Bootstrapping clusters with kubeadm]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/
[Creating a cluster with kubeadm]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
[Options for Highly Available topology]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/
[Creating Highly Available clusters with kubeadm]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
[Slacked Etcd typology _Etcd]: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/ha-topology/#%E5%A0%86%E5%8F%A0-stacked-etcd-%E6%8B%93%E6%89%91
[Kubernetes_Metrics Server]: https://cookcode.blog.csdn.net/article/details/109413053
[Prometheus_Grafana_Kubenetes]: https://cookcode.blog.csdn.net/article/details/109512303
[VMware]: https://cookcode.blog.csdn.net/article/details/81916395
[Yum]: https://blog.csdn.net/nklinsirui/article/details/80146083
[CentOS7_Squid HTTP Proxy]: https://cookcode.blog.csdn.net/article/details/109225537
[https_docs.docker.com_config_daemon_systemd_httphttps-proxy]: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy
[https_github.com_docker_for-linux_issues_415]: https://github.com/docker/for-linux/issues/415
[Docker_Yum]: https://blog.csdn.net/nklinsirui/article/details/80490537
[Kubernetes_gcr.io]: https://blog.csdn.net/nklinsirui/article/details/80581286
[https_github.com_cookcodeblog_docker-library]: https://github.com/cookcodeblog/docker-library
[https_serverfault.com_questions_1029654_deleting-a-control-node-from-the-cluster-kills-the-apiserver]: https://serverfault.com/questions/1029654/deleting-a-control-node-from-the-cluster-kills-the-apiserver
[https_kubernetes.io_docs_setup_production-environment_tools_kubeadm_install-kubeadm]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
[https_github.com_kubernetes_kubeadm_blob_master_docs_ha-considerations.md_options-for-software-load-balancing]: https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#options-for-software-load-balancing
[https_kubernetes.io_docs_setup_production-environment_container-runtimes_docker]: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#docker
[https_docs.docker.com_engine_install_centos]: https://docs.docker.com/engine/install/centos/
[https_kubernetes.io_docs_setup_production-environment_tools_kubeadm_create-cluster-kubeadm_pod-network]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network
[https_kubernetes.io_docs_concepts_cluster-administration_networking_how-to-implement-the-kubernetes-networking-model]: https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model
[https_kubernetes.io_docs_concepts_cluster-administration_addons]: https://kubernetes.io/docs/concepts/cluster-administration/addons/
[https_docs.projectcalico.org_getting-started_kubernetes]: https://docs.projectcalico.org/getting-started/kubernetes/
[https_kubernetes.io_docs_reference_setup-tools_kubeadm_kubeadm-init]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/
[Unable to update cni config_ No networks found in _etc_cni_net.d]: https://github.com/kubernetes/kubernetes/issues/54918
[https_docs.projectcalico.org_releases]: https://docs.projectcalico.org/releases
[https_github.com_kubernetes_kubernetes_issues_93472]: https://github.com/kubernetes/kubernetes/issues/93472
[https_octetz.com_docs_2019_2019-03-26-ha-control-plane-kubeadm]: https://octetz.com/docs/2019/2019-03-26-ha-control-plane-kubeadm/
[https_blog.scottlowe.org_2019_08_12_converting-kubernetes-to-ha-control-plane]: https://blog.scottlowe.org/2019/08/12/converting-kubernetes-to-ha-control-plane/
[https_kubernetes.io_docs_reference_setup-tools_kubeadm_kubeadm-join]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/
[https_blog.inkubate.io_install-and-configure-a-multi-master-kubernetes-cluster-with-kubeadm]: https://blog.inkubate.io/install-and-configure-a-multi-master-kubernetes-cluster-with-kubeadm/
[https_dockerlabs.collabnix.com_kubernetes_beginners_Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.html]: https://dockerlabs.collabnix.com/kubernetes/beginners/Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.html
[https_www.unixmen.com_configure-high-available-load-balancer-haproxy-keepalived]: https://www.unixmen.com/configure-high-available-load-balancer-haproxy-keepalived/
[https_snapdev.net_2015_09_08_install-haproxy-and-keepalived-on-centos-7-for-mariadb-cluster]: https://snapdev.net/2015/09/08/install-haproxy-and-keepalived-on-centos-7-for-mariadb-cluster/
[https_www.unixmen.com_installing-haproxy-for-load-balancing-on-centos-7]: https://www.unixmen.com/installing-haproxy-for-load-balancing-on-centos-7
[kubelet_255]: https://blog.csdn.net/nklinsirui/article/details/80583971
[https_kifarunix.com_configure-highly-available-haproxy-with-keepalived-on-ubuntu-20-04]: https://kifarunix.com/configure-highly-available-haproxy-with-keepalived-on-ubuntu-20-04/