《Kubernets部署篇：Centos7.6部署kubernetes1.17.4高可用集群(方案二)》

「爱情、让人受尽委屈。」 2023-01-16 02:55 167阅读 0赞

### 文章目录 ###

*  一、架构图
 *  二、部署环境
 *  三、环境初始化
 *   *  3.1、内核升级
     *  3.2、系统初始化配置
 *  四、ssh双机互信
 *  五、Docker部署
 *  六、kubernetes部署
 *   *  6.1、设置kubernetes源
     *  6.2、安装kubelet、kubeadm和kubectl
     *  6.3、k8s相关镜像下载
     *  6.4、kubernetes master节点部署
     *   *  6.4.1、创建初始化配置文件
         *  6.4.2、k8s-master-149起虚ip
         *  6.4.3、master初始化
         *  6.4.3、配置kubectl
         *  6.4.4、配置flannel网络
         *  6.4.5、设置kubelet开机自启
         *  6.4.6、证书分发
         *  6.4.7、control plane节点加入k8s集群
     *  6.5、kubernetes worker节点部署
 *  七、LVS+Keepalived部署
 *   *  7.1、安装ipvs
     *  7.2、加载ipvsadm模块
     *  7.3、keepalived安装
 *  八、master节点配置
 *  九、client节点配置
 *  十、k8s集群高可用测试
 *   *  10.1、组件所在节点查看
     *   *  10.1.1、apiserver节点查看
         *  10.1.2、controller-manager和scheduler节点查看
     *  10.2、k8s-master-155关机
     *   *  10.2.1、关闭k8s-master-155
         *  10.2.1、apiserver组件节点查看
         *  10.2.3、controller-manager和scheduler组件节点查看
 *  十一、k8s集群集群功能性测试
 *  十二、lvs-keepalived集群高可用测试
 *  总结：整理不易，如果对你有帮助，可否点赞关注一下？

--------------------

# 一、架构图 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3ODE0MTEy_size_16_color_FFFFFF_t_70]  
**集群模式高可用架构说明**

<table> 
 <thead> 
  <tr> 
   <th>核心组件</th> 
   <th>高可用模式</th> 
   <th>高可用实现方式</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>apiserver</td> 
   <td>集群</td> 
   <td>lvs+keepalived</td> 
  </tr> 
  <tr> 
   <td>controller-manager</td> 
   <td>主备</td> 
   <td>leader election</td> 
  </tr> 
  <tr> 
   <td>scheduler</td> 
   <td>主备</td> 
   <td>leader election</td> 
  </tr> 
  <tr> 
   <td>etcd</td> 
   <td>集群</td> 
   <td>kubeadm</td> 
  </tr> 
 </tbody> 
</table>

**1、apiserver 通过lvs-keepalived实现高可用，vip将请求分发至各个control plane节点的apiserver组件。**  
**2、controller-manager k8s内部通过选举方式产生领导者(由–leader-elect 选型控制，默认为true)，同一时刻集群内只有一个controller-manager组件运行。**  
**3、scheduler k8s内部通过选举方式产生领导者(由–leader-elect 选型控制，默认为true)，同一时刻集群内只有一个scheduler组件运行。**  
**4、etcd 通过运行kubeadm方式自动创建集群来实现高可用，部署的节点数为奇数，3节点方式最多容忍一台机器宕机。**

--------------------

# 二、部署环境 #

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>系统版本</th> 
   <th>内核版本</th> 
   <th>IP地址</th> 
   <th>备注</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>VIP</td> 
   <td></td> 
   <td></td> 
   <td>192.168.1.234</td> 
   <td>VIP（（不是实际的主机））</td> 
  </tr> 
  <tr> 
   <td>lvs-keepalived-01</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.143</td> 
   <td>LVS+Keepalived</td> 
  </tr> 
  <tr> 
   <td>lvs-keepalived-02</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.144</td> 
   <td>LVS+Keepalived</td> 
  </tr> 
  <tr> 
   <td>k8s-master-149</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.149</td> 
   <td>master节点</td> 
  </tr> 
  <tr> 
   <td>k8s-master-155</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.155</td> 
   <td>master节点</td> 
  </tr> 
  <tr> 
   <td>k8s-master-160</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.160</td> 
   <td>master节点</td> 
  </tr> 
  <tr> 
   <td>k8s-worker-161</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.161</td> 
   <td>worker节点</td> 
  </tr> 
  <tr> 
   <td>k8s-worker-162</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.162</td> 
   <td>worker节点</td> 
  </tr> 
  <tr> 
   <td>k8s-worker-166</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.166</td> 
   <td>worker节点</td> 
  </tr> 
  <tr> 
   <td>k8s-client-170</td> 
   <td>centos7.6.1810</td> 
   <td>5.11.16</td> 
   <td>192.168.1.170</td> 
   <td>client节点</td> 
  </tr> 
 </tbody> 
</table>

**说明：共有9台服务器，2台为lvs-keepalived集群，3台master集群，3台work集群，1台client。**

--------------------

# 三、环境初始化 #

**说明：所有节点均需要升级内核。**

## 3.1、内核升级 ##

**说明：centos7.6系统内核默认是3.10.0，这里建议内核版本为5.11.16。**

内核选择：
    kernel-lt（lt=long-term）长期有效
    kernel-ml（ml=mainline） 主流版本

**升级步骤如下：**

# 1、下载内核
    CSDN下载地址：https://download.csdn.net/download/m0_37814112/17047556
    wget https://elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-ml-5.11.16-1.el7.elrepo.x86_64.rpm
    
    # 2、安装内核
    yum localinstall kernel-ml-5.11.16-1.el7.elrepo.x86_64.rpm -y
    
    # 3、查看当前内核版本
    grub2-editenv list
    
    # 4、查看所有内核启动grub2
    awk -F \' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
    
    # 5、修改为最新的内核启动
    grub2-set-default 'CentOS Linux (5.11.16-1.el7.elrepo.x86_64) 7 (Core)'
    
    # 6、再次查看内核版本
    grub2-editenv list
    
    # 7、reboot重启服务器（必须）

--------------------

## 3.2、系统初始化配置 ##

# 1、分别在相应的主机节点上修改主机名
    nginx1节点：  hostnamectl set-hostname lvs-keepalived-01
    nginx2节点：  hostnamectl set-hostname lvs-keepalived-02
    master1节点： hostnamectl set-hostname k8s-master-149
    master2节点： hostnamectl set-hostname k8s-master-155
    master3节点： hostnamectl set-hostname k8s-master-160
    worker1节点： hostnamectl set-hostname k8s-worker-161
    worker2节点： hostnamectl set-hostname k8s-worker-162
    worker3节点： hostnamectl set-hostname k8s-worker-166
    client节点：  hostnamectl set-hostname k8s-client-170
    
    # 2、配置hosts解析（master节点和worker节点）
    cat >> /etc/hosts << EOF 192.168.1.149 k8s-master-149 192.168.1.155 k8s-master-155 192.168.1.160 k8s-master-160 192.168.1.161 k8s-worker-161 192.168.1.162 k8s-worker-162 192.168.1.166 k8s-worker-166 EOF
    
    # 2、内核参数修改（所有节点）
    cat > /etc/sysctl.d/k8s.conf <<EOF vm.swappiness = 0 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-arptables = 1 EOF
    sysctl --system 
    
    # 3、永久新增br_netfilter模块（所有节点）
    # k8s网络使用flannel，该网络需要设置内核参数bridge-nf-call-iptables=1，修改这个参数需要系统有br_netfilter模块。
    cat > /etc/rc.sysinit << EOF #!/bin/bash for file in /etc/sysconfig/modules/*.modules ; do [ -x $file ] && $file done EOF
    
    cat > /etc/sysconfig/modules/br_netfilter.modules << EOF modprobe br_netfilter EOF
    
    chmod 755 /etc/sysconfig/modules/br_netfilter.modules
    
    # 4、关闭防火墙（所有节点）
    systemctl stop firewalld && systemctl disable firewalld
    
    # 5、永久关闭selinux（所有节点）
    sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
    sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
    
    # 6、关闭swap（所有节点）
    swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab
    
    # 7、安装ipvsadm（所有节点）
    yum install ipvsadm -y
    cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack" for kernel_module in \${ipvs_modules}; do /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1 if [ $? -eq 0 ]; then /sbin/modprobe \${kernel_module} fi done EOF
    chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep --color ip_vs >/dev/null
    lsmod | grep ip_vs
    
    # 8、修改文件描述符和进程数（根据实际情况修改）（所有节点）
    vim /etc/security/limits.conf
    root soft nofile 65535
    root hard nofile 65535
    root soft nproc 65535
    root hard nproc 65535
    root soft memlock unlimited
    root hard memlock unlimited
    * soft nofile 65535
    * hard nofile 65535
    * soft nproc 65535
    * hard nproc 65535
    * soft memlock unlimited
    * hard memlock unlimited
    
    # 9、reboot重启服务器（必须）（所有节点）

**如下图所示：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3ODE0MTEy_size_16_color_FFFFFF_t_70 1]

--------------------

# 四、ssh双机互信 #

# 1、配置k8s-master-149到k8s-master-155、k8s-master-160免密登录，本步骤只在k8s-master-149上执行。
    # 用法如下
    [root@k8s-master-149 ~]# ./ssh_key.sh 
    usage: ./ssh_key.sh user host password ssh-pub-key ssh-key port
    usage: ./ssh_key.sh root host131 pass123 /root/.ssh/id_rsa.pub /root/.ssh/id_rsa 22
    # 操作如下
    yum install expect -y
    ssh-keygen -t rsa
    ./ssh_key.sh root 192.168.1.155 123456 /root/.ssh/id_rsa.pub /root/.ssh/id_rsa 22
    ./ssh_key.sh root 192.168.1.160 123456 /root/.ssh/id_rsa.pub /root/.ssh/id_rsa 22
    
    # 验证
    ssh root@192.168.1.155
    ssh root@192.168.1.160
    
    # 2、脚本内容如下
    vim ssh_key.sh
    
    #!/bin/sh
    #####################################################
    # Function: config ssh
    #
    # Params: 1: user: user name for ssh login
    # 2: host: host name for ssh login
    # 3: password: password for ssh-copy-id
    # 4: sshpubkey: ssh publick key
    # 5: sshkey: ssh key
    # 6: sshport: ssh port
    #
    # Return:
    # 0: ssh config succussfully
    # 1: parameter number check failed
    # 2: ssh public key does not exist
    # 3: ssh key does not exist
    # 4: command(ssh/ssh-copy-id/expect) not found
    # 5: ssh config check failed
    #
    #####################################################
    PARAM_USER=$1         #被连接主机用户
    PARAM_HOST=$2         #被连接主机IP地址
    PARAM_PASSWORD=$3     #被连接主机用户密码 
    PARAM_SSH_KEY_PUB=$4  
    PARAM_SSH_KEY=$5
    PARAM_SSH_PORT=$6
    
    #RETURN STATE
    RET_SSH_CONNECTED=0
    RET_SSH_PARAM_FAILED=1
    RET_SSH_PUBKEY_FAILED=2
    RET_SSH_KEY_FAILED=3
    RET_CMD_NOTFOUND=4
    RET_SSH_CHECK_FAILED=5
    
    #DISPLAY_MODE
    DISPLAY_MODE_DEBUG="DEBUG"
    DISPLAY_MODE_QUIET="QUIET"
    
    #SSH CONFIG TIMEOUT
    EXPECT_TIMEOUT=30
    
    var_display_mode=${DISPLAY_MODE_DEBUG}
    var_expect_timeout=${EXPECT_TIMEOUT}
    
    # display output message
    print_message(){ 
      if [ _"${var_display_mode}" != _"${DISPLAY_MODE_QUIET}" ]; then
        echo $*
      fi
    }
    
    # function usage 
    usage(){ 
      print_message "usage: $0 user host password ssh-pub-key ssh-key port "
      print_message " usage: $0 root host131 pass123 /root/.ssh/id_rsa.pub /root/.ssh/id_rsa 22"
      print_message ""
    }
    
    # check command existence
    check_command_existence(){ 
      command_name="$1"
    
      which "${command_name}" >/dev/null 2>&1
      if [ $? -ne 0 ]; then
        print_message "command: ${command_name} does not exist"
        exit ${RET_CMD_NOTFOUND}
      fi
    }
    
    # check ssh connection
    check_ssh_connection(){ 
      var_ssh_user="$1"
      var_ssh_host="$2"
    
      check_command_existence ssh
    
      # check connection
      ssh -p ${PARAM_SSH_PORT} ${var_ssh_user}@${var_ssh_host} hostname >/dev/null 2>&1
      if [ $? -eq 0 ]; then
        print_message "ssh connection to [$var_ssh_host] with user [$var_ssh_user] success configured."
        exit ${RET_SSH_CONNECTED}
      else
        print_message "ssh connection to [$var_ssh_host] with user [$var_ssh_user] failed."
        exit ${RET_SSH_CHECK_FAILED}
      fi
    }
    
    # config ssh by using ssh-copy-id & expect
    config_ssh(){ 
      var_user="$1"
      var_host="$2"
      var_password="$3"
      var_key_pub="$4"
      var_key="$5"
      var_port="$6"
    
      check_command_existence ssh-copy-id
      check_command_existence expect
    
      expect <<-EOF set time ${var_expect_timeout} spawn ssh-copy-id -p ${var_port} -i ${var_key_pub} -o "IdentityFile ${var_key}" ${var_user}@${var_host} expect { "*yes/no" { send "yes\r"; exp_continue } "*password:" { send "${var_password}\r" } } expect eof EOF
    }
    
    # check parameters
    if [ $# -ne 6 ]; then
      usage
      exit ${RET_SSH_PARAM_FAILED}
    fi
    
    if [ ! -f "${PARAM_SSH_KEY_PUB}" ]; then
      print_message "ssh public key does not exist: ${PARAM_SSH_KEY_PUB}"
      usage
      exit ${RET_SSH_PUBKEY_FAILED}
    fi
    
    if [ ! -f "${PARAM_SSH_KEY}" ]; then
      print_message "ssh key does not exist: ${PARAM_SSH_PUB}"
      usage
      exit ${RET_SSH_KEY_FAILED}
    fi
    
    # config ssh
    config_ssh ${PARAM_USER} ${PARAM_HOST} ${PARAM_PASSWORD} ${PARAM_SSH_KEY_PUB} ${PARAM_SSH_KEY} ${PARAM_SSH_PORT}
    
    # check connection
    check_ssh_connection ${PARAM_USER} ${PARAM_HOST}

--------------------

# 五、Docker部署 #

**说明：以下操作kubernetes集群master节点和worker节点均需要执行。**

# 1、安装依赖包
    yum install yum-utils device-mapper-persistent-data lvm2 -y 
    
    # 2、设置Docker源（建议使用阿里云docker的yum源）
    yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    或
    yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
    
    # 3、docker安装版本查看
    yum list docker-ce --showduplicates | sort -r
    
    # 4、指定安装的docker版本为18.09.9
    yum install docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io -y
    
    # 5、启动Docker
    systemctl start docker && systemctl enable docker
    
    # 6、修改Cgroup Driver
    # 修改docker数据目录，默认是/var/lib/docker,建议数据目录为主机上最大磁盘空间目录或子目录
    # 配置阿里云加速器
    
    vim /etc/docker/daemon.json
    { 
      "registry-mirrors": ["https://v16stybc.mirror.aliyuncs.com"],
      "exec-opts": ["native.cgroupdriver=systemd"],
      "graph": "/data/docker"  
    }
    systemctl daemon-reload && systemctl restart docker

**如下图所示：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3ODE0MTEy_size_16_color_FFFFFF_t_70 2]

--------------------

# 六、kubernetes部署 #

## 6.1、设置kubernetes源 ##

**说明：以下操作无论是master节点和worker节点均需要执行。**

cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF
    
    yum clean all && yum -y makecache

--------------------

## 6.2、安装kubelet、kubeadm和kubectl ##

**说明：以下操作无论是master节点和worker节点均需要执行。**

# 1、版本查看
    yum list kubelet --showduplicates | sort -r
    
    # 2、安装1.17.4版本
    yum install kubelet-1.17.4 kubeadm-1.17.4 kubectl-1.17.4 -y

--------------------

## 6.3、k8s相关镜像下载 ##

**说明：以下操作无论是master节点和worker节点均需要执行。**

# k8s镜像下载
    vim get_image.sh 
    #!/bin/bash
    url=registry.cn-hangzhou.aliyuncs.com/google_containers
    version=v1.17.4
    images=(`kubeadm config images list --kubernetes-version=$version|awk -F '/' '{print $2}'`)
    for imagename in ${images[@]} ; do
      docker pull $url/$imagename
      docker tag $url/$imagename k8s.gcr.io/$imagename
      docker rmi -f $url/$imagename
    done
    
    # flannel网络镜像下载
    docker pull quay.io/coreos/flannel:v0.14.0-rc1

**如下图所示：**  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L20wXzM3ODE0MTEy_size_16_color_FFFFFF_t_70 3]  
**说明：  
1、以上关于k8s镜像下载，无论master节点还是worker节点均需要下载。**  
**2、flannel镜像CSDN下载地址：[flannel:v0.14.0-rc1][flannel_v0.14.0-rc1]**

--------------------

## 6.4、kubernetes master节点部署 ##

### 6.4.1、创建初始化配置文件 ###