Kubeadm部署基于Calico K8s集群

蔚落 2022-01-15 23:43 430阅读 0赞

[2019独角兽企业重金招聘Python工程师标准>>> ][2019_Python_] ![hot3.png][]

1 环境

<table> 
 <tbody> 
  <tr> 
   <th>&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;Host Name&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th> 
   <th>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Role&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th> 
   <th>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;IP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th> 
  </tr> 
 </tbody> 
 <tbody> 
  <tr> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;master1</td> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;k8s-1001</td> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;172.31.135.239</td> 
  </tr> 
  <tr> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;node1</td> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;k8s-1002</td> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; 172.31.135.238</td> 
  </tr> 
  <tr> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;node2</td> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;k8s-1003</td> 
   <td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;172.31.135.237</td> 
  </tr> 
 </tbody> 
</table>

2 内核调优

cat > /etc/sysctl.d/k8s.conf <<EOF
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    net.ipv4.ip_forward = 1
    vm.swappiness=0
    EOF
    
    modprobe br_netfilter
    sysctl -p /etc/sysctl.d/k8s.conf

3 修改文件描述符

echo "* soft nofile 65536" >> /etc/security/limits.conf
    echo "* hard nofile 65536" >> /etc/security/limits.conf
    echo "* soft nproc 65536"  >> /etc/security/limits.conf
    echo "* hard nproc 65536"  >> /etc/security/limits.conf
    echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
    echo "* hard memlock  unlimited"  >> /etc/security/limits.conf
    
    
    hard limits自AIX 4.1版本开始引入。hard limits 应由AIX系统管理员设置，只有security组的成员可以将此值增大，用户本身可以减小此限定值,但是其更改将随着该用户从系统退出而失效
    
    soft limits 是AIX核心使用的限制进程对系统资源的使用的上限值。此值可由任何人更改，但不能超出hard limits值。这里要注意的是只有security组的成员可使更改永久生效普通用户的更改在其退出系统后将失效
    
    1)soft nofile和hard nofile示，单个用用户的软限制为1000，硬限制为1200，即表示单用户能打开的最大文件数量为1000，不管它开启多少个shell。
    
    2）soft nproc和hard nproc 单个用户可用的最大进程数量,软限制和硬限制
    
    3）memlock 一个任务锁住的物理内存的最大值（这里设置成无限制）

4  配置k8s yum 源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
    EOF

5 配置docker yum源

cd /etc/yum.repos.d
    wget https://download.docker.com/linux/centos/docker-ce.repo

6 时间同步

一个集群内的时间同步必不可少

systemctl enable ntpdate.service
    echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
    crontab /tmp/crontab2.tmp
    systemctl start ntpdate.service
    
    ntpdate -u ntp.api.bz

7  关闭SELinux、防火墙

systemctl stop firewalld
    systemctl disable firewalld
    setenforce 0
    sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

8 关闭系统的Swap

swapoff -a
    yes | cp /etc/fstab /etc/fstab_bak
    cat /etc/fstab_bak |grep -v swap > /etc/fstab

9 配置hosts 解析

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    172.31.135.239	k8s-1001	 k8s-1001
    172.31.135.237  k8s-1003     k8s-1003 
    172.31.135.238  k8s-1002     k8s-1002

10 配置节点免密登录

ssh-keygen
    ssh-copy-id -i ~/.ssh/id_rsa.pub  用户名字@192.168.x.xxx

11 安装依赖等

yum install -y epel-release
    yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl lrzsz wget

12 配置ipvs模块

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
    #!/bin/bash
    modprobe -- ip_vs
    modprobe -- ip_vs_rr
    modprobe -- ip_vs_wrr
    modprobe -- ip_vs_sh
    modprobe -- nf_conntrack_ipv4
    EOF
    chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
    
    
    
    
    yum install ipset ipvsadm -y

13 安装docker

yum list docker-ce --showduplicates | sort -r 
    
    yum install -y docker-ce-18.06.1.ce-3.el7
    
    最号先不要安装最新版本
    systemctl daemon-reload
    systemctl enable docker
    systemctl start docker

14 master 和node 节点安装 kubelet kubeadm kubectl

yum install -y kubelet kubeadm kubectl  master
    systemctl enable kubelet
    暂不启动 kubelet

15 如果在国内请按照下面步骤进行

生成默认配置

kubeadm config print init-defaults > /root/kubeadm.conf

修改 /root/kubeadm.conf，使用国内阿里的imageRepository: registry.aliyuncs.com/google\_containers

下载镜像

kubeadm config images pull --config /root/kubeadm.conf

16 如过网络允许可以直接初始化集群

kubeadm init --kubernetes-version=v1.14.1 --pod-network-cidr=10.244.0.0/16
    
    
    这里没有设置--service-cidr 是因为我们下面需要部署calico网络，calico会帮助我们设置service网络，如果此地设置了service网络会导致calico部署不成功

保存这段内容

mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
        --discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1

16 创建相关文件夹

mkdir -p $HOME/.kube
     cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

18 如果以后还有机器加入集群如何获取token 和 hash值？

获得token和hash值
    1）获取token
    kubeadm token list
    默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Token
    kubeadm token create
    
    2）获取hash值
    openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

19 验证

kubectl get pods --all-namespaces
    
    NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
    kube-system   coredns-fb8b8dccf-pdf9r            0/1     Pending   0          6m50s
    kube-system   coredns-fb8b8dccf-rngcz            0/1     Pending   0          6m50s
    kube-system   etcd-k8s-1001                      1/1     Running   0          5m52s
    kube-system   kube-apiserver-k8s-1001            1/1     Running   0          6m4s
    kube-system   kube-controller-manager-k8s-1001   1/1     Running   0          5m51s
    kube-system   kube-proxy-b8dhg                   1/1     Running   0          6m50s
    kube-system   kube-scheduler-k8s-1001            1/1     Running   0          5m47s
    
    coredns 是pending状态 先不用管它，因为这个没有网络插件的导致的

20 部署calico网络插件

官方文档

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
    
    
    kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml

我们需要修改calico文件

https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
    
    vim calico.yaml
    
    1）修改ipip模式关闭 和typha_service_name
    
    - name: CALICO_IPV4POOL_IPIP
    value: "off"
    
    
    typha_service_name: "calico-typha"
    
    
    
    
    calico网络，默认是ipip模式（在每台node主机创建一个tunl0网口，这个隧道链接所有的node容器网络，官网推荐不同的ip网段适合，比如aws的不同区域主机），
    
    修改成BGP模式，它会以daemonset方式安装在所有node主机，每台主机启动一个bird(BGP client)，它会将calico网络内的所有node分配的ip段告知集群内的主机，并通过本机的网卡eth0或者ens33转发数据；
    
    2）修改replicas
    
      replicas: 1
      revisionHistoryLimit: 2
    
    3）修改pod的网段CALICO_IPV4POOL_CIDR
    
    - name: CALICO_IPV4POOL_CIDR
    value: "10.244.0.0/16"
    4）如果手动下载镜像请查看calico.yaml 文件里面标注的镜像版本 否则可以直接执行会自动下载
    5）部署calico
    kubectl apply -f calico.yaml
    
    6）查看
    kubectl get po --all-namespaces
    此时你会发现是pending状态是因为node节点还没有相关组件
    7） 验证是否为bgp模式
    # ip route show
    default via 172.31.143.253 dev eth0 
    blackhole 10.244.0.0/24 proto bird 
    10.244.0.2 dev caliac6de7553e8 scope link 
    10.244.0.3 dev cali1591fcccf0f scope link 
    10.244.1.0/24 via 172.31.135.237 dev eth0 proto bird 
    10.244.2.0/24 via 172.31.135.238 dev eth0 proto bird 
    169.254.0.0/16 dev eth0 scope link metric 1002 
    172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
    172.31.128.0/20 dev eth0 proto kernel scope link src 172.31.135.239

21 将node加入到节点

kubeadm join 172.31.135.239:6443 --token ljzfdh.5qccrqv482klk96h \
        --discovery-token-ca-cert-hash sha256:dc65895e08a9c0f531943940b44f6ef144dd3a7e5f76973758927a6e107281a1 
    
    
    master上查看集群
    [root@k8s-1001 ~]# kubectl get node
    NAME       STATUS   ROLES    AGE    VERSION
    k8s-1001   Ready    master   37m    v1.14.1
    k8s-1002   Ready    <none>   99s    v1.14.1
    k8s-1003   Ready    <none>   115s   v1.14.1
    [root@k8s-1001 ~]# kubectl get node -o wide
    NAME       STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
    k8s-1001   Ready    master   37m    v1.14.1   172.31.135.239   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
    k8s-1002   Ready    <none>   103s   v1.14.1   172.31.135.238   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
    k8s-1003   Ready    <none>   119s   v1.14.1   172.31.135.237   <none>        CentOS Linux 7 (Core)   3.10.0-862.14.4.el7.x86_64   docker://18.6.1
    
    [root@k8s-1001 ~]# kubectl get pod -o wide -n kube-system
    NAME                               READY   STATUS    RESTARTS   AGE     IP               NODE       NOMINATED NODE   READINESS GATES
    calico-node-8z92v                  2/2     Running   0          2m1s    172.31.135.238   k8s-1002   <none>           <none>
    calico-node-k542k                  2/2     Running   0          7m32s   172.31.135.239   k8s-1001   <none>           <none>
    calico-node-n4jgf                  2/2     Running   0          2m17s   172.31.135.237   k8s-1003   <none>           <none>
    calico-typha-55968bfd7b-c5r4z      1/1     Running   0          7m33s   172.31.135.237   k8s-1003   <none>           <none>
    coredns-fb8b8dccf-pdf9r            1/1     Running   0          37m     10.244.0.3       k8s-1001   <none>           <none>
    coredns-fb8b8dccf-rngcz            1/1     Running   0          37m     10.244.0.2       k8s-1001   <none>           <none>
    etcd-k8s-1001                      1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
    kube-apiserver-k8s-1001            1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
    kube-controller-manager-k8s-1001   1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>
    kube-proxy-b8dhg                   1/1     Running   0          37m     172.31.135.239   k8s-1001   <none>           <none>
    kube-proxy-nvlmz                   1/1     Running   0          2m17s   172.31.135.237   k8s-1003   <none>           <none>
    kube-proxy-rfb77                   1/1     Running   0          2m1s    172.31.135.238   k8s-1002   <none>           <none>
    kube-scheduler-k8s-1001            1/1     Running   0          36m     172.31.135.239   k8s-1001   <none>           <none>

22 开启kube-proxy  lvs

修改ConfigMap的kube-system/kube-proxy中的config.conf，`mode: "ipvs"`：
    
    kubectl edit cm kube-proxy -n kube-system
     kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'
    
    # ipvsadm -L -n
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  10.96.0.1:443 rr
      -> 172.31.135.239:6443          Masq    1      0          0         
    TCP  10.96.0.10:53 rr
      -> 10.244.0.2:53                Masq    1      0          0         
      -> 10.244.0.3:53                Masq    1      0          0         
    TCP  10.96.0.10:9153 rr
      -> 10.244.0.2:9153              Masq    1      0          0         
      -> 10.244.0.3:9153              Masq    1      0          0         
    TCP  10.111.3.127:5473 rr
      -> 172.31.135.237:5473          Masq    1      0          0         
    UDP  10.96.0.10:53 rr
      -> 10.244.0.2:53                Masq    1      0          0         
      -> 10.244.0.3:53                Masq    1      0          0

23 master默认是污点，pod是不会调度到master

1）开放master可被调度
    kubectl taint node k8s-1001 node-role.kubernetes.io/master-
    2）
    如果要恢复 Master Only 状态
    kubectl taint node k8s-1001 node-role.kubernetes.io/master="":NoSchedule

24 创建一个测试pod 验证集群

kubectl run net-test --image=alpine --replicas=2 sleep 360
    
    # kubectl get pod -o wide 
    NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
    net-test-7d6d58cc8-78wd6   1/1     Running   0          6s    10.244.1.2   k8s-1003   <none>           <none>
    net-test-7d6d58cc8-hjdhw   1/1     Running   0          6s    10.244.2.2   k8s-1002   <none>           <none>

转载于:https://my.oschina.net/54188zz/blog/3042397

[2019_Python_]: https://my.oschina.net/u/2663968/blog/3061667
[hot3.png]: /images/20220115/003c1473896f45508b7066b06825c92c.png