二进制包20分钟快速安装部署 Kubernetes v1.14.0 集群

落日映苍穹つ 2022-02-13 03:41 119阅读 0赞

# 二进制包20分钟快速部署 Kubernetes v1.14.0 集群 #

## 一 环境 ##

<table> 
 <thead> 
  <tr> 
   <th>操作系统</th> 
   <th>Docker版本</th> 
   <th>Kubernetes版本</th> 
   <th>Etcd版本</th> 
   <th>Flannel版本</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="center">CentOS Linux release 7.6.1810</td> 
   <td align="center">Docker version 18.09.4</td> 
   <td align="center">v1.14.0</td> 
   <td align="center">Version: 3.3.12</td> 
   <td align="center">v0.11.0</td> 
  </tr> 
 </tbody> 
</table>

## 二 架构 ##

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>IP</th> 
   <th>角色</th> 
   <th>部署应用</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="center">gysl-master</td> 
   <td align="center">10.1.1.60</td> 
   <td align="center">Msater</td> 
   <td align="center">Docker/Kube-apiserver/kube-scheduler/kube-controller-manager/etcd</td> 
  </tr> 
  <tr> 
   <td align="center">gysl-node1</td> 
   <td align="center">10.1.1.61</td> 
   <td align="center">Node</td> 
   <td align="center">Docker/Kubelet/kube-proxy/flanneld/etcd</td> 
  </tr> 
  <tr> 
   <td align="center">gysl-node2</td> 
   <td align="center">10.1.1.62</td> 
   <td align="center">Node</td> 
   <td align="center">Docker/Kubelet/kube-proxy/flanneld/etcd</td> 
  </tr> 
  <tr> 
   <td align="center">gysl-node3</td> 
   <td align="center">10.1.1.63</td> 
   <td align="center">Node</td> 
   <td align="center">Docker/Kubelet/kube-proxy/flanneld/etcd</td> 
  </tr> 
 </tbody> 
</table>

## 三 安装过程 ##

通过几个小时的努力，完成本次部署脚本的编写，安装脚本支持任意多个节点，主要通过三个脚本实现本次安装。

### 3.1 初始化脚本 ###

#!/bin/bash
    declare -A HostIP EtcdIP
    HostIP=( [gysl-master]='10.1.1.60' [gysl-node1]='10.1.1.61' [gysl-node2]='10.1.1.62' [gysl-node3]='10.1.1.63' )
    EtcdIP=( [etcd-master]='10.1.1.60' [etcd-01]='10.1.1.61' [etcd-02]='10.1.1.62' [etcd-03]='10.1.1.63' )
    BinaryDir='/usr/local/bin'
    KubeConf='/etc/kubernetes/conf.d'
    KubeCA='/etc/kubernetes/ca.d'
    EtcdConf='/etc/etcd/conf.d'
    EtcdCA='/etc/etcd/ca.d'
    FlanneldConf='/etc/flanneld'
    
    mkdir -p {${KubeConf},${KubeCA},${EtcdConf},${EtcdCA},${FlanneldConf}}
    for hostname in ${!HostIP[@]}
        do
            cat>>/etc/hosts<<EOF
    ${HostIP[${hostname}]} ${hostname}
    EOF
        done
    # Install the Docker engine. This needs to be executed on every machine.
    curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo>&/dev/null
    if [ $? -eq 0 ] ;
        then
            yum remove docker \
                          docker-client \
                          docker-client-latest \
                          docker-common \
                          docker-latest \
                          docker-latest-logrotate \
                          docker-logrotate \
                          docker-selinux \
                          docker-engine-selinux \
                          docker-engine>&/dev/null
            yum list docker-ce --showduplicates|grep "^doc"|sort -r
            yum -y install docker-ce-18.09.3-3.el7
            rm -f /etc/yum.repos.d/docker-ce.repo
            systemctl enable docker --now && systemctl status docker
        else
            echo "Install failed! Please try again! ";
            exit 110
    fi
    # Modify related kernel parameters. 
    cat>/etc/sysctl.d/docker.conf<<EOF
    net.ipv4.ip_forward = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    EOF 
    sysctl -p /etc/sysctl.d/docker.conf>&/dev/null 
    # Turn off and disable the firewalld.  
    systemctl stop firewalld  
    systemctl disable firewalld  
    # Disable the SELinux.  
    sed -i.bak 's/=enforcing/=disabled/' /etc/selinux/config  
    # Disable the swap.  
    sed -i.bak 's/^.*swap/#&/g' /etc/fstab
    # Install EPEL/vim/git.  
    yum -y install epel-release vim git tree
    yum repolist
    # Alias vim. 
    cat>/etc/profile.d/vim.sh<<EOF
    alias vi='vim'
    EOF
    source /etc/profile.d/vim.sh
    echo "set nu">>/etc/vimrc
    # Reboot the machine.  
    reboot

需要每个节点都执行。

### 3.2 安装脚本 ###

安装脚本较长，此处省略，日志以供参考，拓展思路。此脚本在Master节点执行即可，安装过程无需连接外网，安装日志如下：

Generating public/private rsa key pair.
    Enter file in which to save the key (/root/.ssh/id_rsa): 
    Created directory '/root/.ssh'.
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /root/.ssh/id_rsa.
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    The key fingerprint is:
    SHA256:rJdnEzx5GyWX9YCxq77ZMc+FCabCqA+3FwmS7LnF9qo Kubernetes
    The key's randomart image is:
    +---[RSA 1024]----+
    |            .o. .|
    |            .. +.|
    |   . .      o + .|
    |    + .. . . =   |
    |   . + .S.= *    |
    |    o ++o. B + o |
    |    .++.=.* + o .|
    |    .+ oo= + = . |
    |    Eo+o  +.. o  |
    +----[SHA256]-----+
    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
    The authenticity of host '10.1.1.62 (10.1.1.62)' can't be established.
    ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
    ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
    Are you sure you want to continue connecting (yes/no)? yes
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    root@10.1.1.62's password: 
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh 'root@10.1.1.62'"
    and check to make sure that only the key(s) you wanted were added.
    
    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
    The authenticity of host '10.1.1.63 (10.1.1.63)' can't be established.
    ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
    ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
    Are you sure you want to continue connecting (yes/no)? yes
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    root@10.1.1.63's password: 
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh 'root@10.1.1.63'"
    and check to make sure that only the key(s) you wanted were added.
    
    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
    The authenticity of host '10.1.1.61 (10.1.1.61)' can't be established.
    ECDSA key fingerprint is SHA256:B4e7Gq9wcgr5N6ys8U72NEhNWxIFrvng5eI7GAXLf6s.
    ECDSA key fingerprint is MD5:ea:33:04:40:f8:31:a2:d0:91:c4:b4:37:48:fa:51:d6.
    Are you sure you want to continue connecting (yes/no)? yes
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    root@10.1.1.61's password: 
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh 'root@10.1.1.61'"
    and check to make sure that only the key(s) you wanted were added.
    
    etcd-v3.3.12-linux-amd64/
    etcd-v3.3.12-linux-amd64/README.md
    etcd-v3.3.12-linux-amd64/Documentation/
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/interacting_v3.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_concurrency_reference_v3.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/limit.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/local_cluster.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_grpc_gateway.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/grpc_naming.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/rpc.swagger.json
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/v3lock.swagger.json
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/apispec/swagger/v3election.swagger.json
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/experimental_apis.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-guide/api_reference_v3.md
    etcd-v3.3.12-linux-amd64/Documentation/integrations.md
    etcd-v3.3.12-linux-amd64/Documentation/README.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-1-0-alpha-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-storage-memory-benchmark.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/README.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-rc-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-rc-memory-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-3-watch-memory-benchmark.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-2-2-0-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/benchmarks/etcd-3-demo-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/rfc/
    etcd-v3.3.12-linux-amd64/Documentation/rfc/v3api.md
    etcd-v3.3.12-linux-amd64/Documentation/docs.md
    etcd-v3.3.12-linux-amd64/Documentation/production-users.md
    etcd-v3.3.12-linux-amd64/Documentation/metrics.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/
    etcd-v3.3.12-linux-amd64/Documentation/v2/authentication.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/proxy.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/glossary.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/docker_guide.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/configuration.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/README.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-1-0-alpha-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-storage-memory-benchmark.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/README.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-rc-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-rc-memory-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-3-watch-memory-benchmark.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-2-2-0-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/benchmarks/etcd-3-demo-benchmarks.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/rfc/
    etcd-v3.3.12-linux-amd64/Documentation/v2/rfc/v3api.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/libraries-and-tools.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/discovery_protocol.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/runtime-configuration.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_3.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/auth_api.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/errorcode.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/admin_guide.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_2.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/upgrade_2_1.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/clustering.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/other_apis.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/production-users.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/metrics.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/runtime-reconf-design.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/etcd_alert.rules.yml
    etcd-v3.3.12-linux-amd64/Documentation/v2/security.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/branch_management.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/internal-protocol-versioning.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/members_api.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/platforms/
    etcd-v3.3.12-linux-amd64/Documentation/v2/platforms/freebsd.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/faq.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/backward_compatibility.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/04_to_2_snapshot_migration.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/etcd_alert.rules
    etcd-v3.3.12-linux-amd64/Documentation/v2/api.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/api_v3.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/reporting_bugs.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/tuning.md
    etcd-v3.3.12-linux-amd64/Documentation/v2/dev/
    etcd-v3.3.12-linux-amd64/Documentation/v2/dev/release.md
    etcd-v3.3.12-linux-amd64/Documentation/branch_management.md
    etcd-v3.3.12-linux-amd64/Documentation/platforms/
    etcd-v3.3.12-linux-amd64/Documentation/platforms/container-linux-systemd.md
    etcd-v3.3.12-linux-amd64/Documentation/platforms/freebsd.md
    etcd-v3.3.12-linux-amd64/Documentation/platforms/aws.md
    etcd-v3.3.12-linux-amd64/Documentation/faq.md
    etcd-v3.3.12-linux-amd64/Documentation/dl_build.md
    etcd-v3.3.12-linux-amd64/Documentation/reporting_bugs.md
    etcd-v3.3.12-linux-amd64/Documentation/tuning.md
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_4.md
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrading-etcd.md
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_2.md
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_1.md
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_0.md
    etcd-v3.3.12-linux-amd64/Documentation/upgrades/upgrade_3_3.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-internal/
    etcd-v3.3.12-linux-amd64/Documentation/dev-internal/logging.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-internal/discovery_protocol.md
    etcd-v3.3.12-linux-amd64/Documentation/dev-internal/release.md
    etcd-v3.3.12-linux-amd64/Documentation/learning/
    etcd-v3.3.12-linux-amd64/Documentation/learning/auth_design.md
    etcd-v3.3.12-linux-amd64/Documentation/learning/glossary.md
    etcd-v3.3.12-linux-amd64/Documentation/learning/data_model.md
    etcd-v3.3.12-linux-amd64/Documentation/learning/api_guarantees.md
    etcd-v3.3.12-linux-amd64/Documentation/learning/why.md
    etcd-v3.3.12-linux-amd64/Documentation/learning/api.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/authentication.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/versioning.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/hardware.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/grafana.json
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/monitoring.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/configuration.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/v2-migration.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/maintenance.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/runtime-configuration.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/recovery.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/clustering.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd3_alert.rules
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/runtime-reconf-design.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd3_alert.rules.yml
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/security.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/performance.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/etcd-sample-grafana.png
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/grpc_proxy.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/gateway.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/container.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/supported-platform.md
    etcd-v3.3.12-linux-amd64/Documentation/op-guide/failures.md
    etcd-v3.3.12-linux-amd64/Documentation/demo.md
    etcd-v3.3.12-linux-amd64/README-etcdctl.md
    etcd-v3.3.12-linux-amd64/etcdctl
    etcd-v3.3.12-linux-amd64/READMEv2-etcdctl.md
    etcd-v3.3.12-linux-amd64/etcd
    flanneld
    mk-docker-opts.sh
    README.md
    kubernetes/
    kubernetes/server/
    kubernetes/server/bin/
    kubernetes/server/bin/kube-controller-manager.docker_tag
    kubernetes/server/bin/kube-apiserver.tar
    kubernetes/server/bin/kube-proxy
    kubernetes/server/bin/cloud-controller-manager.docker_tag
    kubernetes/server/bin/mounter
    kubernetes/server/bin/kube-proxy.docker_tag
    kubernetes/server/bin/kubelet
    kubernetes/server/bin/kube-scheduler.docker_tag
    kubernetes/server/bin/kube-controller-manager.tar
    kubernetes/server/bin/kubectl
    kubernetes/server/bin/kube-apiserver
    kubernetes/server/bin/kube-apiserver.docker_tag
    kubernetes/server/bin/kube-controller-manager
    kubernetes/server/bin/kube-proxy.tar
    kubernetes/server/bin/cloud-controller-manager
    kubernetes/server/bin/kube-scheduler.tar
    kubernetes/server/bin/apiextensions-apiserver
    kubernetes/server/bin/kubeadm
    kubernetes/server/bin/hyperkube
    kubernetes/server/bin/kube-scheduler
    kubernetes/server/bin/cloud-controller-manager.tar
    kubernetes/addons/
    kubernetes/kubernetes-src.tar.gz
    kubernetes/LICENSES
    2019/03/31 20:34:23 [INFO] generating a new CA key and certificate from CSR
    2019/03/31 20:34:23 [INFO] generate received request
    2019/03/31 20:34:23 [INFO] received CSR
    2019/03/31 20:34:23 [INFO] generating key: rsa-2048
    2019/03/31 20:34:23 [INFO] encoded CSR
    2019/03/31 20:34:23 [INFO] signed certificate with serial number 316253512009054883826466120107550244311105093255
    2019/03/31 20:34:23 [INFO] generate received request
    2019/03/31 20:34:23 [INFO] received CSR
    2019/03/31 20:34:23 [INFO] generating key: rsa-2048
    2019/03/31 20:34:23 [INFO] encoded CSR
    2019/03/31 20:34:23 [INFO] signed certificate with serial number 288189004496636237074496723049170901716100041831
    2019/03/31 20:34:23 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
    websites. For more information see the Baseline Requirements for the Issuance and Management
    of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
    specifically, section 10.2.3 ("Information Requirements").
    /etc/etcd/ca.d
    ├── ca-config.json
    ├── ca.csr
    ├── ca-csr.json
    ├── ca-key.pem
    ├── ca.pem
    ├── server.csr
    ├── server-csr.json
    ├── server-key.pem
    └── server.pem
    
    0 directories, 9 files
    ca-key.pem                                                                                                                                                 100% 1675    27.1KB/s   00:00    
    ca.pem                                                                                                                                                     100% 1265     2.2MB/s   00:00    
    server-key.pem                                                                                                                                             100% 1679   639.6KB/s   00:00    
    server.pem                                                                                                                                                 100% 1346     1.9MB/s   00:00    
    etcd                                                                                                                                                       100%   18MB  13.1MB/s   00:01    
    etcdctl                                                                                                                                                    100%   15MB  41.5MB/s   00:00    
    etcd.service                                                                                                                                               100%  994     1.0MB/s   00:00    
    etcd.conf                                                                                                                                                  100%  520   527.8KB/s   00:00    
    Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
    Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
    ca-key.pem                                                                                                                                                 100% 1675    92.3KB/s   00:00    
    ca.pem                                                                                                                                                     100% 1265     1.7MB/s   00:00    
    server-key.pem                                                                                                                                             100% 1679   328.8KB/s   00:00    
    server.pem                                                                                                                                                 100% 1346     1.6MB/s   00:00    
    etcd                                                                                                                                                       100%   18MB  40.7MB/s   00:00    
    etcdctl                                                                                                                                                    100%   15MB  46.0MB/s   00:00    
    etcd.service                                                                                                                                               100%  994     1.0MB/s   00:00    
    etcd.conf                                                                                                                                                  100%  520   838.6KB/s   00:00    
    Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
    Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.
    ca-key.pem                                                                                                                                                 100% 1675   106.9KB/s   00:00    
    ca.pem                                                                                                                                                     100% 1265     1.0MB/s   00:00    
    server-key.pem                                                                                                                                             100% 1679     1.3MB/s   00:00    
    server.pem                                                                                                                                                 100% 1346     1.5MB/s   00:00    
    etcd                                                                                                                                                       100%   18MB  31.4MB/s   00:00    
    etcdctl                                                                                                                                                    100%   15MB  37.7MB/s   00:00    
    etcd.service                                                                                                                                               100%  994   916.5KB/s   00:00    
    etcd.conf                                                                                                                                                  100%  520   487.5KB/s   00:00    
    Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
    ● etcd.service - Etcd Server
       Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:37:32 CST; 28ms ago
     Main PID: 7373 (etcd)
        Tasks: 7
       Memory: 9.1M
       CGroup: /system.slice/etcd.service
               └─7373 /usr/local/bin/etcd --name=etcd-01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.1.1.61:2380 --listen-client-urls=https://10.1.1.61:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.1.1.61:2379 --initial-advertise-peer-urls=https://10.1.1.61:2380 --initial-cluster=etcd-master=https://10.1.1.60:2380,etcd-01=https://10.1.1.61:2380,etcd-02=https://10.1.1.62:2380,etcd-03=https://10.1.1.63:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/etc/etcd/ca.d/server.pem --key-file=/etc/etcd/ca.d/server-key.pem --peer-cert-file=/etc/etcd/ca.d/server.pem --peer-key-file=/etc/etcd/ca.d/server-key.pem --trusted-ca-file=/etc/etcd/ca.d/ca.pem --peer-trusted-ca-file=/etc/etcd/ca.d/ca.pem
    
    3月 31 20:37:32 gysl-node1 etcd[7373]: 1c3555118a39401e initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
    3月 31 20:37:32 gysl-node1 etcd[7373]: raft.node: 1c3555118a39401e elected leader 63ac3c747757aa28 at term 138
    3月 31 20:37:32 gysl-node1 etcd[7373]: published {Name:etcd-01 ClientURLs:[https://10.1.1.61:2379]} to cluster 575c8b9e68fd927d
    3月 31 20:37:32 gysl-node1 etcd[7373]: ready to serve client requests
    3月 31 20:37:32 gysl-node1 etcd[7373]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
    3月 31 20:37:32 gysl-node1 etcd[7373]: ready to serve client requests
    3月 31 20:37:32 gysl-node1 systemd[1]: Started Etcd Server.
    3月 31 20:37:32 gysl-node1 etcd[7373]: serving client requests on 10.1.1.61:2379
    3月 31 20:37:32 gysl-node1 etcd[7373]: set the initial cluster version to 3.0
    3月 31 20:37:32 gysl-node1 etcd[7373]: enabled capabilities for version 3.0
    Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
    ● etcd.service - Etcd Server
       Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:37:34 CST; 124ms ago
     Main PID: 7551 (etcd)
        Tasks: 7
       Memory: 10.3M
       CGroup: /system.slice/etcd.service
               └─7551 /usr/local/bin/etcd --name=etcd-master --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://10.1.1.60:2380 --listen-client-urls=https://10.1.1.60:2379,http://127.0.0.1:2379 --advertise-client-urls=https://10.1.1.60:2379 --initial-advertise-peer-urls=https://10.1.1.60:2380 --initial-cluster=etcd-master=https://10.1.1.60:2380,etcd-01=https://10.1.1.61:2380,etcd-02=https://10.1.1.62:2380,etcd-03=https://10.1.1.63:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/etc/etcd/ca.d/server.pem --key-file=/etc/etcd/ca.d/server-key.pem --peer-cert-file=/etc/etcd/ca.d/server.pem --peer-key-file=/etc/etcd/ca.d/server-key.pem --trusted-ca-file=/etc/etcd/ca.d/ca.pem --peer-trusted-ca-file=/etc/etcd/ca.d/ca.pem
    
    3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 63ac3c747757aa28 (stream Message reader)
    3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 1c3555118a39401e (stream Message reader)
    3月 31 20:37:34 gysl-master etcd[7551]: published {Name:etcd-master ClientURLs:[https://10.1.1.60:2379]} to cluster 575c8b9e68fd927d
    3月 31 20:37:34 gysl-master etcd[7551]: ready to serve client requests
    3月 31 20:37:34 gysl-master etcd[7551]: serving client requests on 10.1.1.60:2379
    3月 31 20:37:34 gysl-master etcd[7551]: ready to serve client requests
    3月 31 20:37:34 gysl-master etcd[7551]: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
    3月 31 20:37:34 gysl-master etcd[7551]: 78df1ab24a6f1302 initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 3 active peer(s)
    3月 31 20:37:34 gysl-master systemd[1]: Started Etcd Server.
    3月 31 20:37:34 gysl-master etcd[7551]: established a TCP streaming connection with peer 76bcb3b85e42210d (stream Message reader)
    Please wait a moment!
    2019/03/31 20:38:34 [INFO] generating a new CA key and certificate from CSR
    2019/03/31 20:38:34 [INFO] generate received request
    2019/03/31 20:38:34 [INFO] received CSR
    2019/03/31 20:38:34 [INFO] generating key: rsa-2048
    2019/03/31 20:38:34 [INFO] encoded CSR
    2019/03/31 20:38:34 [INFO] signed certificate with serial number 284879897535931954074635242912207100624264127544
    2019/03/31 20:38:34 [INFO] generate received request
    2019/03/31 20:38:34 [INFO] received CSR
    2019/03/31 20:38:34 [INFO] generating key: rsa-2048
    2019/03/31 20:38:34 [INFO] encoded CSR
    2019/03/31 20:38:34 [INFO] signed certificate with serial number 163588537762519336822862885460408698694735647771
    2019/03/31 20:38:34 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
    websites. For more information see the Baseline Requirements for the Issuance and Management
    of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
    specifically, section 10.2.3 ("Information Requirements").
    2019/03/31 20:38:34 [INFO] generate received request
    2019/03/31 20:38:34 [INFO] received CSR
    2019/03/31 20:38:34 [INFO] generating key: rsa-2048
    2019/03/31 20:38:35 [INFO] encoded CSR
    2019/03/31 20:38:35 [INFO] signed certificate with serial number 269430846139878968754015022650791204259891937310
    2019/03/31 20:38:35 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
    websites. For more information see the Baseline Requirements for the Issuance and Management
    of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
    specifically, section 10.2.3 ("Information Requirements").
    /etc/kubernetes/ca.d
    ├── ca-config.json
    ├── ca.csr
    ├── ca-csr.json
    ├── ca-key.pem
    ├── ca.pem
    ├── kube-proxy.csr
    ├── kube-proxy-csr.json
    ├── kube-proxy-key.pem
    ├── kube-proxy.pem
    ├── server.csr
    ├── server-csr.json
    ├── server-key.pem
    └── server.pem
    
    0 directories, 13 files
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
    ● kube-apiserver.service - Kubernetes API Server
       Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:38:35 CST; 41ms ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 7628 (kube-apiserver)
        Tasks: 1
       Memory: 14.0M
       CGroup: /system.slice/kube-apiserver.service
               └─7628 /usr/local/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.1.1.60:2379,https://10.1.1.61:2379,https://10.1.1.62:2379,https://10.1.1.63:2379 --bind-address=10.1.1.60 --secure-port=6443 --advertise-address=10.1.1.60 --allow-privileged=true --service-cluster-ip-range=10.0.0.0/24 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/conf.d/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/etc/kubernetes/ca.d/server.pem --tls-private-key-file=/etc/kubernetes/ca.d/server-key.pem --client-ca-file=/etc/kubernetes/ca.d/ca.pem --service-account-key-file=/etc/kubernetes/ca.d/ca-key.pem --etcd-cafile=/etc/etcd/ca.d/ca.pem --etcd-certfile=/etc/etcd/ca.d/server.pem --etcd-keyfile=/etc/etcd/ca.d/server-key.pem
    
    3月 31 20:38:35 gysl-master systemd[1]: Started Kubernetes API Server.
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
    ● kube-scheduler.service - Kubernetes Scheduler
       Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:38:36 CST; 20s ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 7673 (kube-scheduler)
        Tasks: 7
       Memory: 47.5M
       CGroup: /system.slice/kube-scheduler.service
               └─7673 /usr/local/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
    
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.299502    7673 shared_informer.go:123] caches populated
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.399931    7673 shared_informer.go:123] caches populated
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.500642    7673 shared_informer.go:123] caches populated
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.601146    7673 shared_informer.go:123] caches populated
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.604604    7673 controller_utils.go:1027] Waiting for caches to sync for scheduler controller
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705500    7673 shared_informer.go:123] caches populated
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705529    7673 controller_utils.go:1034] Caches are synced for scheduler controller
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.705631    7673 leaderelection.go:217] attempting to acquire leader lease  kube-system/kube-scheduler...
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.737674    7673 leaderelection.go:227] successfully acquired lease kube-system/kube-scheduler
    3月 31 20:38:43 gysl-master kube-scheduler[7673]: I0331 20:38:43.838862    7673 shared_informer.go:123] caches populated
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
    ● kube-controller-manager.service - Kubernetes Controller Manager
       Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:38:56 CST; 20s ago
         Docs: https://github.com/kubernetes/kubernetes
     Main PID: 7725 (kube-controller)
        Tasks: 6
       Memory: 132.3M
       CGroup: /system.slice/kube-controller-manager.service
               └─7725 /usr/local/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.0.0.0/24 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ca.d/ca.pem --cluster-signing-key-file=/etc/kubernetes/ca.d/ca-key.pem --root-ca-file=/etc/kubernetes/ca.d/ca.pem --service-account-private-key-file=/etc/kubernetes/ca.d/ca-key.pem
    
    3月 31 20:38:59 gysl-master kube-controller-manager[7725]: I0331 20:38:59.915581    7725 request.go:530] Throttling request took 1.356935667s, request: GET:http://127.0.0.1:8080/apis/scheduling.k8s.io/v1?timeout=32s
    3月 31 20:38:59 gysl-master kube-controller-manager[7725]: I0331 20:38:59.965276    7725 request.go:530] Throttling request took 1.406608026s, request: GET:http://127.0.0.1:8080/apis/scheduling.k8s.io/v1beta1?timeout=32s
    3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.015978    7725 request.go:530] Throttling request took 1.457255375s, request: GET:http://127.0.0.1:8080/apis/coordination.k8s.io/v1?timeout=32s
    3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.065993    7725 request.go:530] Throttling request took 1.507246887s, request: GET:http://127.0.0.1:8080/apis/coordination.k8s.io/v1beta1?timeout=32s
    3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067050    7725 resource_quota_controller.go:427] syncing resource quota controller with updated resources from discovery: map[/v1, Resource=configmaps:{} /v1, Resource=endpoints:{} /v1, Resource=events:{} /v1, Resource=limitranges:{} /v1, Resource=persistentvolumeclaims:{} /v1, Resource=pods:{} /v1, Resource=podtemplates:{} /v1, Resource=replicationcontrollers:{} /v1, Resource=resourcequotas:{} /v1, Resource=secrets:{} /v1, Resource=serviceaccounts:{} /v1, Resource=services:{} apps/v1, Resource=controllerrevisions:{} apps/v1, Resource=daemonsets:{} apps/v1, Resource=deployments:{} apps/v1, Resource=replicasets:{} apps/v1, Resource=statefulsets:{} autoscaling/v1, Resource=horizontalpodautoscalers:{} batch/v1, Resource=jobs:{} batch/v1beta1, Resource=cronjobs:{} coordination.k8s.io/v1, Resource=leases:{} events.k8s.io/v1beta1, Resource=events:{} extensions/v1beta1, Resource=daemonsets:{} extensions/v1beta1, Resource=deployments:{} extensions/v1beta1, Resource=ingresses:{} extensions/v1beta1, Resource=networkpolicies:{} extensions/v1beta1, Resource=replicasets:{} networking.k8s.io/v1, Resource=networkpolicies:{} networking.k8s.io/v1beta1, Resource=ingresses:{} policy/v1beta1, Resource=poddisruptionbudgets:{} rbac.authorization.k8s.io/v1, Resource=rolebindings:{} rbac.authorization.k8s.io/v1, Resource=roles:{}]
    3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067168    7725 resource_quota_monitor.go:180] QuotaMonitor unable to use a shared informer for resource "extensions/v1beta1, Resource=networkpolicies": no informer found for extensions/v1beta1, Resource=networkpolicies
    3月 31 20:39:00 gysl-master kube-controller-manager[7725]: I0331 20:39:00.067189    7725 resource_quota_monitor.go:243] quota synced monitors; added 0, kept 30, removed 0
    3月 31 20:39:00 gysl-master kube-controller-manager[7725]: E0331 20:39:00.067197    7725 resource_quota_controller.go:437] failed to sync resource monitors: couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies"
    3月 31 20:39:13 gysl-master kube-controller-manager[7725]: I0331 20:39:13.677245    7725 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
    3月 31 20:39:14 gysl-master kube-controller-manager[7725]: I0331 20:39:14.215322    7725 pv_controller_base.go:407] resyncing PV controller
    clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
    Cluster "kubernetes" set.
    User "kubelet-bootstrap" set.
    Context "default" created.
    Switched to context "default".
    Cluster "kubernetes" set.
    User "kube-proxy" set.
    Context "default" created.
    Switched to context "default".
    member 1c3555118a39401e is healthy: got healthy result from https://10.1.1.61:2379
    member 63ac3c747757aa28 is healthy: got healthy result from https://10.1.1.63:2379
    member 76bcb3b85e42210d is healthy: got healthy result from https://10.1.1.62:2379
    member 78df1ab24a6f1302 is healthy: got healthy result from https://10.1.1.60:2379
    cluster is healthy
    { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
    kubelet                                                                                                                                                    100%  122MB  26.1MB/s   00:04    
    kube-proxy                                                                                                                                                 100%   35MB  21.9MB/s   00:01    
    flanneld                                                                                                                                                   100%   34MB  20.5MB/s   00:01    
    mk-docker-opts.sh                                                                                                                                          100% 2139   916.8KB/s   00:00    
    flanneld.conf                                                                                                                                              100%  247    55.8KB/s   00:00    
    flanneld.service                                                                                                                                           100%  389    82.7KB/s   00:00    
    kubelet.yaml                                                                                                                                               100%  263   319.4KB/s   00:00    
    kubelet.conf                                                                                                                                               100%  365   326.0KB/s   00:00    
    kube-proxy.conf                                                                                                                                            100%  158   184.0KB/s   00:00    
    kubelet.service                                                                                                                                            100%  267   234.2KB/s   00:00    
    kube-proxy.service                                                                                                                                         100%  234   130.5KB/s   00:00    
    bootstrap.kubeconfig                                                                                                                                       100% 2163     1.5MB/s   00:00    
    kube-proxy.kubeconfig                                                                                                                                      100% 6265     4.4MB/s   00:00    
    Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
    ● flanneld.service - Flanneld overlay address etcd agent
       Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:27 CST; 430ms ago
      Process: 7536 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
     Main PID: 7508 (flanneld)
        Tasks: 7
       Memory: 6.7M
       CGroup: /system.slice/flanneld.service
               └─7508 /usr/local/bin/flanneld --ip-masq
    
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.720837    7508 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.721919    7508 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.24.0/24 -j RETURN
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.722994    7508 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.724549    7508 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.730116    7508 iptables.go:155] Adding iptables rule: -d 172.17.0.0/16 -j ACCEPT
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.737143    7508 main.go:429] Waiting for 22h59m59.914613166s to renew lease
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.737262    7508 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.744276    7508 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.24.0/24 -j RETURN
    3月 31 20:39:27 gysl-node2 systemd[1]: Started Flanneld overlay address etcd agent.
    3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.766442    7508 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:28 CST; 10ms ago
         Docs: https://docs.docker.com
     Main PID: 7579 (dockerd)
        Tasks: 8
       Memory: 32.1M
       CGroup: /system.slice/docker.service
               └─7579 /usr/bin/dockerd --bip=172.17.24.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock
    
    3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843896719+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
    3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843917442+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154920, CONNECTING" module=grpc
    3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.843973658+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154920, READY" module=grpc
    3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.844332744+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
    3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.848229255+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
    3月 31 20:39:27 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:27.848828116+08:00" level=info msg="Loading containers: start."
    3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.081132437+08:00" level=info msg="Loading containers: done."
    3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.167227705+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
    3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.167281411+08:00" level=info msg="Daemon has completed initialization"
    3月 31 20:39:28 gysl-node2 dockerd[7579]: time="2019-03-31T20:39:28.175538228+08:00" level=info msg="API listen on /var/run/docker.sock"
    Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
    ● kubelet.service - Kubernetes Kubelet
       Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:28 CST; 61ms ago
     Main PID: 7727 (kubelet)
        Tasks: 1
       Memory: 2.1M
       CGroup: /system.slice/kubelet.service
               └─7727 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.62 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
    
    3月 31 20:39:28 gysl-node2 systemd[1]: Started Kubernetes Kubelet.
    
    ● kube-proxy.service - Kubernetes Proxy
       Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:28 CST; 7ms ago
     Main PID: 7728 (systemd)
        Tasks: 0
       Memory: 0B
       CGroup: /system.slice/kube-proxy.service
               └─7728 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
    kubelet                                                                                                                                                    100%  122MB  27.7MB/s   00:04    
    kube-proxy                                                                                                                                                 100%   35MB  13.8MB/s   00:02    
    flanneld                                                                                                                                                   100%   34MB  33.6MB/s   00:01    
    mk-docker-opts.sh                                                                                                                                          100% 2139     1.1MB/s   00:00    
    flanneld.conf                                                                                                                                              100%  247   225.5KB/s   00:00    
    flanneld.service                                                                                                                                           100%  389   357.8KB/s   00:00    
    kubelet.yaml                                                                                                                                               100%  263   193.7KB/s   00:00    
    kubelet.conf                                                                                                                                               100%  365   331.3KB/s   00:00    
    kube-proxy.conf                                                                                                                                            100%  158   130.4KB/s   00:00    
    kubelet.service                                                                                                                                            100%  267   295.5KB/s   00:00    
    kube-proxy.service                                                                                                                                         100%  234   198.3KB/s   00:00    
    bootstrap.kubeconfig                                                                                                                                       100% 2163     2.0MB/s   00:00    
    kube-proxy.kubeconfig                                                                                                                                      100% 6265     3.7MB/s   00:00    
    Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
    ● flanneld.service - Flanneld overlay address etcd agent
       Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:39 CST; 391ms ago
      Process: 7534 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
     Main PID: 7502 (flanneld)
        Tasks: 7
       Memory: 9.3M
       CGroup: /system.slice/flanneld.service
               └─7502 /usr/local/bin/flanneld --ip-masq
    
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088309    7502 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088315    7502 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.091984    7502 iptables.go:167] Deleting iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.095011    7502 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.100.0/24 -j RETURN
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.098419    7502 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.099751    7502 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.103532    7502 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.106520    7502 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.100.0/24 -j RETURN
    3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.113480    7502 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
    3月 31 20:39:39 gysl-node3 systemd[1]: Started Flanneld overlay address etcd agent.
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:39 CST; 10ms ago
         Docs: https://docs.docker.com
     Main PID: 7573 (dockerd)
        Tasks: 8
       Memory: 31.9M
       CGroup: /system.slice/docker.service
               └─7573 /usr/bin/dockerd --bip=172.17.100.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock
    
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230510356+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230556184+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154910, CONNECTING" module=grpc
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.230711652+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154910, READY" module=grpc
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.231101930+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.234478410+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.234950238+08:00" level=info msg="Loading containers: start."
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.406988224+08:00" level=info msg="Loading containers: done."
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.497837879+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.497901197+08:00" level=info msg="Daemon has completed initialization"
    3月 31 20:39:39 gysl-node3 dockerd[7573]: time="2019-03-31T20:39:39.502801194+08:00" level=info msg="API listen on /var/run/docker.sock"
    Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
    ● kubelet.service - Kubernetes Kubelet
       Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:39 CST; 58ms ago
     Main PID: 7721 (kubelet)
        Tasks: 1
       Memory: 4.2M
       CGroup: /system.slice/kubelet.service
               └─7721 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.63 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
    
    3月 31 20:39:39 gysl-node3 systemd[1]: Started Kubernetes Kubelet.
    
    ● kube-proxy.service - Kubernetes Proxy
       Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:39 CST; 21ms ago
     Main PID: 7722 (systemd)
        Tasks: 0
       Memory: 0B
       CGroup: /system.slice/kube-proxy.service
               └─7722 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
    
    3月 31 20:39:39 gysl-node3 systemd[1]: Started Kubernetes Proxy.
    kubelet                                                                                                                                                    100%  122MB  32.7MB/s   00:03    
    kube-proxy                                                                                                                                                 100%   35MB  14.4MB/s   00:02    
    flanneld                                                                                                                                                   100%   34MB  30.4MB/s   00:01    
    mk-docker-opts.sh                                                                                                                                          100% 2139     3.5MB/s   00:00    
    flanneld.conf                                                                                                                                              100%  247   227.9KB/s   00:00    
    flanneld.service                                                                                                                                           100%  389   359.0KB/s   00:00    
    kubelet.yaml                                                                                                                                               100%  263   197.9KB/s   00:00    
    kubelet.conf                                                                                                                                               100%  365   517.1KB/s   00:00    
    kube-proxy.conf                                                                                                                                            100%  158   244.5KB/s   00:00    
    kubelet.service                                                                                                                                            100%  267   379.4KB/s   00:00    
    kube-proxy.service                                                                                                                                         100%  234   324.5KB/s   00:00    
    bootstrap.kubeconfig                                                                                                                                       100% 2163   429.6KB/s   00:00    
    kube-proxy.kubeconfig                                                                                                                                      100% 6265     4.7MB/s   00:00    
    Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
    ● flanneld.service - Flanneld overlay address etcd agent
       Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:49 CST; 319ms ago
      Process: 7580 ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env (code=exited, status=0/SUCCESS)
     Main PID: 7550 (flanneld)
        Tasks: 7
       Memory: 6.7M
       CGroup: /system.slice/flanneld.service
               └─7550 /usr/local/bin/flanneld --ip-masq
    
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.414921    7550 vxlan_network.go:60] watching for new subnet leases
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.415303    7550 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.96.0/24 -j RETURN
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.416682    7550 iptables.go:167] Deleting iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.418320    7550 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 -d 172.17.0.0/16 -j RETURN
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.438055    7550 iptables.go:155] Adding iptables rule: -d 172.17.0.0/16 -j ACCEPT
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.443066    7550 main.go:429] Waiting for 22h59m59.922013672s to renew lease
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.443213    7550 iptables.go:155] Adding iptables rule: -s 172.17.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
    3月 31 20:39:49 gysl-node1 systemd[1]: Started Flanneld overlay address etcd agent.
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.459736    7550 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.96.0/24 -j RETURN
    3月 31 20:39:49 gysl-node1 flanneld[7550]: I0331 20:39:49.469674    7550 iptables.go:155] Adding iptables rule: ! -s 172.17.0.0/16 -d 172.17.0.0/16 -j MASQUERADE
    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:49 CST; 9ms ago
         Docs: https://docs.docker.com
     Main PID: 7622 (dockerd)
        Tasks: 8
       Memory: 28.6M
       CGroup: /system.slice/docker.service
               └─7622 /usr/bin/dockerd --bip=172.17.96.1/24 --ip-masq=false --mtu=1450 -H fd:// --containerd=/run/containerd/containerd.sock
    
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549105373+08:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549111902+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549148708+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154bb0, CONNECTING" module=grpc
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549210269+08:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420154bb0, READY" module=grpc
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.549578647+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.554893473+08:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.555866350+08:00" level=info msg="Loading containers: start."
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.695192119+08:00" level=info msg="Loading containers: done."
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.729225641+08:00" level=info msg="Docker daemon" commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
    3月 31 20:39:49 gysl-node1 dockerd[7622]: time="2019-03-31T20:39:49.729282016+08:00" level=info msg="Daemon has completed initialization"
    Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
    Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
    ● kubelet.service - Kubernetes Kubelet
       Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:50 CST; 50ms ago
     Main PID: 7770 (kubelet)
        Tasks: 1
       Memory: 2.1M
       CGroup: /system.slice/kubelet.service
               └─7770 /usr/local/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.1.1.61 --kubeconfig=/etc/kubernetes/conf.d/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/conf.d/bootstrap.kubeconfig --config=/etc/kubernetes/conf.d/kubelet.yaml --cert-dir=/etc/kubernetes/ca.d --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
    
    3月 31 20:39:50 gysl-node1 systemd[1]: Started Kubernetes Kubelet.
    
    ● kube-proxy.service - Kubernetes Proxy
       Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
       Active: active (running) since 日 2019-03-31 20:39:50 CST; 20ms ago
     Main PID: 7771 (systemd)
        Tasks: 0
       Memory: 0B
       CGroup: /system.slice/kube-proxy.service
               └─7771 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
    
    3月 31 20:39:50 gysl-node1 systemd[1]: Started Kubernetes Proxy.
    [root@gysl-master ~]# kubectl get cs,nodes
    NAME                                 STATUS    MESSAGE             ERROR
    componentstatus/scheduler            Healthy   ok
    componentstatus/controller-manager   Healthy   ok
    componentstatus/etcd-0               Healthy   {"health":"true"}
    componentstatus/etcd-2               Healthy   {"health":"true"}
    componentstatus/etcd-1               Healthy   {"health":"true"}
    componentstatus/etcd-3               Healthy   {"health":"true"}
    
    NAME             STATUS   ROLES   AGE     VERSION
    node/10.1.1.61   Ready    node    4m23s   v1.14.0
    node/10.1.1.62   Ready    node    4m22s   v1.14.0
    node/10.1.1.63   Ready    node    4m22s   v1.14.0

### 3.3 安装失败回滚脚本 ###

## 四 总结 ##

通过脚本实现自动化安装是一个良好的习惯，可以达到事半功倍的效果，以后工作中要注意培养这种习惯！