Docker（十）--K8s管理docker集群

冷不防 2021-11-05 21:00 335阅读 0赞

## 一.概念解释 ##

> Kubernetes的名字来自希腊语，意思是“舵手” 或 “领航员”。K8s是将8个字母“ubernete”替换为“8”的缩写。  
> Kubernetes是Google开源的一个容器编排引擎，它支持自动化部署、大规模可伸缩、应用容器化管理。在生产环境中部署一个应用程序时，通常要部署该应用的多个实例以便对应用请求进行负载均衡。  
> 在Kubernetes中，我们可以创建多个容器，每个容器里面运行一个应用实例，然后通过内置的负载均衡策略，实现对这一组应用实例的管理、发现、访问，而这些细节都不需要运维人员去进行复杂的手工配置和处理。

特点：

*  可移植: 支持公有云，私有云，混合云，多重云（multi-cloud）
 *  可扩展: 模块化，插件化，可挂载，可组合
 *  自动化: 自动部署，自动重启，自动复制，自动伸缩/扩展

## 二.构建K8s管理集群 ##

1.清除实验环境(server1、server2、server3操作相同)

*  清除列表管理(之前部署swarm的集群)

[root@server1 ~]# docker service ls
    [root@server1 ~]# docker service rm web
    [root@server1 ~]# docker stack rm portainer

*  清理卷

[root@server1 ~]# docker volume ls
    [root@server1 ~]# docker volume prune

*  删除容器

[root@server1 ~]# docker rm -f  `docker ps -aq`

*  删除网络

[root@server1 ~]# docker network ls
    [root@server1 ~]# docker network rm we_net1

*  节点离开swarm集群

[root@server1 ~]# docker swarm leave -f

*  禁用swap分区

[root@server1 ~]# swapoff -a
    [root@server1 ~]# vim /etc/fstab

![在这里插入图片描述][20190809103242742.png]  
![在这里插入图片描述][2019080910325048.png]

[root@server2 ~]# swapoff -a
    [root@server2 ~]# vim /etc/fstab

[root@server3 ~]# swapoff -a
    [root@server3 ~]# vim /etc/fstab

2.为server1,server2,server3安装软件包

[root@server1 ~]# ls
    cri-tools-1.13.0-0.x86_64.rpm  kubectl-1.15.0-0.x86_64.rpm  kubernetes-cni-0.7.5-0.x86_64.rpm
    kubeadm-1.15.0-0.x86_64.rpm    kubelet-1.15.0-0.x86_64.rpm
    [root@server1 ~]# yum install * -y
    [root@server1 ~]# scp * server2:
    [root@server1 ~]# scp * server3:

[root@server2 ~]# ls
    cri-tools-1.13.0-0.x86_64.rpm  kubectl-1.15.0-0.x86_64.rpm  kubernetes-cni-0.7.5-0.x86_64.rpm
    kubeadm-1.15.0-0.x86_64.rpm    kubelet-1.15.0-0.x86_64.rpm
    [root@server2 ~]# yum install *

[root@server3 ~]# ls
    cri-tools-1.13.0-0.x86_64.rpm  kubectl-1.15.0-0.x86_64.rpm  kubernetes-cni-0.7.5-0.x86_64.rpm
    kubeadm-1.15.0-0.x86_64.rpm    kubelet-1.15.0-0.x86_64.rpm
    [root@server3 ~]# yum install * -y

3.所有节点导入相关镜像

[root@server1 ~]# for i in *.tar; do docker load -i $i ;done

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70]  
4.将镜像传给server2、server3，并导入镜像

[root@server1 ~]# scp *.tar server2:
    [root@server1 ~]# scp *.tar server3:

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 1]

[root@server2 ~]# for i in *.tar; do docker load -i $i ;done

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 2]

[root@server3 ~]# for i in *.tar; do docker load -i $i ;done

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 3]

5.在所有节点的/etc/sysctl.d/k8s.conf内写入内容并查看

[root@server1 ~]# vim /etc/sysctl.d/k8s.conf
    [root@server1 ~]# cat /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-ip6tables = 1
    net.bridge.bridge-nf-call-iptables = 1
    [root@server1 ~]# scp /etc/sysctl.d/k8s.conf server2:/etc/sysctl.d/
    k8s.conf                                                                      100%   79     0.1KB/s   00:00    
    [root@server1 ~]# scp /etc/sysctl.d/k8s.conf server3:/etc/sysctl.d/
    k8s.conf           
    
    [root@server1 ~]# sysctl --system

![在这里插入图片描述][20190809112710237.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 4]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 5]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 6]  
6.在server1，server2,server3开启kubelet服务（此时查看状态仍未开启）

[root@server1 ~]# systemctl start kubelet
    [root@server1 ~]# systemctl enable kubelet

![在这里插入图片描述][20190809113008632.png]  
![在这里插入图片描述][20190809113011623.png]  
![在这里插入图片描述][20190809113014620.png]

6.初始化管理节点

[root@server1 ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.25.31.1

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 7]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 8]注意：如果第一次初始化集群失败，需要执行命令"kubeadm reset"进行重置，重置之后再执行初始化集群的命令，进行集群初始化。

7.创建用户kubeadm，并赋予所有权限

[root@server1 k8s]# useradd kubeadm
    [root@server1 k8s]# vim /etc/sudoers

![在这里插入图片描述][20190809150806334.png]

8.切换到kubeadm用户下创建文件并将其复制

[root@server1 ~]# su - kubeadm 
    [kubeadm@server1 ~]$ mkdir -p $HOME/.kube
    [kubeadm@server1 ~]$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    [kubeadm@server1 ~]$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
    [kubeadm@server1 ~]$ exit
    logout

9.在server1上配置 kubectl 命令补齐功能:

[kubeadm@server1 ~]$ echo "source <(kubectl completion bash)" >> .bashrc
    [kubeadm@server1 ~]$ logout
    [root@server1 ~]# su - kubeadm

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 9]

10.在server1上将kube-flannel.yml文件发送到/home/k8s目录下。因为kube-flannel.yml文件原来的/root/k8s目录下，普通用户k8s无法访问

[root@server1 ~]# cp kube-flannel.yml /home/kubeadm/
    [root@server1 ~]# su - kubeadm 
    Last login: Fri Aug  9 14:25:33 CST 2019 on pts/0
    [kubeadm@server1 ~]$ ls
    kube-flannel.yml
    [kubeadm@server1 ~]$ kubectl apply -f kube-flannel.yml
    clusterrole.rbac.authorization.k8s.io/flannel created
    clusterrolebinding.rbac.authorization.k8s.io/flannel created
    serviceaccount/flannel created
    configmap/kube-flannel-cfg created
    daemonset.extensions/kube-flannel-ds-amd64 created
    daemonset.extensions/kube-flannel-ds-arm64 created
    daemonset.extensions/kube-flannel-ds-arm created
    daemonset.extensions/kube-flannel-ds-ppc64le created
    daemonset.extensions/kube-flannel-ds-s390x created

11.在所有节点导入flannel

[root@server1 ~]# docker load -i flannel.tar 
    cd7100a72410: Loading layer  4.403MB/4.403MB
    3b6c03b8ad66: Loading layer  4.385MB/4.385MB
    93b0fa7f0802: Loading layer  158.2kB/158.2kB
    4165b2148f36: Loading layer  36.33MB/36.33MB
    b883fd48bb96: Loading layer   5.12kB/5.12kB
    Loaded image: quay.io/coreos/flannel:v0.10.0-amd64
    [root@server1 ~]# scp flannel.tar server2:
    flannel.tar                             100%   43MB  43.2MB/s   00:00    
    [root@server1 ~]# scp flannel.tar server3:
    flannel.tar                             100%   43MB  43.2MB/s   00:0

server2:

[root@server2 ~]# docker load  -i flannel.tar 
    cd7100a72410: Loading layer  4.403MB/4.403MB
    3b6c03b8ad66: Loading layer  4.385MB/4.385MB
    93b0fa7f0802: Loading layer  158.2kB/158.2kB
    4165b2148f36: Loading layer  36.33MB/36.33MB
    b883fd48bb96: Loading layer   5.12kB/5.12kB
    Loaded image: quay.io/coreos/flannel:v0.10.0-amd64

server3：

[root@server3 ~]# docker load -i flannel.tar 
    cd7100a72410: Loading layer  4.403MB/4.403MB
    3b6c03b8ad66: Loading layer  4.385MB/4.385MB
    93b0fa7f0802: Loading layer  158.2kB/158.2kB
    4165b2148f36: Loading layer  36.33MB/36.33MB
    b883fd48bb96: Loading layer   5.12kB/5.12kB
    Loaded image: quay.io/coreos/flannel:v0.10.0-amd64

11.在server2和server3上加载ipvs内核模块并使其临时生效

[root@server2 ~]# modprobe ip_vs_wrr
    [root@server2 ~]# modprobe ip_vs_rr
    [root@server2 ~]# modprobe ip_vs_sh
    [root@server2 ~]# modprobe ip_vs

[root@server3 ~]# modprobe ip_vs_wrr
    [root@server3 ~]# modprobe ip_vs_rr
    [root@server3 ~]# modprobe ip_vs_sh
    [root@server3 ~]# modprobe ip_vs

12.server2、server3加入节点

[root@server3 ~]# kubeadm join 172.25.31.1:6443 --token pu2qxd.7i5itunef7xta8wg \
    >     --discovery-token-ca-cert-hash sha256:d2999b6e117e399234cf47884a183229aabb834dc700ecacf42449c9d36a0074

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 10]

[root@server2 ~]# kubeadm join 172.25.31.1:6443 --token pu2qxd.7i5itunef7xta8wg \
    >     --discovery-token-ca-cert-hash sha256:d2999b6e117e399234cf47884a183229aabb834dc700ecacf42449c9d36a0074

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 11]13.获取默认namespace（default）下的pod，查看所有节点的状态是否都是ready，查看pods状态是否是running

[kubeadm@server1 ~]$ kubectl get nodes
    [kubeadm@server1 ~]$ kubectl get pods --all-namespaces

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 12]  
14.查看master状态

[kubeadm@server1 ~]$ kubectl get cs
    [kubeadm@server1 ~]$ kubectl get node
    [kubeadm@server1 ~]$  kubectl get all -n kube-system

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 13]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 14]  
注意：  
1.master节点要求cpu>=2  
2.

## 三.布置UI界面 ##

1.在所有节点导入kubernetes-dashboard.tar镜像

server1：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 15]server2:  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 16]server3:  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 17]  
2.获取或者自己编写kubernetes-dashboard.yaml并发送到/home/kubeadm/

[root@server1 ~]# cp kubernetes-dashboard.yaml /home/kubeadm/
    [root@server1 ~]# cd /home/kubeadm/
    [root@server1 kubeadm]# ls
    kube-flannel.yml  kubernetes-dashboard.yaml

# Copyright 2017 The Kubernetes Authors.
    #
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    
    # ------------------- Dashboard Secret ------------------- #
    
    apiVersion: v1
    kind: Secret
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard-certs
      namespace: kube-system
    type: Opaque
    
    ---
    # ------------------- Dashboard Service Account ------------------- #
    
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kube-system
    
    ---
    # ------------------- Dashboard Role & Role Binding ------------------- #
    
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: kubernetes-dashboard-minimal
      namespace: kube-system
    rules:
      # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["create"]
      # Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
    - apiGroups: [""]
      resources: ["configmaps"]
      verbs: ["create"]
      # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
    - apiGroups: [""]
      resources: ["secrets"]
      resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
      verbs: ["get", "update", "delete"]
      # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
    - apiGroups: [""]
      resources: ["configmaps"]
      resourceNames: ["kubernetes-dashboard-settings"]
      verbs: ["get", "update"]
      # Allow Dashboard to get metrics from heapster.
    - apiGroups: [""]
      resources: ["services"]
      resourceNames: ["heapster"]
      verbs: ["proxy"]
    - apiGroups: [""]
      resources: ["services/proxy"]
      resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
      verbs: ["get"]
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: kubernetes-dashboard-minimal
      namespace: kube-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: kubernetes-dashboard-minimal
    subjects:
    - kind: ServiceAccount
      name: kubernetes-dashboard
      namespace: kube-system
    
    ---
    # ------------------- Dashboard Deployment ------------------- #
    
    kind: Deployment
    apiVersion: apps/v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kube-system
    spec:
      replicas: 1
      revisionHistoryLimit: 10
      selector:
        matchLabels:
          k8s-app: kubernetes-dashboard
      template:
        metadata:
          labels:
            k8s-app: kubernetes-dashboard
        spec:
          containers:
          - name: kubernetes-dashboard
            image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1
            ports:
            - containerPort: 8443
              protocol: TCP
            args:
              - --auto-generate-certificates
              # Uncomment the following line to manually specify Kubernetes API server Host
              # If not specified, Dashboard will attempt to auto discover the API server and connect
              # to it. Uncomment only if the default does not work.
              # - --apiserver-host=http://my-address:port
            volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
            livenessProbe:
              httpGet:
                scheme: HTTPS
                path: /
                port: 8443
              initialDelaySeconds: 30
              timeoutSeconds: 30
          volumes:
          - name: kubernetes-dashboard-certs
            secret:
              secretName: kubernetes-dashboard-certs
          - name: tmp-volume
            emptyDir: {}
          serviceAccountName: kubernetes-dashboard
          # Comment the following tolerations if Dashboard must not be deployed on master
          tolerations:
          - key: node-role.kubernetes.io/master
            effect: NoSchedule
    
    ---
    # ------------------- Dashboard Service ------------------- #
    
    kind: Service
    apiVersion: v1
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
      name: kubernetes-dashboard
      namespace: kube-system
    spec:
      ports:
        - port: 443
          targetPort: 8443
      selector:
        k8s-app: kubernetes-dashboard

3.切换用户

[root@server1 kubeadm]# su - kubeadm 
    Last login: Fri Aug  9 14:57:20 CST 2019 on pts/0
    [kubeadm@server1 ~]$ ls
    kube-flannel.yml  kubernetes-dashboard.yaml
    [kubeadm@server1 ~]$ vim kubernetes-dashboard.yaml 
    [kubeadm@server1 ~]$ kubectl create -f kubernetes-dashboard.yaml

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 18]  
4.更改type

[kubeadm@server1 ~]$ kubectl edit service kubernetes-dashboard -n kube-system
    service/kubernetes-dashboard edited
    ##编辑类型type：NodePort

![在这里插入图片描述][20190809160717635.png]

5.获取NodePort

[kubeadm@server1 ~]$ kubectl describe svc kubernetes-dashboard -n kube-system
    [kubeadm@server1 ~]$ kubectl get pods -n kube-system

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 19]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 20]6.新建用户，用来管理UI界面

[kubeadm@server1 ~]$ vim dashboard-admin.yaml
    [kubeadm@server1 ~]$ kubectl create -f dashboard-admin.yaml 
    serviceaccount/admin-user created
    clusterrolebinding.rbac.authorization.k8s.io/admin-user created

文件内容：

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: admin-user
      namespace: kube-system
    
    ---
    
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: admin-user
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: admin-user
      namespace: kube-system

7.获取令牌（token）

[kubeadm@server1 ~]$ kubectl get secrets -n kube-system | grep admin
    admin-user-token-zgt2v                           kubernetes.io/service-account-token   3      3m54s
    [kubeadm@server1 ~]$ kubectl describe secrets admin-user-token-zgt2v -n kube-system

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 21]

测试：

浏览器输入：[https://172.25.31.1:31912][https_172.25.31.1_31912]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 22]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 23]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 24]

注意：

*  确保每个节点都加入镜像
 *  每个节点都要有网关，并能上网，（刚开始要连网）

[20190809103242742.png]: /images/20211104/1c15fc595054406ebfec4b780bc7db25.png
[2019080910325048.png]: /images/20211104/d5569da87de142dfada2f7fc904254ee.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70]: /images/20211104/c3a1875e77cd46d180ab9eca423c52e8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 1]: /images/20211104/0af7a1cfbcd44378870bc77219f62af4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 2]: /images/20211104/ca18c809e4914e5f9c01ef6839f2d7ee.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 3]: /images/20211104/3a9ac7d1acdd483ea39f324bb5d11b41.png
[20190809112710237.png]: /images/20211104/2d4c9aeb8ff240b7bdf68c6833baf8e2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 4]: /images/20211104/a9e0df46504e491fab1d8cb93ce7a0ef.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 5]: /images/20211104/19f315ffb9f94b5bbd9b177ab012837f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 6]: /images/20211104/c0a1d2469a964eb19b7d73659b4c1abb.png
[20190809113008632.png]: /images/20211104/d326c45218124846b36b44f65a7043c2.png
[20190809113011623.png]: /images/20211104/3940d6056151411d88f172a0c1225120.png
[20190809113014620.png]: /images/20211104/6bb783c7ef4b48bf9b159330f78fa1a0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 7]: /images/20211104/f8991c019a244566ad7d86843cc5df9a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 8]: /images/20211104/1032f3756eff4891b61a3430fa6d02af.png
[20190809150806334.png]: /images/20211104/b5e586bfb1d3440abc50480406bf779b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 9]: /images/20211104/92acdf70d9cf4d6981128f7258c0f047.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 10]: /images/20211104/04a50e62745849839649633175e7ae74.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 11]: /images/20211104/0ab39391a6034a9bbc115f7a9535008b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 12]: /images/20211104/23f86af45f3d41c3bd5b73944d6270e1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 13]: /images/20211104/c3c80ad21f6347d7a84061f4eeab2885.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 14]: /images/20211104/dfddee6c29dd456bb5a3d2fa7238df2c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 15]: /images/20211104/9d0de298caa3491898d254dc1c2e5307.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 16]: /images/20211104/67836daa82b4432d9c1bafc90f3a3cb1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 17]: /images/20211104/dc88fe4542514a4187d7460507cbdfb6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 18]: /images/20211104/e7c96eb052de45b3bdcee0753d73074b.png
[20190809160717635.png]: /images/20211104/58b2affbfe5d460d8a3fe9d1f10fbbc9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 19]: /images/20211104/c5631931b44c4df5a8b74c8516b2446f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 20]: /images/20211104/d15aba05387c437eb40d73ab76f4cf73.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 21]: /images/20211104/da0e43c6d7024e93b5e63252a5d0c511.png
[https_172.25.31.1_31912]: https://172.25.31.1:31912
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 22]: /images/20211104/d842dcc667ca40bb9f766100d1475e74.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 23]: /images/20211104/6e263f78e5ab4657a0178e79b5842690.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDg4OTYxNg_size_16_color_FFFFFF_t_70 24]: /images/20211104/7f1fe532c5e44e16bdf304705ad1087d.png