初始化一个k8s集群

电玩女神 2022-11-29 11:38 189阅读 0赞

#### 环境搭建 ####

Windows环境下：VMWare15.5安装好的Ubuntu20.04

--------------------

需要完成下面这些步骤：

*  设置CPU核心数为2个以上（Vmware设置CPU核心数为2个即可）
 *  [关闭防火墙][Link 1]
 *  [Ubuntu关闭swap分区][Ubuntu_swap]
 *  [使用root登录终端][root]
 *  [安装Docker][Docker]
 *  [安装kubectl、kubelet、kubeadm][kubectl_kubelet_kubeadm]
 *

--------------------

### 初始化一个k8s集群（一个节点的集群） ###

使用kubeadm init命令初始化主服务器  
主服务器中会启动一个apiserver用于后面通过客户端向其发送请求，组建集群、控制集群

kubeadm init \
            --apiserver-advertise-address 192.168.117.110 \
            --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
            --kubernetes-version 1.18.0\
            --pod-network-cidr 10.11.0.0/16 \
            --service-cidr 10.20.0.0/16

### 参数说明 ###

1.  apiserver-advertise-address API服务器的主机ip地址，将上面`192.168.117.110`替换成主机的ip
2.  image-repository 默认镜像仓库地址：`k8s.gcr.io`国内无法访问，因此换成阿里的地址
3.  kubernetes-version k8s的版本，`注意版本问题`如果安装了更新版的k8s则将其改成对应的版本，一般末尾为0最后，比如我装的是1.18.8这可以填1.18.0-1.18.8，最好和软件版本一致即可。
4.  pod-network-cidr 生成的pod网络 /16表示掩码
5.  service-cidr 生产的service之间的网络 /16表示掩码

--------------------

如果遇到下面的错误：  
![在这里插入图片描述][20200820032821785.png_pic_center]  
第一个问题将`cpu核心数`设置为2个即可  
第二个问题是关于`swap分区`的问题

--------------------

临时禁用swap

sudo swapoff -a

永久禁用swap，编辑fstab

vi /etc/fstab

![在这里插入图片描述][20200820032728694.png_pic_center]  
最后重启即可

--------------------

成功初始化节点会有如下信息  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxODEzMjA4_size_16_color_FFFFFF_t_70_pic_center]  
注意最后一行打印的信息将其复制出来，在其它节点通过这条命令将节点加入这个集群中  
另外根据截图中的提示执行下面的命令

mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config

另外提示需要部署一个pod网络  
首先、创建一个yml文件，并且填入如下内容，大致的目的就是配置kubernates的安装策略  
在自己电脑上创建一个k8s.yml文本文件，复制下面内容进去，然后上传该文件到服务器上

---
    apiVersion: policy/v1beta1
    kind: PodSecurityPolicy
    metadata:
      name: psp.flannel.unprivileged
      annotations:
        seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
        seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
        apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
        apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
    spec:
      privileged: false
      volumes:
        - configMap
        - secret
        - emptyDir
        - hostPath
      allowedHostPaths:
        - pathPrefix: "/etc/cni/net.d"
        - pathPrefix: "/etc/kube-flannel"
        - pathPrefix: "/run/flannel"
      readOnlyRootFilesystem: false
      # Users and groups
      runAsUser:
        rule: RunAsAny
      supplementalGroups:
        rule: RunAsAny
      fsGroup:
        rule: RunAsAny
      # Privilege Escalation
      allowPrivilegeEscalation: false
      defaultAllowPrivilegeEscalation: false
      # Capabilities
      allowedCapabilities: ['NET_ADMIN']
      defaultAddCapabilities: []
      requiredDropCapabilities: []
      # Host namespaces
      hostPID: false
      hostIPC: false
      hostNetwork: true
      hostPorts:
      - min: 0
        max: 65535
      # SELinux
      seLinux:
        # SELinux is unused in CaaSP
        rule: 'RunAsAny'
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: flannel
    rules:
      - apiGroups: ['extensions']
        resources: ['podsecuritypolicies']
        verbs: ['use']
        resourceNames: ['psp.flannel.unprivileged']
      - apiGroups:
          - ""
        resources:
          - pods
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - nodes/status
        verbs:
          - patch
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: flannel
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: flannel
    subjects:
    - kind: ServiceAccount
      name: flannel
      namespace: kube-system
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: flannel
      namespace: kube-system
    ---
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: kube-flannel-cfg
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    data:
      cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] }
      net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } }
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-amd64
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - amd64
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-amd64
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-amd64
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-arm64
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - arm64
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-arm64
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-arm64
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-arm
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - arm
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-arm
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-arm
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-ppc64le
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - ppc64le
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-ppc64le
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-ppc64le
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg
    ---
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: kube-flannel-ds-s390x
      namespace: kube-system
      labels:
        tier: node
        app: flannel
    spec:
      selector:
        matchLabels:
          app: flannel
      template:
        metadata:
          labels:
            tier: node
            app: flannel
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: beta.kubernetes.io/os
                        operator: In
                        values:
                          - linux
                      - key: beta.kubernetes.io/arch
                        operator: In
                        values:
                          - s390x
          hostNetwork: true
          tolerations:
          - operator: Exists
            effect: NoSchedule
          serviceAccountName: flannel
          initContainers:
          - name: install-cni
            image: quay.io/coreos/flannel:v0.11.0-s390x
            command:
            - cp
            args:
            - -f
            - /etc/kube-flannel/cni-conf.json
            - /etc/cni/net.d/10-flannel.conflist
            volumeMounts:
            - name: cni
              mountPath: /etc/cni/net.d
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          containers:
          - name: kube-flannel
            image: quay.io/coreos/flannel:v0.11.0-s390x
            command:
            - /opt/bin/flanneld
            args:
            - --ip-masq
            - --kube-subnet-mgr
            resources:
              requests:
                cpu: "100m"
                memory: "50Mi"
              limits:
                cpu: "100m"
                memory: "50Mi"
            securityContext:
              privileged: false
              capabilities:
                 add: ["NET_ADMIN"]
            env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: run
              mountPath: /run/flannel
            - name: flannel-cfg
              mountPath: /etc/kube-flannel/
          volumes:
            - name: run
              hostPath:
                path: /run/flannel
            - name: cni
              hostPath:
                path: /etc/cni/net.d
            - name: flannel-cfg
              configMap:
                name: kube-flannel-cfg

上传到服务器后，根据这个yml配置文件部署一个flannel网络

kubectl apply -f k8s.yml

完成后才可以执行后面的操作，将其它节点加入到当前的集群中

--------------------

### 将其它节点加入集群 ###

登录其它节点，以root身份执行这条命令

需要注意下面的这条命令是我当时获得的命令，你需要换成自己同样方式获取的这个加入节点命令

kubeadm join 192.168.117.110:6443 --token h7m4c0.5v5adnyafbx18j60 \
        --discovery-token-ca-cert-hash sha256:3334d8e5fabbf52922c00b4d705a3c7fc50fc81e7f3f7a21a1860e5b6e2dcf26

如果这条命令过时了，或者找不到了，执行下面命令获取一条新的永久加入集群命令

kubeadm token create --ttl 0 --print-join-command

登录到另外一台服务器（kubeadm、kubelet、kubectl以及docker都已经安装），将上面这行命令执行一下即可加入集群，加入这台服务器的ip是`192.168.117.111`

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxODEzMjA4_size_16_color_FFFFFF_t_70_pic_center 1]

#### 可能遇到的错误 ####

![在这里插入图片描述][20200916030515803.png_pic_center]

##### 解决方法： #####

重命名要加入的节点主机名（只有不与集群中其它节点名冲突即可）k8s应该是默认用主机名作为集群节点名

###### 临时修改hostname： ######

hostname node2

###### 永久修改hostname： ######

vim /etc/hostname

修改文件中内容`k8s-master`为`node2`，不与k8s-master重名即可

重启kubelet服务

systemctl restart kubelet

修改hosts文件内容

vim /etc/hosts

将其中ip映射的名称改成和hostname文件一样的域名名称如下  
![在这里插入图片描述][20200917013322924.png_pic_center]

然后重新执行。

如果出现ca.crt 已存在![在这里插入图片描述][20200916030757749.png_pic_center]  
删除这个文件即可

rm -f /etc/kubernetes/pki/ca.crt

重新执行加入集群命令

--------------------

### 最后验证节点加入集群 ###

在master节点输入命令查看节点状况

kubectl get nodes

如下说明节点加入成功！！  
如果是NoReady状态等待一会，直到变成Ready说明节点真正加入了集群

![在这里插入图片描述][20200916040046175.png_pic_center]  
如果一直是NoReady状态，需要检查一下问题，肯定是哪里出了问题，例如我遇到的情况是  
克隆出来的其它节点和`k8s-master`有着一样的信息，`/etc/hostname`文件中的值也是`k8s-master`，而k8s默认的节点名称就是hostname中的值,因此需要修改这个文件内容的值，不与已有集群中节点重名

kubeadm reset

然后重新加入，如果遇到文件已存在，则删除后重新加入集群

--------------------

以下是测试内容

### 初始化一个Tomcat集群 ###

###### 下载镜像并启动tomcat服务器 ######

kubectl create deployment tomcat --image=tomcat

###### 查看pod启动的tomcat服务器信息 ######

kubectl get pods -o wide

其中NODE表示tomcat再那台服务器上运行着  
![在这里插入图片描述][20200916040713681.png_pic_center]

###### 设置副本数为3个 ######

kubectl scale --replicas=3 deployment tomcat

![在这里插入图片描述][20200916040950138.png_pic_center]

###### 暴露tomcat端口 ######

kubectl expose deployment tomcat --port=80 --target-port=8080 --type=NodePort

###### 获取暴露的端口 ######

kubectl get svc -o wide

![在这里插入图片描述][20200916042339147.png_pic_center]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxODEzMjA4_size_16_color_FFFFFF_t_70_pic_center 2]

[Link 1]: https://blog.csdn.net/qq_41813208/article/details/103208450
[Ubuntu_swap]: https://blog.csdn.net/qq_41813208/article/details/108624367
[root]: https://blog.csdn.net/qq_41813208/article/details/103201539
[Docker]: https://blog.csdn.net/qq_41813208/article/details/103021332
[kubectl_kubelet_kubeadm]: https://blog.csdn.net/qq_41813208/article/details/107123578
[20200820032821785.png_pic_center]: /images/20221124/c54bbe3c8fe14f478e12a8dcd3a5052d.png
[20200820032728694.png_pic_center]: /images/20221124/064b4fc5239d45c4ac86dbe2fcbf4d51.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxODEzMjA4_size_16_color_FFFFFF_t_70_pic_center]: /images/20221124/5a2aa04aec5d42dfaed7f169035cd3df.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxODEzMjA4_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221124/b7084cd87a43492fa0e7730e068225aa.png
[20200916030515803.png_pic_center]: /images/20221124/567b4c67e1294762b996cb964a8d62c1.png
[20200917013322924.png_pic_center]: /images/20221124/45f1094dc1ae434a86d3555dfc169d97.png
[20200916030757749.png_pic_center]: /images/20221124/143ad920175a474e8d76be351213def5.png
[20200916040046175.png_pic_center]: /images/20221124/0a023dc56ab640aeb76a0cbfc7af931e.png
[20200916040713681.png_pic_center]: /images/20221124/758ca77fb59d4960970e70a4abd795e6.png
[20200916040950138.png_pic_center]: /images/20221124/6ead92e5c6754b84a99068caa5db5099.png
[20200916042339147.png_pic_center]: /images/20221124/918a2227745c47768ce7bc4be4cb7860.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQxODEzMjA4_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221124/472c6faefa46407ca2517a863e1f0780.png