k8s集群中使用GlusterFS存储

女爷i 2022-12-04 01:16 126阅读 0赞

glusterfs在k8s集群中快速搭建的方法：

[https://github.com/kubernetes-retired/external-storage/tree/master/gluster/glusterfs][https_github.com_kubernetes-retired_external-storage_tree_master_gluster_glusterfs]

前提：需要k8s集群，至少需要2个node节点

本示例使用的k8s版本为1.18

安装使用

**1. 给每个node节点添加标记**

$ kubectl label nodes <storage-node-name> storagenode=glusterfs

**2.安装GlusterFS DaemonSet**

yaml地址：[https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/glusterfs-daemonset.yaml][https_github.com_kubernetes-retired_external-storage_blob_master_gluster_glusterfs_deploy_glusterfs-daemonset.yaml]

1.18版本需要将apiVersion改为apps/v1

spec加上selector便签

参考示例：glusterfs-daemonset.yaml

kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: glusterfs
      labels:
        glusterfs: daemonset
      annotations:
        description: GlusterFS DaemonSet
        tags: glusterfs
    spec:
    selector:
       matchLabels:
        glusterfs: pod
      template:
        metadata:
          name: glusterfs
          labels:
            glusterfs-node: pod
        spec:
          nodeSelector:
            storagenode: glusterfs
          hostNetwork: true
          containers:
          - image: gluster/gluster-centos:latest
            imagePullPolicy: IfNotPresent
            name: glusterfs
            volumeMounts:
            - name: glusterfs-heketi
              mountPath: "/var/lib/heketi"
            - name: glusterfs-run
              mountPath: "/run"
            - name: glusterfs-lvm
              mountPath: "/run/lvm"
            - name: glusterfs-etc
              mountPath: "/etc/glusterfs"
            - name: glusterfs-logs
              mountPath: "/var/log/glusterfs"
            - name: glusterfs-config
              mountPath: "/var/lib/glusterd"
            - name: glusterfs-dev
              mountPath: "/dev"
            - name: glusterfs-misc
              mountPath: "/var/lib/misc/glusterfsd"
            - name: glusterfs-cgroup
              mountPath: "/sys/fs/cgroup"
              readOnly: true
            - name: glusterfs-ssl
              mountPath: "/etc/ssl"
              readOnly: true
            securityContext:
              capabilities: {}
              privileged: true
            readinessProbe:
              timeoutSeconds: 3
              initialDelaySeconds: 40
              exec:
                command:
                - "/bin/bash"
                - "-c"
                - systemctl status glusterd.service
              periodSeconds: 25
              successThreshold: 1
              failureThreshold: 15
            livenessProbe:
              timeoutSeconds: 3
              initialDelaySeconds: 40
              exec:
                command:
                - "/bin/bash"
                - "-c"
                - systemctl status glusterd.service
              periodSeconds: 25
              successThreshold: 1
              failureThreshold: 15
          volumes:
          - name: glusterfs-heketi
            hostPath:
              path: "/var/lib/heketi"
          - name: glusterfs-run
          - name: glusterfs-lvm
            hostPath:
              path: "/run/lvm"
          - name: glusterfs-etc
            hostPath:
              path: "/etc/glusterfs"
          - name: glusterfs-logs
            hostPath:
              path: "/var/log/glusterfs"
          - name: glusterfs-config
            hostPath:
              path: "/var/lib/glusterd"
          - name: glusterfs-dev
            hostPath:
              path: "/dev"
          - name: glusterfs-misc
            hostPath:
              path: "/var/lib/misc/glusterfsd"
          - name: glusterfs-cgroup
            hostPath:
              path: "/sys/fs/cgroup"
          - name: glusterfs-ssl
            hostPath:
              path: "/etc/ssl"

$ kubectl create -f glusterfs-daemonset.yaml

创建glusterfs的daemonset

$ kubectl get pods  -l glusterfs=pod -o wide

![20200904183404254.png][]

**3. 将节点添加到trusted pool**

$ kubectl exec -ti glusterfs-drlgp gluster peer probe 192.168.20.165

$ kubectl exec -ti glusterfs-z5m4j gluster peer probe 192.168.20.45

**4. RBAC权限(Kubernetes 1.8+)**

Kubernetes 1.8+ provisioner访问k8s集群的API需要绑定一个ServiceAccount

rbac.yaml地址：

[https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/rbac.yaml][https_github.com_kubernetes-retired_external-storage_blob_master_gluster_glusterfs_deploy_rbac.yaml]

apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: glfs-provisioner
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: glfs-provisioner-runner
    rules:
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events", "pods/exec"]
        verbs: ["create", "update", "patch"]
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get", "list", "watch", "create", "update", "patch"]
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["services"]
        verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: run-glfs-provisioner
    subjects:
      - kind: ServiceAccount
        name: glfs-provisioner
        namespace: default
    # update namespace above to your namespace in order to make this work
    roleRef:
      kind: ClusterRole
      name: glfs-provisioner-runner
      apiGroup: rbac.authorization.k8s.io

kubectl create -f rbac.yaml

**5.创建StorageClass**

storageclass.yaml

地址：

https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/storageclass.yaml

kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: glusterfs-simple
    provisioner: gluster.org/glusterfs-simple
    parameters:
      forceCreate: "true"
      brickrootPaths: "192.168.20.165:/tmp/,192.168.20.45:/tmp"

$ kubectl create -f storageclass.yaml

**6.创建pvc**

pvc.yaml

pvc.yaml
    
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: gluster-simple-claim
      annotations:
        volume.beta.kubernetes.io/storage-class: "glusterfs-simple"
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi

$ kubectl create -f pvc.yaml

$ kubectl get pv,pvc

![20200904183718863.png][]

感谢：

[https://github.com/kubernetes-retired/external-storage/tree/master/gluster/glusterfs][https_github.com_kubernetes-retired_external-storage_tree_master_gluster_glusterfs]

[https://opensource.ncsa.illinois.edu/confluence/display/~lambert8/GlusterFS+in+Kubernetes][https_opensource.ncsa.illinois.edu_confluence_display_lambert8_GlusterFS_in_Kubernetes]

https://github.com/gluster/gluster-kubernetes/tree/master/deploy/kube-templates

[https_github.com_kubernetes-retired_external-storage_tree_master_gluster_glusterfs]: https://github.com/kubernetes-retired/external-storage/tree/master/gluster/glusterfs
[https_github.com_kubernetes-retired_external-storage_blob_master_gluster_glusterfs_deploy_glusterfs-daemonset.yaml]: https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/glusterfs-daemonset.yaml
[20200904183404254.png]: /images/20221123/d72b3b3c3070417a979c9ebb59481d8b.png
[https_github.com_kubernetes-retired_external-storage_blob_master_gluster_glusterfs_deploy_rbac.yaml]: https://github.com/kubernetes-retired/external-storage/blob/master/gluster/glusterfs/deploy/rbac.yaml
[20200904183718863.png]: /images/20221123/b220f72f198949708a966bf60b11b971.png
[https_opensource.ncsa.illinois.edu_confluence_display_lambert8_GlusterFS_in_Kubernetes]: https://opensource.ncsa.illinois.edu/confluence/display/~lambert8/GlusterFS+in+Kubernetes