k8s(十二)、分布式存储Ceph RBD使用

逃离我推掉我的手 2022-05-09 02:12 813阅读 0赞

# 前言 #

上篇文章介绍了k8s使用pv/pvc 的方式使用cephfs,  
[k8s(十一)、分布式存储Cephfs使用][k8s_Cephfs]

**Ceph存储**有三种存储接口,分别是:  
**对象存储 Ceph Object Gateway  
块设备 RBD  
文件系统 CEPHFS**

Kubernetes支持后两种存储接口,支持的接入模式如下图:  
![在这里插入图片描述][70]

**在本篇将测试使用ceph rbd作持久化存储后端**

# RBD创建测试 #

**rbd的使用分为3个步骤:**  
1.服务端/客户端创建块设备image  
2.客户端将image映射进linux系统内核,内核识别出该块设备后生成dev下的文件标识  
3.格式化块设备并挂载使用

# 创建
    [root@h020112 ~]# rbd create --size 10240000 rbd/test1    #创建指定大小的node
    [root@h020112 ~]# rbd info test1
    rbd image 'test1':
    	size 10000 GB in 2560000 objects
    	order 22 (4096 kB objects)
    	block_name_prefix: rbd_data.31e3b6b8b4567
    	format: 2
    	features: layering, exclusive-lock,object-map, fast-diff, deep-flatten
    	flags: 
    
    #映射进内核操作之前,首先查看内核版本,jw版本的ceph默认format为2,默认开启5种特性,2.x及之前的内核版本需手动调整format为1
    ,4.x之前要关闭object-map, fast-diff, deep-flatten功能才能成功映射到内核,这里使用的是centos7.4,内核版本3.10
    
    [root@h020112 ~]# rbd feature disable test1 object-map fast-diff deep-flatten
    
    # 映射进内核
    [root@h020112 ~]# rbd map test1
    [root@h020112 ~]# ls /dev/rbd0
    /dev/rbd0
    
    #挂载使用
    [root@h020112 ~]# mkdir /mnt/cephrbd
    [root@h020112 ~]#  mkfs.ext4 -m0 /dev/rbd0
    [root@h020112 ~]#  mount /dev/rbd0 /mnt/cephrbd/

# Kubernetes Dynamic Pv 使用 #

首先回顾一下上篇cephfs在k8s中的使用方式:  
**绑定使用流程: pv指定存储类型和接口,pvc绑定pv,pod挂载指定pvc.这种一一对应的方式,即为静态pv**  
例如cephfs的使用流程:  
![在这里插入图片描述][70 1]

那么什么是**动态pv**呢?  
**在现有PV不满足PVC的请求时，可以使用存储分类(StorageClass)，pvc对接sc后,k8s会动态地创建pv,并且向相应的存储后端发出资源申请.  
描述具体过程为：首先创建sc分类，PVC请求已创建的sc的资源，来自动创建pv,这样就达到动态配置的效果。即通过一个叫 Storage Class的对象由存储系统根据PVC的要求自动创建pv并使用存储资源**

Kubernetes的pv支持rbd,从第一部分测试使用rbd的过程可以看出,存储实际使用的单位是image,首先必须在ceph端创建image,才能创建pv/pvc,再在pod里面挂载使用.这是一个标准的静态pv的绑定使用流程.但是这样的流程就带来了一个弊端,即一个完整的存储资源挂载使用的操作被分割成了两段,一段在ceph端划分rbd image,一段由k8s端绑定pv,这样使得管理的复杂性提高,不利于自动化运维.

因此,动态pv是一个很好的选择,ceph rbd接口的动态pv的实现流程是这样的:  
**根据pvc及sc的声明,自动生成pv,且向sc指定的ceph后端发出申请自动创建相应大小的rbd image,最终绑定使用**

**工作流程图:**  
![在这里插入图片描述][70 2]

下面开始演示  
**1.创建secret**  
key获取方式参考上篇文章

~/mytest/ceph/rbd# cat ceph-secret-rbd.yaml 
    apiVersion: v1
    kind: Secret
    metadata:
      name: ceph-secret-rbd
    type: "kubernetes.io/rbd"
    data:
      key: QVFCL3E1ZGIvWFdxS1JBQTUyV0ZCUkxldnRjQzNidTFHZXlVYnc9PQ==

**2.创建sc**

~/mytest/ceph/rbd# cat ceph-secret-rbd.yaml
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: cephrbd-test2
    provisioner: kubernetes.io/rbd
    reclaimPolicy: Retain
    parameters:
      monitors: 192.168.20.112:6789,192.168.20.113:6789,192.168.20.114:6789
      adminId: admin
      adminSecretName: ceph-secret-rbd
      pool: rbd
      userId: admin
      userSecretName: ceph-secret-rbd
      fsType: ext4
      imageFormat: "2"
      imageFeatures: "layering"

**创建pvc**

~/mytest/ceph/rbd# cat cephrbd-pvc.yaml 
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: cephrbd-pvc2
    spec:
      storageClassName: cephrbd-test2
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi

**创建deploy,容器挂载使用创建的pvc**

:~/mytest/ceph/rbd# cat deptest13dbdm.yaml 
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      labels:
        app: deptest13dbdm
      name: deptest13dbdm
      namespace: default
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: deptest13dbdm
      strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: deptest13dbdm
        spec:
          containers:
            image: registry.xxx.com:5000/mysql:56v6
            imagePullPolicy: Always
            name: deptest13dbdm
            ports:
            - containerPort: 3306
              protocol: TCP
            resources:
              limits:
                memory: 16Gi
              requests:
                memory: 512Mi
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /var/lib/mysql
              name: datadir
              subPath: deptest13dbdm
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
          volumes:
          - name: datadir
            persistentVolumeClaim:
              claimName: cephrbd-pvc2

**这里会遇到一个大坑:**

~# kubectl describe pvc cephrbd-pvc2 
    ...
    Error creating rbd image: executable file not found in $PATH

这是因为kube-controller-manager需要调用rbd客户端api来创建rbd image,集群的每个节点都已经安装ceph-common,但是集群是使用kubeadm部署的,kube-controller-manager组件是以容器方式工作的,内部不包含rbd的执行文件,后面在这个issue里,找到了解决的办法:  
[Github issue][]  
使用外部的rbd-provisioner,提供给kube-controller-manager以rbd的执行入口,注意,issue内的demo指定的image版本有bug,不能指定rbd挂载后使用的文件系统格式,需要使用最新的版本demo,项目地址:[rbd-provisioner][]  
**完整demo yaml文件:**

root@h009027:~/mytest/ceph/rbd/rbd-provisioner# cat rbd-provisioner.yaml 
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-provisioner
    rules:
      - apiGroups: [""]
        resources: ["persistentvolumes"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: [""]
        resources: ["persistentvolumeclaims"]
        verbs: ["get", "list", "watch", "update"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses"]
        verbs: ["get", "list", "watch"]
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["create", "update", "patch"]
      - apiGroups: [""]
        resources: ["services"]
        resourceNames: ["kube-dns","coredns"]
        verbs: ["list", "get"]
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: rbd-provisioner
    subjects:
      - kind: ServiceAccount
        name: rbd-provisioner
        namespace: default
    roleRef:
      kind: ClusterRole
      name: rbd-provisioner
      apiGroup: rbac.authorization.k8s.io
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: rbd-provisioner
    rules:
    - apiGroups: [""]
      resources: ["secrets"]
      verbs: ["get"]
    - apiGroups: [""]
      resources: ["endpoints"]
      verbs: ["get", "list", "watch", "create", "update", "patch"]
    
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: rbd-provisioner
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: rbd-provisioner
    subjects:
    - kind: ServiceAccount
      name: rbd-provisioner
      namespace: default
    
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: rbd-provisioner
    spec:
      replicas: 1
      strategy:
        type: Recreate
      template:
        metadata:
          labels:
            app: rbd-provisioner
        spec:
          containers:
          - name: rbd-provisioner
            image: quay.io/external_storage/rbd-provisioner:latest
            env:
            - name: PROVISIONER_NAME
              value: ceph.com/rbd
          serviceAccount: rbd-provisioner
    
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: rbd-provisioner

部署完成后,重新创建上方的sc/pv,这个时候,pod就正常运行起来了,查看pvc,可以看到此pvc触发自动生成了一个pv且绑定了此pv,pv绑定的StorageClass则是刚创建的sc:

# pod
    ~/mytest/ceph/rbd/rbd-provisioner# kubectl get pods | grep deptest13
    deptest13dbdm-c9d5bfb7c-rzzv6                     2/2       Running            0          6h
    
    # pvc
    ~/mytest/ceph/rbd/rbd-provisioner# kubectl get pvc cephrbd-pvc2
    NAME           STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
    cephrbd-pvc2   Bound     pvc-d976c2bf-c2fb-11e8-878f-141877468256   20Gi       RWO            cephrbd-test2   23h
    
    # 自动创建的pv
    ~/mytest/ceph/rbd/rbd-provisioner# kubectl get pv pvc-d976c2bf-c2fb-11e8-878f-141877468256
    NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                  STORAGECLASS    REASON    AGE
    pvc-d976c2bf-c2fb-11e8-878f-141877468256   20Gi       RWO            Retain           Bound     default/cephrbd-pvc2   cephrbd-test2             23h
    
    # sc
    ~/mytest/ceph/rbd/rbd-provisioner# kubectl get sc cephrbd-test2
    NAME            PROVISIONER    AGE
    cephrbd-test2   ceph.com/rbd   23h

**再回到ceph节点上看看是否自动创建了rbd image**

~# rbd ls
    kubernetes-dynamic-pvc-d988cfb1-c2fb-11e8-b2df-0a58ac1a084e
    test1
    
    ~# rbd info kubernetes-dynamic-pvc-d988cfb1-c2fb-11e8-b2df-0a58ac1a084e
    rbd image 'kubernetes-dynamic-pvc-d988cfb1-c2fb-11e8-b2df-0a58ac1a084e':
    	size 20480 MB in 5120 objects
    	order 22 (4096 kB objects)
    	block_name_prefix: rbd_data.47d0b6b8b4567
    	format: 2
    	features: layering
    	flags:

可以发现,动态创建pv/rbd成功

# 挂载测试 #

回顾最上方的kubernetes支持的接入模式截图,rbd是不支持ReadWriteMany的,而k8s的理念是像牲口一样管理你的容器,容器节点有可能随时横向扩展,横向缩减,在节点间漂移,因此,不支持多点挂载的RBD模式,显然不适合做多应用节点之间的共享文件存储,这里进行一次进一步的验证

#### 多点挂载测试 ####

**创建一个新的deploy,并挂载同一个pvc:**

root@h009027:~/mytest/ceph/rbd# cat deptest14dbdm.yaml 
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "4"
      creationTimestamp: 2018-09-26T07:32:41Z
      generation: 4
      labels:
        app: deptest14dbdm
        domain: dev.deptest14dbm.kokoerp.com
        nginx_server: ""
      name: deptest14dbdm
      namespace: default
      resourceVersion: "26647927"
      selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/deptest14dbdm
      uid: 5a1a62fe-c15e-11e8-878f-141877468256
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: deptest14dbdm
      strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1
        type: RollingUpdate
      template:
        metadata:
          creationTimestamp: null
          labels:
            app: deptest14dbdm
            domain: dev.deptest14dbm.kokoerp.com
            nginx_server: ""
        spec:
          containers:
          - env:
            - name: DB_NAME
              value: deptest14dbdm
            image: registry.youkeshu.com:5000/mysql:56v6
            imagePullPolicy: Always
            name: deptest14dbdm
            ports:
            - containerPort: 3306
              protocol: TCP
            resources:
              limits:
                memory: 16Gi
              requests:
                memory: 512Mi
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /var/lib/mysql
              name: datadir
              subPath: deptest14dbdm
          - env:
            - name: DATA_SOURCE_NAME
              value: exporter:6qQidF3F5wNhrLC0@(127.0.0.1:3306)/
            image: registry.youkeshu.com:5000/mysqld-exporter:v0.10
            imagePullPolicy: Always
            name: mysql-exporter
            ports:
            - containerPort: 9104
              protocol: TCP
            resources: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: ClusterFirst
          nodeSelector:
            192.168.20.109: ""
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
          volumes:
          - name: datadir
            persistentVolumeClaim:
              claimName: cephrbd-pvc2

**使用nodeselector将此deptest13和deptest14限制在同一个node中:**

root@h009027:~/mytest/ceph/rbd# kubectl get pods -o wide | grep deptest1[3-4]
    deptest13dbdm-c9d5bfb7c-rzzv6                     2/2       Running            0          6h        172.26.8.82    h020109
    deptest14dbdm-78b8bb7995-q84dk                    2/2       Running            0          2m        172.26.8.84    h020109

**不难发现在同一个节点中,是可以支持多个pod共享rbd image运行的,在容器的工作节点上查看mount条目也可以证实这一点:**

192.168.20.109:~# mount | grep rbd
    /dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-kubernetes-dynamic-pvc-d988cfb1-c2fb-11e8-b2df-0a58ac1a084e type ext4 (rw,relatime,stripe=1024,data=ordered)
    /dev/rbd0 on /var/lib/kubelet/pods/68b8e3c5-c38d-11e8-878f-141877468256/volumes/kubernetes.io~rbd/pvc-d976c2bf-c2fb-11e8-878f-141877468256 type ext4 (rw,relatime,stripe=1024,data=ordered)
    /dev/rbd0 on /var/lib/kubelet/pods/1e9b04b1-c3c3-11e8-878f-141877468256/volumes/kubernetes.io~rbd/pvc-d976c2bf-c2fb-11e8-878f-141877468256 type ext4 (rw,relatime,stripe=1024,data=ordered)

**修改新deploy的nodeselector指向其他的node:**

~/mytest/ceph/rbd# kubectl get pods -o wide| grep deptest14db
    deptest14dbdm-78b8bb7995-q84dk                    0/2       Terminating         0          7m        172.26.8.84    h020109
    deptest14dbdm-79fd495579-rkl92                    0/2       ContainerCreating   0          49s       <none>         h009028
    
    # kubectl describe pod deptest14dbdm-79fd495579-rkl92
    ...
      Warning  FailedAttachVolume     1m    attachdetach-controller  Multi-Attach error for volume "pvc-d976c2bf-c2fb-11e8-878f-141877468256" Volume is already exclusively attached to one node and can't be attached to another

pod将阻塞在ContainerCreating状态,描述报错很清晰地指出控制器不支持多点挂载此pv.  
进行下一步测试前先删除此临时新增的deploy

#### 滚动更新测试 ####

**在deptest13dbdm的deploy文件中指定的更新方式为滚动更新:**

strategy:
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1
        type: RollingUpdate

**这里我们尝试一下,将正在运行的pod,从一个节点调度到另一个节点,rbd 是否能迁移到新的节点上挂载:**

# 修改nodeselector
    root@h009027:~/mytest/ceph/rbd# kubectl edit deploy deptest13dbdm
    deployment "deptest13dbdm" edited
    
    # 观察pod调度迁移过程
    root@h009027:~/mytest/ceph/rbd# kubectl get pods -o wide| grep deptest13
    deptest13dbdm-6999f5757c-fk9dp                    0/2       ContainerCreating   0          4s        <none>         h009028
    deptest13dbdm-c9d5bfb7c-rzzv6                     2/2       Terminating         0          6h        172.26.8.82    h020109
    
    root@h009027:~/mytest/ceph/rbd# kubectl describe pod deptest13dbdm-6999f5757c-fk9dp 
    ...
    Events:
      Type     Reason                 Age                From                 Message
      ----     ------                 ----               ----                 -------
      Normal   Scheduled              2m                 default-scheduler    Successfully assigned deptest13dbdm-6999f5757c-fk9dp to h009028
      Normal   SuccessfulMountVolume  2m                 kubelet, h009028  MountVolume.SetUp succeeded for volume "default-token-7q8pq"
      Warning  FailedMount            45s                kubelet, h009028  Unable to mount volumes for pod "deptest13dbdm-6999f5757c-fk9dp_default(28e9cdd6-c3c5-11e8-878f-141877468256)": timeout expired waiting for volumes to attach/mount for pod "default"/"deptest13dbdm-6999f5757c-fk9dp". list of unattached/unmounted volumes=[datadir]
      Normal   SuccessfulMountVolume  35s (x2 over 36s)  kubelet, h009028  MountVolume.SetUp succeeded for volume "pvc-d976c2bf-c2fb-11e8-878f-141877468256"
      Normal   Pulling                27s                kubelet, h009028  pulling image "registry.youkeshu.com:5000/mysql:56v6"
      Normal   Pulled                 27s                kubelet, h009028  Successfully pulled image "registry.youkeshu.com:5000/mysql:56v6"
      Normal   Pulling                27s                kubelet, h009028  pulling image "registry.youkeshu.com:5000/mysqld-exporter:v0.10"
      Normal   Pulled                 26s                kubelet, h009028  Successfully pulled image "registry.youkeshu.com:5000/mysqld-exporter:v0.10"
      Normal   Created                26s                kubelet, h009028  Created container
      Normal   Started                26s                kubelet, h009028  Started container
    
    root@h009027:~/mytest/ceph/rbd# kubectl get pods -o wide| grep deptest13
    deptest13dbdm-6999f5757c-fk9dp                    2/2       Running            0          3m        172.26.0.87    h009028
    
    #在原node上查看内核中映射的rbd image块设备描述文件已消失,挂载已取消
    root@h020109:~/tpcc-mysql# ls /dev/rbd*
    ls: cannot access '/dev/rbd*': No such file or directory
    root@h020109:~/tpcc-mysql# mount | grep rbd
    root@h020109:~/tpcc-mysql#

**可以看出,是可以自动完成在节点间迁移的滚动更新的,整个调度的过程有些漫长,接近3分钟,这是因为其中至少涉及如下几个步骤:**  
1.原pod终结,释放资源  
2.原node取消挂载rbd,释放内核映射  
3.新node内核映射rbd image,挂载rbd到文件目录  
4.创建pod,挂载指定目录

**结论:**  
ceph rbd不支持多节点挂载,支持滚动更新.

## 后续 ##

在完成了使用cephfs和ceph rbd这两种k8s分布式存储的对接方式测试之后,下一篇文章将针对这两种方式进行数据库性能测试及适用场景讨论

[k8s_Cephfs]: https://blog.csdn.net/ywq935/article/details/82716366
[70]: /images/20220509/4eece12198bc4cf7b3025b0d905e551d.png
[70 1]: /images/20220509/af40c144316c4db19b830d4912e55f11.png
[70 2]: /images/20220509/d46baa153d254df091cb2091665d9b0a.png
[Github issue]: https://github.com/kubernetes/kubernetes/issues/38923
[rbd-provisioner]: https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/deploy/rbac