k8s上使用operator部署redis集群

小灰灰 2022-11-02 13:21 864阅读 0赞

通过operator部署redis集群

operator部署有状态的应用会简单很多，GitHub已有很多写好的operator：https://github.com/operator-framework/awesome-operators
这里直接参考ucloud的operator部署redis集群：https://github.com/ucloud/redis-cluster-operator
Redis Cluster Operator在Kubernetes上管理Redis-Cluster集群
在这里插入图片描述
每个主节点及其从节点都由statefulSet管理，为每个statefulSet创建无头svc，并为所有节点创建clusterIP服务。
每个有状态集都使用PodAntiAffinity来确保主节点和从节点分散在不同的节点上。同时，当操作员在每个有状态集中选择主节点时，它会优先选择具有不同k8s节点的容器作为主节点。
从GitHub上下载项目
git clone https://github.com/ucloud/redis-cluster-operator.git
或者直接通过浏览器下载压缩包，上传到k8s集群在这里插入图片描述
在名称空间：redis-cluster下部署Redis集群，注意修改yaml文件的namespace参数
创建名称空间Namespace：redis-cluster

  1. ]# cat ns-redis-cluster.yaml
  2. apiVersion: v1
  3. kind: Namespace
  4. metadata:
  5.   name: redis-cluster
  7. ]# kubectl apply -f ns-redis-cluster.yaml
  8. namespace/redis-cluster created
  9. kubectl get ns
  10. 'NAME                   STATUS   AGE
  11. default                Active   1d
  12. ingress-nginx          Active   1d
  13. kube-node-lease        Active   1d
  14. kube-public            Active   1d
  15. kube-system            Active   1d
  16. kubernetes-dashboard   Active   1d
  17. redis-cluster          Active   18s

创建自定义资源（CRD）

  1. ]# cd redis-cluster-operator/ 
  2. kubectl apply -f deploy/crds/

创建operator

  1. ]# cd deploy/
  2. ]# kubectl apply -f service_account.yaml 
  3. ]# cd cluster/
  4. ]# kubectl apply -f cluster_role.yaml 
  5. ]# kubectl apply -f cluster_role_binding.yaml 
  6. ]# kubectl apply -f operator.yaml 
  7. ]# kubectl get pod -n redis-cluster 
  8. NAME                                     READY   STATUS    RESTARTS   AGE
  9. redis-cluster-operator-d4b695fd6-9vc85   1/1     Running   0          33s

创建redis集群，预先配置了动态PV存储，并设置资源限制
默认创建三主三从集群

  1. ]# cat redis-cluster.yaml 
  2. apiVersion: redis.kun/v1alpha1
  3. kind: DistributedRedisCluster
  4. metadata:
  5.   annotations:
  6.     # if your operator run as cluster-scoped, add this annotations
  7.     redis.kun/scope: cluster-scoped
  8.   name: example-distributedrediscluster
  9.   namespace: redis-cluster
  10. spec:
  11.   image: redis:5.0.4-alpine
  12.   imagePullPolicy: IfNotPresent
  13.   masterSize: 3            #master节点数量
  14.   clusterReplicas: 1    #每个master节点的从节点数量
  15.   serviceName: redis-svc
  16.   # resources config
  17.   resources:
  18.     limits:
  19.       cpu: 300m
  20.       memory: 200Mi
  21.     requests:
  22.       cpu: 200m
  23.       memory: 150Mi
  24.   # pv storage
  25.   storage:
  26.     type: persistent-claim
  27.     size: 2Gi
  28.     class: nfs-storage
  29.     deleteClaim: true
  30. ]# kubectl apply -f redis-cluster.yaml

查看创建的pod，svc，pvc，pv

  1. ]# kubectl get pod,svc,pvc,pv -n redis-cluster 
  2. NAME                                          READY   STATUS    RESTARTS   AGE
  3. pod/drc-example-distributedrediscluster-0-0   1/1     Running   0          11m
  4. pod/drc-example-distributedrediscluster-0-1   1/1     Running   0          10m
  5. pod/drc-example-distributedrediscluster-1-0   1/1     Running   0          11m
  6. pod/drc-example-distributedrediscluster-1-1   1/1     Running   0          10m
  7. pod/drc-example-distributedrediscluster-2-0   1/1     Running   0          11m
  8. pod/drc-example-distributedrediscluster-2-1   1/1     Running   0          10m
  9. pod/redis-cluster-operator-d4b695fd6-9vc85    1/1     Running   0          17m
  11. NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
  12. service/redis-cluster-operator-metrics   ClusterIP   10.108.239.105   <none>        8383/TCP,8686/TCP    17m
  13. service/redis-svc                        ClusterIP   10.104.55.186    <none>        6379/TCP,16379/TCP   11m
  14. service/redis-svc-0                      ClusterIP   None             <none>        6379/TCP,16379/TCP   11m
  15. service/redis-svc-1                      ClusterIP   None             <none>        6379/TCP,16379/TCP   11m
  16. service/redis-svc-2                      ClusterIP   None             <none>        6379/TCP,16379/TCP   11m
  18. NAME                                                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  19. persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-0-0   Bound    pvc-8da269a1-d704-40ee-9034-59c242229c42   2Gi        RWO            nfs-storage    11m
  20. persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-0-1   Bound    pvc-2f90ccb0-87d2-4152-a292-d672941aa27a   2Gi        RWO            nfs-storage    10m
  21. persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-1-0   Bound    pvc-e34aeac8-a555-42e8-bd6f-d6e84e417780   2Gi        RWO            nfs-storage    11m
  22. persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-1-1   Bound    pvc-22a46a79-965b-41e7-9c5b-e314b158488a   2Gi        RWO            nfs-storage    10m
  23. persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-2-0   Bound    pvc-78640340-27a9-45a9-ae3c-de68405e00d4   2Gi        RWO            nfs-storage    11m
  24. persistentvolumeclaim/redis-data-drc-example-distributedrediscluster-2-1   Bound    pvc-0ef4b886-38c2-43b6-aef3-e78bdd0c71de   2Gi        RWO            nfs-storage    10m
  26. NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                              STORAGECLASS   REASON   AGE
  27. persistentvolume/pvc-0ef4b886-38c2-43b6-aef3-e78bdd0c71de   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-2-1   nfs-storage             10m
  28. persistentvolume/pvc-22a46a79-965b-41e7-9c5b-e314b158488a   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-1-1   nfs-storage             10m
  29. persistentvolume/pvc-2f90ccb0-87d2-4152-a292-d672941aa27a   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-0-1   nfs-storage             10m
  30. persistentvolume/pvc-78640340-27a9-45a9-ae3c-de68405e00d4   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-2-0   nfs-storage             11m
  31. persistentvolume/pvc-8da269a1-d704-40ee-9034-59c242229c42   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-0-0   nfs-storage             11m
  32. persistentvolume/pvc-e34aeac8-a555-42e8-bd6f-d6e84e417780   2Gi        RWO            Retain           Bound    redis-cluster/redis-data-drc-example-distributedrediscluster-1-0   nfs-storage             11m

验证redis集群

  1. ]# kubectl exec -it -n redis-cluster drc-example-distributedrediscluster-0-0 -- sh
  2. /data # redis-cli -c -h redis-svc
  3. redis-svc:6379> cluster info
  4. cluster_state:ok
  5. cluster_slots_assigned:16384
  6. cluster_slots_ok:16384
  7. cluster_slots_pfail:0
  8. cluster_slots_fail:0
  9. cluster_known_nodes:6
  10. cluster_size:3
  11. cluster_current_epoch:5
  12. cluster_my_epoch:5
  13. cluster_stats_messages_ping_sent:225
  14. cluster_stats_messages_pong_sent:237
  15. cluster_stats_messages_sent:462
  16. cluster_stats_messages_ping_received:232
  17. cluster_stats_messages_pong_received:225
  18. cluster_stats_messages_meet_received:5
  19. cluster_stats_messages_received:462
  20. redis-svc:6379> set a 100
  21. -> Redirected to slot [15495] located at 10.244.166.135:6379
  22. OK
  23. 10.244.166.135:6379> get a
  24. "100"

删除redis集群

  1. ]# cd redis-cluster-operator/ 
  2. ]# kubectl delete -f redis-cluster.yaml
  3. ]# cd cluster/
  4. ]# kubectl delete -f operator.yaml 
  5. ]# kubectl delete -f cluster_role_binding.yaml 
  6. ]# kubectl delete -f cluster_role.yaml 
  7. ]# kubectl delete-f service_account.yaml 
  8. ]# kubectl delete -f deploy/crds/
  9. ]# kubectl delete -f ns-redis-cluster.yaml

yaml文件

ns-redis-cluster.yaml

  1. ]# cat ns-redis-cluster.yaml 
  2. apiVersion: v1
  3. kind: Namespace
  4. metadata:
  5.   name: redis-cluster

redis.kun_distributedredisclusters_crd.yaml

  1. ]# cat redis.kun_distributedredisclusters_crd.yaml 
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: distributedredisclusters.redis.kun
  6.   namespace: redis-cluster
  7. spec:
  8.   group: redis.kun
  9.   names:
  10.     kind: DistributedRedisCluster
  11.     listKind: DistributedRedisClusterList
  12.     plural: distributedredisclusters
  13.     singular: distributedrediscluster
  14.     shortNames:
  15.     - drc
  16.   scope: Namespaced
  17.   additionalPrinterColumns:
  18.   - JSONPath: .spec.masterSize
  19.     description: The number of redis master node in the ensemble
  20.     name: MasterSize
  21.     type: integer
  22.   - JSONPath: .status.status
  23.     description: The status of redis cluster
  24.     name: Status
  25.     type: string
  26.   - JSONPath: .metadata.creationTimestamp
  27.     name: Age
  28.     type: date
  29.   - JSONPath: .status.numberOfMaster
  30.     priority: 1
  31.     description: The current master number of redis cluster
  32.     name: CurrentMasters
  33.     type: integer
  34.   - JSONPath: .spec.image
  35.     priority: 1
  36.     description: The image of redis cluster
  37.     name: Images
  38.     type: string
  39.   subresources:
  40.     status: { }
  41.   validation:
  42.     openAPIV3Schema:
  43.       description: DistributedRedisCluster is the Schema for the distributedredisclusters
  44.         API
  45.       properties:
  46.         apiVersion:
  47.           description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
  48.           type: string
  49.         kind:
  50.           description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
  51.           type: string
  52.         metadata:
  53.           type: object
  54.         spec:
  55.           description: DistributedRedisClusterSpec defines the desired state of
  56.             DistributedRedisCluster
  57.           properties:
  58.             masterSize:
  59.               format: int32
  60.               type: integer
  61.               minimum: 3
  62.               maximum: 10
  63.             clusterReplicas:
  64.               format: int32
  65.               type: integer
  66.               minimum: 1
  67.               maximum: 3
  68.             serviceName:
  69.               type: string
  70.               pattern: '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'
  71.           type: object
  72.         status:
  73.           description: DistributedRedisClusterStatus defines the observed state
  74.             of DistributedRedisCluster
  75.           type: object
  76.       type: object
  77.   version: v1alpha1
  78.   versions:
  79.   - name: v1alpha1
  80.     served: true
  81.     storage: true

redis.kun_redisclusterbackups_crd.yaml

  1. ]# cat redis.kun_redisclusterbackups_crd.yaml 
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: redisclusterbackups.redis.kun
  6.   namespace: redis-cluster
  7. spec:
  8.   group: redis.kun
  9.   names:
  10.     kind: RedisClusterBackup
  11.     listKind: RedisClusterBackupList
  12.     plural: redisclusterbackups
  13.     singular: redisclusterbackup
  14.     shortNames:
  15.       - drcb
  16.   scope: Namespaced
  17.   additionalPrinterColumns:
  18.     - JSONPath: .metadata.creationTimestamp
  19.       name: Age
  20.       type: date
  21.     - JSONPath: .status.phase
  22.       description: The phase of redis cluster backup
  23.       name: Phase
  24.       type: string
  25.   subresources:
  26.     status: { }
  27.   versions:
  28.     - name: v1alpha1
  29.       # Each version can be enabled/disabled by Served flag.
  30.       served: true
  31.       # One and only one version must be marked as the storage version.
  32.       storage: true
  33.   validation:
  34.     openAPIV3Schema:
  35.       description: RedisClusterBackup is the Schema for the redisclusterbackups
  36.         API
  37.       properties:
  38.         apiVersion:
  39.           description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources'
  40.           type: string
  41.         kind:
  42.           description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds'
  43.           type: string
  44.         metadata:
  45.           type: object
  46.         spec:
  47.           description: RedisClusterBackupSpec defines the desired state of RedisClusterBackup
  48.           type: object
  49.         status:
  50.           description: RedisClusterBackupStatus defines the observed state of RedisClusterBackup
  51.           type: object
  52.       type: object

service_account.yaml

  1. ]# cat service_account.yaml 
  2. apiVersion: v1
  3. kind: ServiceAccount
  4. metadata:
  5.   name: redis-cluster-operator
  6.   namespace: redis-cluster

cluster_role.yaml

  1. ]# cat cluster_role.yaml 
  2. apiVersion: rbac.authorization.k8s.io/v1
  3. kind: ClusterRole
  4. metadata:
  5.   name: redis-cluster-operator
  6. rules:
  7.   - apiGroups:
  8.       - ""
  9.     resources:
  10.       - pods
  11.       - secrets
  12.       - endpoints
  13.       - persistentvolumeclaims
  14.     verbs:
  15.       - get
  16.       - list
  17.       - watch
  18.       - delete
  19.   - apiGroups:
  20.       - ""
  21.     resources:
  22.       - configmaps
  23.       - pods/exec
  24.       - secrets
  25.       - services
  26.       - events
  27.       - persistentvolumeclaims
  28.     verbs:
  29.       - create
  30.       - get
  31.       - list
  32.       - patch
  33.       - update
  34.       - watch
  35.       - delete
  36.   - apiGroups:
  37.       - ""
  38.     resources:
  39.       - namespaces
  40.     verbs:
  41.       - get
  42.       - list
  43.       - watch
  44.   - apiGroups:
  45.     - batch
  46.     resources:
  47.       - jobs
  48.     verbs:
  49.       - create
  50.       - get
  51.       - list
  52.       - patch
  53.       - update
  54.       - watch
  55.       - delete
  56.   - apiGroups:
  57.       - apps
  58.     resources:
  59.       - deployments
  60.       - replicasets
  61.       - statefulsets
  62.     verbs:
  63.       - create
  64.       - get
  65.       - list
  66.       - patch
  67.       - update
  68.       - watch
  69.       - delete
  70.   - apiGroups:
  71.       - policy
  72.     resources:
  73.       - poddisruptionbudgets
  74.     verbs:
  75.       - create
  76.       - get
  77.       - list
  78.       - patch
  79.       - update
  80.       - watch
  81.       - delete
  82.   - apiGroups:
  83.       - apps
  84.     resourceNames:
  85.       - redis-operator
  86.     resources:
  87.       - deployments/finalizers
  88.     verbs:
  89.       - update
  90.   - apiGroups:
  91.       - redis.kun
  92.     resources:
  93.       - '*'
  94.       - redisclusterbackups
  95.     verbs:
  96.       - delete
  97.       - deletecollection
  98.       - get
  99.       - list
  100.       - patch
  101.       - update
  102.       - watch

cluster_role_binding.yaml

  1. ]# cat cluster_role_binding.yaml 
  2. kind: ClusterRoleBinding
  3. apiVersion: rbac.authorization.k8s.io/v1
  4. metadata:
  5.   name: redis-cluster-operator
  6. subjects:
  7. - kind: ServiceAccount
  8.   name: redis-cluster-operator
  9.   namespace: redis-cluster
  10. roleRef:
  11.   kind: ClusterRole
  12.   name: redis-cluster-operator
  13.   apiGroup: rbac.authorization.k8s.io

operator.yaml

  1. ]# cat operator.yaml 
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: redis-cluster-operator
  6.   namespace: redis-cluster
  7. spec:
  8.   replicas: 1
  9.   selector:
  10.     matchLabels:
  11.       name: redis-cluster-operator
  12.   template:
  13.     metadata:
  14.       labels:
  15.         name: redis-cluster-operator
  16.     spec:
  17.       serviceAccountName: redis-cluster-operator
  18.       containers:
  19.         - name: redis-cluster-operator
  20.           # Replace this with the built image name
  21.           image: fishu/redis-cluster-operator:latest
  22.           imagePullPolicy: IfNotPresent
  23.           command:
  24.           - redis-cluster-operator
  25.           args:
  26.           - --rename-command-path=/etc/redisconf
  27.           - --rename-command-file=redis.conf
  28.           imagePullPolicy: Always
  29.           env:
  30.             - name: WATCH_NAMESPACE
  31.               value: ""
  32.             - name: POD_NAME
  33.               valueFrom:
  34.                 fieldRef:
  35.                   fieldPath: metadata.name
  36.             - name: OPERATOR_NAME
  37.               value: "redis-cluster-operator"
  38.           volumeMounts:
  39.             - name: redisconf
  40.               mountPath: /etc/redisconf
  41.       volumes:
  42.         - name: redisconf
  43.           configMap:
  44.             name: redis-admin
  45. ---
  46. apiVersion: v1
  47. kind: ConfigMap
  48. metadata:
  49.   name: redis-admin
  50.   namespace: redis-cluster
  51. data:
  52.   redis.conf: |-
  53.     rename-command CONFIG lni07z1p
  54.     rename-command BGSAVE pp14qluk
  55.     rename-command DEBUG 8a4insyv
  56.     rename-command SAVE 6on30p6z
  57.     rename-command SHUTDOWN dvui0opr
  58.     rename-command SLAVEOF xwxvcw36
  59.     rename-command BGREWRITEAOF www07fko

redis-cluster.yaml

  1. ]# cat redis-cluster.yaml 
  2. ---
  3. apiVersion: redis.kun/v1alpha1
  4. kind: DistributedRedisCluster
  5. metadata:
  6.   annotations:
  7.     # if your operator run as cluster-scoped, add this annotations
  8.     redis.kun/scope: cluster-scoped
  9.   name: example-distributedrediscluster
  10.   namespace: redis-cluster
  11. spec:
  12.   image: redis:5.0.4-alpine
  13.   imagePullPolicy: IfNotPresent
  14.   masterSize: 3
  15.   clusterReplicas: 1
  16.   serviceName: redis-svc
  17.   # resources config
  18.   resources:
  19.     limits:
  20.       cpu: 300m
  21.       memory: 200Mi
  22.     requests:
  23.       cpu: 200m
  24.       memory: 150Mi
  25.   # pv storage
  26.   storage:
  27.     type: persistent-claim
  28.     size: 2Gi
  29.     class: nfs-storage
  30.     deleteClaim: true

发表评论

表情：
评论列表 （有 0 条评论，864人围观）

还没有评论，来说两句吧...

相关阅读

    相关 k8s部署redis集群

    写在前面 一般来说，REDIS部署有三种模式。 1. 单实例模式，一般用于测试环境。 2. 哨兵模式 3. 集群模式 后两者用于生产部署 `哨兵模式` >

    以你之姓@以你之姓@/ 0 赞/ 143 阅读