在Kubernetes集群上部署redis集群

左手的ㄟ右手 2022-05-10 10:58 594阅读 0赞

一、实验环境

操作系统：CentOS Linux release 7.2.1511 (Core)

GlusterFS： 4.1

redis：4.9.105

  1. # docker pull redis:5.0-rc5-alpine3.8  // 官方镜像
  2. # docker pull racccosta/redis    // redis集群构建工具

docker版本

  1. # docker version
  2. Client:
  3.  Version:      1.12.3
  4.  API version:  1.24
  5.  Go version:   go1.6.3
  6.  Git commit:   6b644ec
  7.  Built:        
  8.  OS/Arch:      linux/amd64

二、方案思路

每个Master都可以拥有多个Slave。当Master下线后，Redis集群会从多个Slave中选举出一个新的Master作为替代，而旧Master重新上线后变成新Master的Slave。

image

三、部署过程

  1. 创建GlusterFS存储
  2. 创建PV

创建GlusterFS存储

打算创建一个6节点的Redis集群，所以共享了6个目录。

  1. #!/bin/bash
  2. for loop in 0 1 2 3 4 5
  3. do
  4. gluster volume create redis-vol-$loop replica 3 paasm1:/dcos/redis-brick/pv$loop paasm2:/dcos/redis-brick/pv$loop paashar:/dcos/redis-brick/pv$loop;
  5. gluster volume start redis-vol-$loop;
  6. done

创建GlusterFS端点定义

  1. # vi glusterfs-endpoints.yml
  3. ---
  4. kind: Endpoints
  5. apiVersion: v1
  6. metadata:
  7.   name: glusterfs-cluster
  8.   namespace: kube-system
  9. subsets:
  10. - addresses:
  11.   - ip: 10.142.71.120
  12.   ports:
  13.   - port: 7096
  14. - addresses:
  15.   - ip: 10.142.71.121
  16.   ports:
  17.   - port: 7096
  18. - addresses:
  19.   - ip: 10.142.71.123
  20.   ports:
  21.   - port: 7096
  23. # kubectl apply -f glusterfs-endpoints.yml

配置 service

  1. # vi glusterfs-service.yml
  3. ---
  4. kind: Service
  5. apiVersion: v1
  6. metadata:
  7.   name: glusterfs-cluster
  8.   namespace: kube-system
  9. spec:
  10.   ports:
  11.   - port: 7096
  13. # kubectl apply -f glusterfs-service.yml

创建PV

每一个Redis Pod都需要一个独立的PV来存储自己的数据，因此包含6个PV：

  1. # vi glusterfs-pv.yml
  3. apiVersion: v1
  4. kind: PersistentVolume
  5. metadata:
  6.   name: pv000
  7.   namespace: kube-system
  8. spec:
  9.   capacity:
  10.     storage: 2Gi
  11.   accessModes:
  12.     - ReadWriteMany
  13.   glusterfs:
  14.     endpoints: "glusterfs-cluster"
  15.     path: "redis-vol-0"
  16.     readOnly: false
  18. ---
  19. apiVersion: v1
  20. kind: PersistentVolume
  21. metadata:
  22.   name: pv001
  23.   namespace: kube-system
  24. spec:
  25.   capacity:
  26.     storage: 2Gi
  27.   accessModes:
  28.     - ReadWriteMany
  29.   glusterfs:
  30.     endpoints: "glusterfs-cluster"
  31.     path: "redis-vol-1"
  32.     readOnly: false
  34. ---
  35. apiVersion: v1
  36. kind: PersistentVolume
  37. metadata:
  38.   name: pv002
  39.   namespace: kube-system
  40. spec:
  41.   capacity:
  42.     storage: 2Gi
  43.   accessModes:
  44.     - ReadWriteMany
  45.   glusterfs:
  46.     endpoints: "glusterfs-cluster"
  47.     path: "redis-vol-2"
  48.     readOnly: false
  51. ---
  52. apiVersion: v1
  53. kind: PersistentVolume
  54. metadata:
  55.   name: pv003
  56.   namespace: kube-system
  57. spec:
  58.   capacity:
  59.     storage: 2Gi
  60.   accessModes:
  61.     - ReadWriteMany
  62.   glusterfs:
  63.     endpoints: "glusterfs-cluster"
  64.     path: "redis-vol-3"
  65.     readOnly: false
  67. ---
  68. apiVersion: v1
  69. kind: PersistentVolume
  70. metadata:
  71.   name: pv004
  72.   namespace: kube-system
  73. spec:
  74.   capacity:
  75.     storage: 2Gi
  76.   accessModes:
  77.     - ReadWriteMany
  78.   glusterfs:
  79.     endpoints: "glusterfs-cluster"
  80.     path: "redis-vol-4"
  81.     readOnly: false
  83. ---
  84. apiVersion: v1
  85. kind: PersistentVolume
  86. metadata:
  87.   name: pv005
  88.   namespace: kube-system
  89. spec:
  90.   capacity:
  91.     storage: 2Gi
  92.   accessModes:
  93.     - ReadWriteMany
  94.   glusterfs:
  95.     endpoints: "glusterfs-cluster"
  96.     path: "redis-vol-5"
  97.     readOnly: false
  99. # kubectl apply -f glusterfs-pv.yml

创建Headless service

  1. # vi redis-service.yml
  3. apiVersion: v1
  4. kind: Service
  5. metadata:
  6.   name: redis-service
  7.   namespace: kube-system
  8.   labels:
  9.     app: redis
  10. spec:
  11.   ports:
  12.   - name: redis-port
  13.     port: 6379
  14.   clusterIP: None
  15.   selector:
  16.     app: redis
  17.     appCluster: redis-cluster
  19. # kubectl apply -f redis-service.yml

创建Redis 集群

  1. # vi redis.yml
  3. apiVersion: apps/v1beta1
  4. kind: StatefulSet
  5. metadata:
  6.   name: redis-app
  7.   namespace: kube-system
  8. spec:
  9.   serviceName: "redis-service"
  10.   replicas: 6
  11.   template:
  12.     metadata:
  13.       labels:
  14.         app: redis
  15.         appCluster: redis-cluster
  16.     spec:
  17.       terminationGracePeriodSeconds: 20
  18.       affinity:
  19.         podAntiAffinity:
  20.           preferredDuringSchedulingIgnoredDuringExecution:
  21.           - weight: 100
  22.             podAffinityTerm:
  23.               labelSelector:
  24.                 matchExpressions:
  25.                 - key: app
  26.                   operator: In
  27.                   values:
  28.                   - redis
  29.               topologyKey: kubernetes.io/hostname
  30.       containers:
  31.       - name: redis
  32.         image: hub.cmss.com:5000/registry.paas/library/redis:5.0
  33.         command:
  34.           - "redis-server"
  35.         args:
  36.           - "--protected-mode"
  37.           - "no"
  38.           - "--cluster-enabled"
  39.           - "yes"
  40.           - "--appendonly"
  41.           - "yes"
  42.         resources:
  43.           requests:
  44.             cpu: "100m"
  45.             memory: "100Mi"
  46.         ports:
  47.             - name: redis
  48.               containerPort: 6379
  49.               protocol: "TCP"
  50.             - name: cluster
  51.               containerPort: 16379
  52.               protocol: "TCP"
  53.         volumeMounts:
  54.           - name: "redis-data"
  55.             mountPath: "/data"
  56.   volumeClaimTemplates:
  57.   - metadata:
  58.       name: redis-data
  59.     spec:
  60.       accessModes: ["ReadWriteMany"]
  61.       resources:
  62.         requests:
  63.           storage: 2Gi
  65. # kubectl apply -f redis.yml

总共创建了6个Redis节点(Pod)，其中3个将用于master，另外3个分别作为master的slave

Redis的数据存储路径使用volumeClaimTemplates声明, 其会绑定到我们先前创建的PV上

  1. # kubectl get pv -n kube-system

可以看到之前创建的6个PV都被绑定

  1. NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                STORAGECLASS   REASON    AGE
  2. pv000     2Gi        RWX            Retain           Bound     kube-system/redis-data-redis-app-2                            1h
  3. pv001     2Gi        RWX            Retain           Bound     kube-system/redis-data-redis-app-3                            1h
  4. pv002     2Gi        RWX            Retain           Bound     kube-system/redis-data-redis-app-4                            1h
  5. pv003     2Gi        RWX            Retain           Bound     kube-system/redis-data-redis-app-5                            1h
  6. pv004     2Gi        RWX            Retain           Bound     kube-system/redis-data-redis-app-0                            1h
  7. pv005     2Gi        RWX            Retain           Bound     kube-system/redis-data-redis-app-1                            1h

初始化Redis集群

为了方便redis-trib的使用, 可以创建 /usr/local/bin/redis-trib 文件

  1. #!/bin/bash
  2. /usr/bin/docker run -it --privileged --rm \
  3. --net=host \
  4. redis-trib:latest \
  5. "$@"
  7. # chmod 777 /usr/local/bin/redis-trib
  9. 直接使用
  10. # redis-trib
  12. Usage: redis-trib <command> <options> <arguments ...>
  14.   create          host1:port1 ... hostN:portN
  15.                   --replicas <arg>
  16.   check           host:port
  17.   info            host:port
  18.   fix             host:port
  19.                   --timeout <arg>
  20.   reshard         host:port
  21.                   --from <arg>
  22.                   --to <arg>
  23.                   --slots <arg>
  24.                   --yes
  25.                   --timeout <arg>
  26.                   --pipeline <arg>
  27.   rebalance       host:port
  28.                   --weight <arg>
  29.                   --auto-weights
  30.                   --use-empty-masters
  31.                   --timeout <arg>
  32.                   --simulate
  33.                   --pipeline <arg>
  34.                   --threshold <arg>
  35.   add-node        new_host:new_port existing_host:existing_port
  36.                   --slave
  37.                   --master-id <arg>
  38.   del-node        host:port node_id
  39.   set-timeout     host:port milliseconds
  40.   call            host:port command arg arg .. arg
  41.   import          host:port
  42.                   --from <arg>
  43.                   --copy
  44.                   --replace
  45.   help            (show this help)
  47. For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.

查看起的应用

  1. # kubectl get po -n kube-system -o wide |grep redis
  3. redis-app-0                                1/1       Running   0          53s       10.222.98.88    paasm3
  4. redis-app-1                                1/1       Running   0          50s       10.222.88.205   paasm2
  5. redis-app-2                                1/1       Running   0          47s       10.222.65.206   paasm1
  6. redis-app-3                                1/1       Running   0          45s       10.222.92.210   paasing
  7. redis-app-4                                1/1       Running   0          41s       10.222.66.47    paashar
  8. redis-app-5                                1/1       Running   0          38s       10.222.88.206   paasm2

利用redis-trib创建集群

添加 master 节点

  1. # redis-trib create --replicas 1 \
  2.     10.222.98.90:6379 \
  3.     10.222.88.208:6379 \
  4.     10.222.65.208:6379 \
  5.     10.222.92.213:6379 \
  6.     10.222.66.49:6379 \
  7.     10.222.88.209:6379

replicas 1的意思，就是每个节点创建1个副本(即：slave)

创建用于访问Service

前面我们创建了用于实现StatefulSet的Headless Service，但该Service没有Cluster Ip，因此不能用于外界访问。所以，我们还需要创建一个Service，专用于为Redis集群提供访问和负载均：

  1. # vi redis-access-service.yml
  3. apiVersion: v1
  4. kind: Service
  5. metadata:
  6.   name: redis-access-service
  7.   namespace: kube-system
  8.   labels:
  9.     app: redis
  10. spec:
  11.   ports:
  12.   - name: redis-port
  13.     protocol: "TCP"
  14.     port: 6379
  15.     targetPort: 6379
  16.   selector:
  17.     app: redis
  18.     appCluster: redis-cluster
  20. # kubectl apply -f redis-access-service.yml
  22. # kubectl get svc -n kube-system |grep redis-access
  24. redis-access-service       ClusterIP   10.233.11.2     <none>        6379/TCP         16h

此时在k8s集群中可以通过10.233.11.2:6379 访问redis

检查集群状态

  1. # redis-trib check 10.233.11.2:6379
  3. M: 7a2984e16a9911296918822eea625e9d69e5d062 10.233.11.2:6379
  4.    slots:5461-10922 (5462 slots) master
  5.    1 additional replica(s)
  6. S: 38e2c1e93aa3fc792b0c1ae4c7c9fe71a6713a0f 10.222.66.49:6379
  7.    slots: (0 slots) slave
  8.    replicates 7a2984e16a9911296918822eea625e9d69e5d062
  9. S: 9f13a91cdc81f102f1f05ef4ac7ecfd7019ea1ca 10.222.88.209:6379
  10.    slots: (0 slots) slave
  11.    replicates fe586bc12346bd26548f9dc909ff9c2309b6db0a
  12. M: fe586bc12346bd26548f9dc909ff9c2309b6db0a 10.222.65.208:6379
  13.    slots:10923-16383 (5461 slots) master
  14.    1 additional replica(s)
  15. S: 7d8f97ab405c1cb281f19feaba439b05f7e51709 10.222.92.213:6379
  16.    slots: (0 slots) slave
  17.    replicates c275b4cd089b000c70bcf21f6fb6f6a6f3dbb0a3
  18. M: c275b4cd089b000c70bcf21f6fb6f6a6f3dbb0a3 10.222.98.90:6379
  19.    slots:0-5460 (5461 slots) master
  20.    1 additional replica(s)
  21. [OK] All nodes agree about slots configuration.
  22. >>> Check for open slots...
  23. >>> Check slots coverage...
  24. [OK] All 16384 slots covered.

参考： https://www.jianshu.com/p/65c4baadf5d9

发表评论

表情：
评论列表 （有 0 条评论，594人围观）

还没有评论，来说两句吧...

相关阅读