k8s高可用集群多个主节点_RabbitMQ(k8s)-随手笔记

桃扇骨 2023-01-03 04:58 228阅读 0赞

## 介绍 ##

基于Erlang开发消息队列(AMQP)

在分布式系统中, 有一些功能我们希望能够提高系统稳定性, 比如说支付、订单功能, 服务后移, 长时间操作的功能, 同步数据

我们通过监听数据变化实现功能联动

![928b74d0195408f0441afb46ee92ffd4.png][]

## 特点 ##

### 分布式 ###

各节点互相冗余元数据(erlang.cookie , 队列、交换机、绑定元数据、vhost元数据)实现分布式集群(类似于session), 通过转发达到集群任意节点进入都是集群主节点

可以任意增加节点实现集群水平扩容,但是如果集群是多主节点是不能任意删除节点, 因为实际数据还是存在于各个主节点中, 但是我们可以使用镜像策略来解决这个问题

# 注意使用正则表达式匹配vhost目录
    $ rabbitmqctl set_policy ha-all "^ha." '{"ha-mode":"all"}'

### 可持久化 ###

// 投递消息时设置持久化
    channel.queueDeclare("test", true, false, false, null)
    // 投递模式设置为2
    channel.basicPublish(x, x, MessageProperties.PERSISTENT_TEXT_PLAIN,x)

## 原理 ##

发送消息: Client -> Exchange -Binding-> Queue -> Server

接收消息: Server -> Channe(线程池) -> Client

Binding可用模式为:

1、Direct, 完全匹配绑定的Queue的名称
    2、fanout, 所有Queue
    3、topic, 适用于通配符匹配的Queue

Message由Header(属性集合)和Body组成

可以通过Vhost去实现多租户(权限)

客户端可以通过设置AutoACK=false去手动应答以实现更加稳定的数据处理请求, 例如推送到死信队列

## 对比 ##

相比于Kafka, RabbitMQ更加稳定, 适用于复杂业务场景而非数据传递

相比于ActiveMQ,RabbitMQ更高性能更丰富, 更适用于云原生

相比于RocketMQ, 更加国际化, 更加方便

## 在K8S中搭建RabbitMQ ##

参考 https://github.com/rabbitmq/rabbitmq-peer-discovery-k8s

核心基于Kubernetes Discovery 插件(rabbitmq\_peer\_discovery\_k8s)

在使用 StatefulSet 时使用 主机 访问即可

## Namespace ##

apiVersion: v1
    kind: Namespace
    metadata:
      name: middleware
      labels:
        name: middleware

## ConfigMap ##

---
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rabbitmq-config
      namespace: middleware
    data:
      enabled_plugins: |
        [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_shovel,rabbitmq_shovel_management,accept,prometheus,prometheus_httpd,prometheus_rabbitmq_exporter,prometheus_process_collector].
      rabbitmq.conf: |
        cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
        cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
        cluster_formation.k8s.address_type = hostname
        cluster_formation.node_cleanup.interval = 10
        cluster_formation.node_cleanup.only_log_warning = true
        cluster_partition_handling = autoheal
        queue_master_locator=min-masters
        loopback_users.guest = false
        cluster_formation.randomized_startup_delay_range.min = 0
        cluster_formation.randomized_startup_delay_range.max = 2
        cluster_formation.k8s.service_name = rabbitmq-headless
        cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq.svc.cluster.local
        vm_memory_high_watermark.absolute = 3.2GB
        disk_free_limit.absolute = 10GB
        default_user = test
        default_pass = test

配置参数讲解:

## Service ##

---
    kind: Service
    apiVersion: v1
    metadata:
      name: rabbitmq-headless
      namespace: middleware
    spec:
      clusterIP: None
      publishNotReadyAddresses: true
      ports:
        - name: amqp
          port: 5672
        - name: http
          port: 15672
      selector:
        app: rabbitmq
    ---
    kind: Service
    apiVersion: v1
    metadata:
      namespace: middleware
      name: rabbitmq-service
    spec:
      type: NodePort
      ports:
        - name: http
          protocol: TCP
          port: 15672
          targetPort: 15672
        - name: amqp
          protocol: TCP
          port: 5672
      selector:
        app: rabbitmq

## RBAC ##

---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: rabbitmq
      namespace: middleware
    ---
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: endpoint-reader
      namespace: middleware
    rules:
      - apiGroups: [""]
        resources: ["endpoints"]
        verbs: ["get"]
    ---
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: endpoint-reader
      namespace: middleware
    subjects:
      - kind: ServiceAccount
        name: rabbitmq
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: endpoint-reader

## StatefulSet ##

---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: rabbitmq
      namespace: middleware
    spec:
      serviceName: rabbitmq-headless
      selector:
        matchLabels:
          app: rabbitmq
      replicas: 1
      template:
        metadata:
          labels:
            app: rabbitmq
          annotations:
            scheduler.alpha.kubernetes.io/affinity: >
              {
                "podAntiAffinity": {
                  "requiredDuringSchedulingIgnoredDuringExecution": [{
                    "labelSelector": {
                      "matchExpressions": [{
                        "key": "app",
                        "operator": "In",
                        "values": ["rabbitmq"]
                      }]
                    },
                    "topologyKey": "kubernetes.io/hostname"
                  }]
                }
              }
        spec:
          nodeSelector:
            middleware: "middleware"
          serviceAccountName: rabbitmq
          terminationGracePeriodSeconds: 10
          containers:
            - name: rabbitmq
              image: registry-vpc.cn-shenzhen.aliyuncs.com/heygears/rabbitmq:3.7
              resources:
                limits:
                  cpu: 0.5
                  memory: 2Gi
                requests:
                  cpu: 0.3
                  memory: 2Gi
              volumeMounts:
                - name: config-volume
                  mountPath: /etc/rabbitmq
                  readOnly: true
                - name: rabbitmq-data
                  mountPath: /var/lib/rabbitmq/mnesia
                  subPath: rabbitmq-ha
              ports:
                - name: http
                  protocol: TCP
                  containerPort: 15672
                - name: amqp
                  protocol: TCP
                  containerPort: 5672
              livenessProbe:
                exec:
                  command: ["rabbitmqctl", "status"]
                initialDelaySeconds: 60
                periodSeconds: 60
                timeoutSeconds: 5
              readinessProbe:
                exec:
                  command: ["rabbitmqctl", "status"]
                initialDelaySeconds: 20
                periodSeconds: 60
                timeoutSeconds: 5
              imagePullPolicy: Always
              env:
                - name: HOSTNAME
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: RABBITMQ_USE_LONGNAME
                  value: "true"
                - name: RABBITMQ_NODENAME
                  value: "rabbit@$(HOSTNAME).rabbitmq-headless.middleware.svc.cluster.local"
                - name: RABBITMQ_ERLANG_COOKIE
                  value: "mycookie"
          volumes:
            - name: config-volume
              configMap:
                name: rabbitmq-config
                items:
                  - key: rabbitmq.conf
                    path: rabbitmq.conf
                  - key: enabled_plugins
                    path: enabled_plugins
            - name: rabbitmq-data
              persistentVolumeClaim:
                claimName: nfs-pvc-middleware

## PV/PVC ##

---
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      namespace: middleware
      name: nfs-pv-middleware
      labels:
        app: nfs
    spec:
      capacity:
        storage: 500Gi
      accessModes:
        - ReadWriteMany
      mountOptions:
        - vers=3
        - nolock,tcp,noresvport
      nfs:
        path: /
        server: xxx.nas.aliyuncs.com
    
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      namespace: middleware
      name: nfs-pvc-middleware
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 100Gi
      selector:
        matchLabels:
          app: nfs

IngressRoute

---
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name: rabbitmq-ingress-route
      namespace: middleware
    spec:
      entryPoints:
        - web
      routes:
        - match: Host(`xxx.com`) && PathPrefix(`/`)
          kind: Rule
          services:
            - name: rabbitmq-service
              port: 15672

[928b74d0195408f0441afb46ee92ffd4.png]: /images/20221119/92f5bce053f14aef80cdd3ee6b4399dd.png