基于Kubernetes的框架和应用系列之四：在Kubernetes安装基于Ceph PVC的mysql galera集群

深藏阁楼爱情的钟 2022-01-15 04:23 344阅读 0赞

### 一、前言 ###

mysql作为互联网LAMP技术栈中最老最坚固的基石之一，具有[众多的集群解决方案][Link 1]，而galera是其中比较优秀的方案之一：

![âmysql galeraâçå¾çæç´¢ç»æ][mysql galera]

**Galera的特点：**

*  多主架构：真正的多点读写的集群，在任何时候读写数据，都是最新的。
 *  同步复制：集群不同节点之间数据实时同步复制，在数据库挂掉之后，数据不会丢失。
 *  并发复制：支持并行复制数据，更好的性能

![1620][]![uploading.4e448015.gif][]转存失败重新上传取消![1620][]

**Galera缺点：**

由于同一个事务需要在集群的多台机器上同时执行成功，这个事务才算成功，网络传输及并发执行会导致性能损失，节点中的最大的tps值取决于速度最慢的那个节点，所以各个节点的处理能力要尽量平衡。

目前基于Galera Cluster的实现方案有三种，较成熟、应用案例较多的是Percona XtraDB Cluster：

*  Galera Cluster for MySQL
 *  Percona XtraDB Cluster
 *  MariaDB Galera Cluster

本文介绍使用Kubernetes statefulset部署Galera集群并且使用基于Ceph的PVC来持久化mysql数据

转载自https://blog.csdn.net/cloudvtech

### 二、部署集群 ###

**2.1 准备工作**

创建namespace

kubectl create ns galera

创建Ceph StorageClass的access key

kubectl -n ceph get secrets/pvc-ceph-client-key -o json | jq '.metadata.namespace = "galera"' | kubectl create -f -
    kubectl get secrets -n galera
    NAME                  TYPE                                  DATA      AGE
    default-token-np974   kubernetes.io/service-account-token   3         18m
    pvc-ceph-client-key   kubernetes.io/rbd

**2.2 部署文件**

galera-service.yaml，提供集群的访问接口

apiVersion: v1
    kind: Service
    metadata:
      annotations:
        service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
      name: galera
      namespace: galera
      labels:
        app: mysql
    spec:
      ports:
      - port: 3306
        name: mysql
      # *.galear.default.svc.cluster.local
      clusterIP: None
      selector:
        app: mysql

galera-statefulset.yaml，部署集群statfulset POD

apiVersion: apps/v1beta1
    kind: StatefulSet
    metadata:
      name: mysql
      namespace: galera
    spec:
      serviceName: "galera"
      replicas: 5
      template:
        metadata:
          labels:
            app: mysql
        spec:
          initContainers:
          - name: install
            image: gcr.io/google_containers/galera-install:0.1
            imagePullPolicy: Always
            args:
            - "--work-dir=/work-dir"
            volumeMounts:
            - name: workdir
              mountPath: "/work-dir"
            - name: config
              mountPath: "/etc/mysql"
          - name: bootstrap
            image: debian:jessie
            command:
            - "/work-dir/peer-finder"
            args:
            - -on-start="/work-dir/on-start.sh"
            - "-service=galera"
            env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            volumeMounts:
            - name: workdir
              mountPath: "/work-dir"
            - name: config
              mountPath: "/etc/mysql"
          containers:
          - name: mysql
            image: gcr.io/google_containers/mysql-galera:e2e
            ports:
            - containerPort: 3306
              name: mysql
            - containerPort: 4444
              name: sst
            - containerPort: 4567
              name: replication
            - containerPort: 4568
              name: ist
            args:
            - --defaults-file=/etc/mysql/my-galera.cnf
            - --user=root
            readinessProbe:
              # TODO: If docker exec is buggy just use gcr.io/google_containers/mysql-healthz:1.0
              exec:
                command:
                - sh
                - -c
                - "mysql -u root -e 'show databases;'"
              initialDelaySeconds: 15
              timeoutSeconds: 5
              successThreshold: 2
            volumeMounts:
            - name: datadir
              mountPath: /var/lib/
            - name: config
              mountPath: /etc/mysql
          volumes:
          - name: config
            emptyDir: {}
          - name: workdir
            emptyDir: {}
      volumeClaimTemplates:
      - metadata:
          name: datadir
          annotations:
            volume.beta.kubernetes.io/storage-class: "ceph-rbd"
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi

**2.3 查看部署状态**

PODs

kubectl get pods  -n galera
    NAME      READY     STATUS    RESTARTS   AGE
    mysql-0   1/1       Running   0          18h
    mysql-1   1/1       Running   0          18h
    mysql-2   1/1       Running   0          18h
    mysql-3   1/1       Running   0          18h
    mysql-4   1/1       Running   0          17h

PVC

kubectl get pv | grep galera
    pvc-2d3d82aa-8cf0-11e9-b430-fa163ebda1b8   1Gi        RWO            Delete           Bound         galera/datadir-mysql-2                                                      ceph-rbd                 18h
    pvc-4bf3908d-8cf0-11e9-b430-fa163ebda1b8   1Gi        RWO            Delete           Bound         galera/datadir-mysql-3                                                      ceph-rbd                 18h
    pvc-5036f0e1-8cf2-11e9-b430-fa163ebda1b8   1Gi        RWO            Delete           Bound         galera/datadir-mysql-4                                                      ceph-rbd                 17h
    pvc-7bf588d7-8cef-11e9-b430-fa163ebda1b8   1Gi        RWO            Delete           Bound         galera/datadir-mysql-0                                                      ceph-rbd                 18h
    pvc-9991b84b-8cef-11e9-b430-fa163ebda1b8   1Gi        RWO            Delete           Bound         galera/datadir-mysql-1                                                      ceph-rbd                 18h

查看galera配置

>     cat /etc/mysql/my-galera.cnf 
>     [mysqld]
>     user = mysql
>     bind-address = 0.0.0.0
>     wsrep_provider = /usr/lib/galera/libgalera_smm.so
>     # TODO: is rsync the best option?
>     wsrep_sst_method = rsync
>     default_storage_engine = innodb
>     binlog_format = row
>     innodb_autoinc_lock_mode = 2
>     innodb_flush_log_at_trx_commit = 0
>     query_cache_size = 0
>     query_cache_type = 0
>     
>     # By default every node is standalone
>     wsrep_cluster_address=gcomm://mysql-0.galera.galera.svc.cluster.local,mysql-1.galera.galera.svc.cluster.local,mysql-2.galera.galera.svc.cluster.local,mysql-3.galera.galera.svc.cluster.local,mysql-4.galera.galera.svc.cluster.local
>     wsrep_cluster_name=mysql
>     wsrep_node_address=mysql-4.galera.galera.svc.cluster.local
>     
>     # TODO: Enable use privileges. This doesn't work
>     # on mysql restart, for some reason after the SST
>     # permissions are not setup correctly.
>     skip-grant-tables

查看galera集群状态

转载自https://blog.csdn.net/cloudvtech

### 三、连接使用集群 ###

**3.1 运行客户端**

mysql-client.yaml

apiVersion: v1
    kind: Pod
    metadata:
      name: mysql-client
      labels:
        app: mysql-client
    spec:
      containers:
      - image: mysql:latest
        command:
          - sleep
          - "3600"
        imagePullPolicy: IfNotPresent
        name: mysql-client
      restartPolicy: Always

**3.2 连接数据库创建表galera**

[root@k8s-master-01 ~]# kubectl exec -it mysql-client sh
    # mysql -h galera -uroot -p
    Enter password: 
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 6560
    Server version: 5.6.28-1trusty (Ubuntu), wsrep_25.13
    
    Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
    
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | mysql              |
    | performance_schema |
    | test               |
    +--------------------+
    4 rows in set (0.00 sec)
    
    mysql> create database galera;
    Query OK, 1 row affected (0.02 sec)

**3.3 在其他节点查看同步结果**

mysql-0

# mysql -h mysql-0.galera.galera.svc.cluster.local -uroot -p
    Enter password: 
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 6591
    Server version: 5.6.28-1trusty (Ubuntu), wsrep_25.13
    
    Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
    
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | galera             |
    | mysql              |
    | performance_schema |
    | test               |
    +--------------------+
    5 rows in set (0.00 sec)

mysql-4

root@mysql-3:/# mysql -h mysql-4.galera.galera.svc.cluster.local -uroot -p
    Enter password: 
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 6484
    Server version: 5.6.28-1trusty (Ubuntu), wsrep_25.13
    
    Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
    
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | information_schema |
    | galera             |
    | mysql              |
    | performance_schema |
    | test               |
    +--------------------+
    5 rows in set (0.00 sec)

转载自https://blog.csdn.net/cloudvtech

[Link 1]: https://blog.csdn.net/xuheng8600/article/details/79846230
[mysql galera]: /images/20220115/448c6a359ca34615a5bc34f5edee4fd8.png
[1620]: https://ask.qcloudimg.com/http-save/developer-news/8vreq7o819.jpeg?imageView2/2/w/1620
[uploading.4e448015.gif]: https://csdnimg.cn/release/mp/img/uploading.4e448015.gif