Centos7下Kubernetes配置使用GlusterFS

比眉伴天荒 2022-06-02 07:44 287阅读 0赞

# 操作环境 #

网络拓扑

![SouthEast][]

环境说明：

OS：

CentOS Linux release 7.4.1708

docker：

Client:
     Version:         1.12.6
     API version:     1.24
     Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
     Go version:      go1.8.3
     Git commit:      ec8512b/1.12.6
     Built:           Mon Dec 11 16:08:42 2017
     OS/Arch:         linux/amd64
    
    Server:
     Version:         1.12.6
     API version:     1.24
     Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
     Go version:      go1.8.3
     Git commit:      ec8512b/1.12.6
     Built:           Mon Dec 11 16:08:42 2017
     OS/Arch:         linux/amd64

kubernetes：

Kubernetes v1.5.2

glusterfs：

glusterfs 3.12.3

# 操作步骤 #

Kubernetes环境已经完成部署完成，部署步骤参照 [ Centos7 下Kubernetes集群安装部署][Centos7 _Kubernetes]。

## 部署Glusterfs ##

这里有5台服务器，10.10.200.226、10.10.200.227作为GlusterFS存储服务器，10.10.200.224、

10.10.200.229、10.10.200.230调用GlusterFS，同样也需要安装Gluster。

1.分别在5台服务器上安装Gluster

[root@k8s-master ~]#yum -y install centos-release-gluster
    [root@k8s-master ~]#yum -y install glusterfs glusterfs-fuse glusterfs-server

2.启动Glusterfs

[root@k8s-master ~]#systemctl enable glusterd
    [root@k8s-master ~]#systemctl start glusterd

3.创建GlusterFS Cluster

编辑/etc/hosts文件

[root@k8s-master ~]# vi /etc/hosts
    127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
    ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
    10.10.200.226   glusterfs-node1
    10.10.200.227   glusterfs-node2
    10.10.200.224   k8s-master
    10.10.200.229   k8s-node1
    10.10.200.230   k8s-node2

选择1台服务器添加其他服务器

[root@gluster-node1 ~]#gluster peer probe k8s-master
    [root@gluster-node1 ~]#gluster peer probe k8s-node1
    [root@gluster-node1 ~]#gluster peer probe k8s-node2
    [root@gluster-node1 ~]#gluster peer probe gluster-node2

查看gluster节点状态

[root@k8s-master yaml]# gluster peer status
    Number of Peers: 4
    
    Hostname: glusterfs-node2
    Uuid: 3c5c5994-0c05-4be0-9ec3-a431f193a1f0
    State: Peer in Cluster (Connected)
    
    Hostname: glusterfs-node1
    Uuid: 31ac3dff-94a3-4166-991b-bf4727113539
    State: Peer in Cluster (Connected)
    
    Hostname: 10.10.200.229
    Uuid: 7c2f03fe-e2cb-412e-a91f-2baf873610ef
    State: Peer in Cluster (Connected)
    Other names:
    10.10.200.229
    
    Hostname: 10.10.200.230
    Uuid: 314a7676-d1eb-4c58-9617-8c510c93961d
    State: Peer in Cluster (Connected)

3.创建volume

这里只选择10.10.200.226/10.10.200.227来进行创建volume

[root@gluster-node1 brick1]# gluster volume create mysql-volume replica 2 glusterfs-node2:/data/brick1/mysql-volume/ glusterfs-node1:/data/brick1/mysql-volume/

启动mysql-volume

[root@gluster-node1 brick1]# gluster volume start mysql-volume
    volume start: mysql-volume: success

查看volume状态

[root@gluster-node1 brick1]# gluster volume status
    Status of volume: mysql-volume
    Gluster process                             TCP Port  RDMA Port  Online  Pid
    ------------------------------------------------------------------------------
    Brick glusterfs-node2:/data/brick1/mysql-vo
    lume                                        49152     0          Y       2976 
    Brick glusterfs-node1:/data/brick1/mysql-vo
    lume                                        49152     0          Y       7835 
    Self-heal Daemon on localhost               N/A       N/A        Y       7856 
    Self-heal Daemon on k8s-master              N/A       N/A        Y       2025 
    Self-heal Daemon on glusterfs-node2         N/A       N/A        Y       2997 
     
    Task Status of Volume mysql-volume
    ------------------------------------------------------------------------------
    There are no active volume tasks

到这里就完成了2台Glusterfs存储服务器的配置

## 配置Kubernetes ##

可以在https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/glusterfs上面找

到kuernetes配置gluster的文件glusterfs-endpoint.json/glusterfs-pod.json/glusterfs-services.json。

这里只需要glusterfs-endpoint.json，glusterfs-service.json。

修改glusterfs-endpoint.json文件如下，主要是ip地址的修改

[root@k8s-master yaml]# vi glusterfs-endpoints.json 
    {
      "kind": "Endpoints",
      "apiVersion": "v1",
      "metadata": {
        "name": "glusterfs-cluster"
      },
      "subsets": [
        {
          "addresses": [
            {
              "ip": "10.10.200.226"
            }
          ],
          "ports": [
            {
              "port": 1
            }
          ]
        },
        {
          "addresses": [
            {
              "ip": "10.10.200.227"
            }
          ],
          "ports": [
            {
              "port": 1
            }
          ]
        }
      ]
    }

glusterfs-service.json文件如下，不需修改

[root@k8s-master yaml]# vi glusterfs-service.json 
    {
      "kind": "Service",
      "apiVersion": "v1",
      "metadata": {
        "name": "glusterfs-cluster"
      },
      "spec": {
        "ports": [
          {"port": 1}
        ]
      }
    }

启用上述两文件

[root@k8s-master yaml]# kubectl apply -f glusterfs-endpoints.json 
    endpoints "glusterfs-cluster" created
    [root@k8s-master yaml]# kubectl apply -f glusterfs-service.json 
    service "glusterfs-cluster" created

[root@k8s-master yaml]# kubectl get ep
    NAME                ENDPOINTS                         AGE
    glusterfs-cluster   10.10.200.226:1,10.10.200.227:1   31s
    kubernetes          10.10.200.224:6443                6d
    [root@k8s-master yaml]# kubectl get service
    NAME                CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    glusterfs-cluster   172.17.62.135   <none>        1/TCP     33s
    kubernetes          10.254.0.1      <none>        443/TCP   6d

创建presistent volume以及presistent volume claim yaml文件

[root@k8s-master yaml]# vi glusterfs-pv.yaml 
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: gluster-dev-volume1
      labels:
        name: mysql1
    
    spec:
      capacity:
        storage: 10Gi
      accessModes:
        - ReadWriteMany
      glusterfs:
        endpoints: "glusterfs-cluster"
        path: "mysql-volume"
        readOnly: false

[root@k8s-master yaml]# vi glusterfs-pvc.yaml 
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: glusterfs-mysql1
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
      selector:
        matchLabels:
          name: "mysql1"

启动上述两文件

[root@k8s-master yaml]# kubectl apply -f glusterfs-pv.yaml 
    persistentvolume "gluster-dev-volume1" created
    [root@k8s-master yaml]# kubectl apply -f glusterfs-pvc.yaml 
    persistentvolumeclaim "glusterfs-mysql1" created
    [root@k8s-master yaml]# kubectl get pv
    NAME                  CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                      REASON    AGE
    gluster-dev-volume1   10Gi       RWX           Retain          Bound     default/glusterfs-mysql1             8s
    [root@k8s-master yaml]# kubectl get pvc
    NAME               STATUS    VOLUME                CAPACITY   ACCESSMODES   AGE
     glusterfs-mysql1 Bound gluster-dev-volume1 10Gi RWX 6s

以上就完成了kubernetes配置glusterfs的全过程，下面我们通过运行mysql测试一下，编写mysql deployment yaml如下

[root@k8s-master yaml]# vi mysql-deployment.yaml 
    apiVersion: v1
    kind: Service
    metadata:
      name: mysql
      labels:
        name: mysql
    spec:
      type: NodePort
      ports:
      - name: mysqlport
        port: 3306
        nodePort: 32006
      selector:
        name: mysql
    ---
    
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: mysql
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            name: mysql
        spec:
          containers:
            - name: mysqlcontainer
              image: docker.io/mysql
              imagePullPolicy: IfNotPresent
              env:
              - name: MYSQL_ROOT_PASSWORD
                value: root123456
              ports:
                - containerPort : 3306
              volumeMounts:
                - name: gluster-mysql-data
                  mountPath: "/var/lib/mysql"
          volumes:
            - name: gluster-mysql-data
              persistentVolumeClaim:
                claimName: glusterfs-mysql1

启动该文件

[root@k8s-master yaml]# kubectl apply -f mysql-deployment.yaml 
    service "mysql" created
    deployment "mysql" configured

查看状态

[root@k8s-master yaml]# kubectl get service
    NAME                CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
    glusterfs-cluster   172.17.62.135    <none>        1/TCP            13m
    kubernetes          10.254.0.1       <none>        443/TCP          6d
    mysql               172.17.231.221   <nodes>       3306:32006/TCP   1m
    [root@k8s-master yaml]# kubectl get pods
    NAME                     READY     STATUS    RESTARTS   AGE
    mysql-3736381796-5j11t   1/1       Running   0          1m
    nginx-controller-1hxkz   1/1       Running   3          3d
    nginx-controller-3xcl8   1/1       Running   3          3d
    [root@k8s-master yaml]# kubectl get pods -o wide
    NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
    mysql-3736381796-5j11t   1/1       Running   0          1m        172.17.99.3   10.10.200.230
    nginx-controller-1hxkz   1/1       Running   3          3d        172.17.99.2   10.10.200.230
    nginx-controller-3xcl8   1/1       Running   3          3d        172.17.17.2   10.10.200.229

登录mysql

[root@k8s-node2 ~]# mysql -h 10.10.200.230 -P 32006 -uroot -proot123456
    Welcome to the MariaDB monitor.  Commands end with ; or \g.
    Your MySQL connection id is 3
    Server version: 5.7.20 MySQL Community Server (GPL)
    
    Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    MySQL [(none)]>

[SouthEast]: /images/20220602/0b808bf9f55e49288725fc652cc155a7.png
[Centos7 _Kubernetes]: http://blog.csdn.net/bobpen/article/details/78958675