Glusterfs 分布式存储安装部署

「爱情、让人受尽委屈。」 2022-10-11 12:27 136阅读 0赞

Glusterfs 分布式存储部署 是存储当中可以选择的一种 现在很多虚拟化 云计算都在用软件存储 例如 ceph

Glusterfs 等等 今天我们部署一下Glusterfs环境

GlusterFs安装

**环境信息：**

<table> 
 <tbody> 
  <tr> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Glusterfs-01 &nbsp; &nbsp; &nbsp;&nbsp;</td> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10.20.0.200 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;</td> 
  </tr> 
  <tr> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Glusterfs-02</td> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp;10.20.0.201</td> 
  </tr> 
  <tr> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Glusterfs-03</td> 
   <td>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10.20.0.202</td> 
  </tr> 
 </tbody> 
</table>

硬盘各一块 8G 测试 /dev/sdb

**服务器版本信息**

# cat /etc/redhat-release 
    CentOS release 6.6 (Final)

\# uname -r  
  2.6.32-504.el6.x86\_64

**软件下载地址**

http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.6/CentOS/epel-6.6/x86_64

**依赖包安装**

直接用Yum安装 每台都需要安装

# yum -y install libibverbs librdmacm xfsprogs nfs-utils rpcbind  libaio liblvm2app lvm2-devel

**软件安装**

安装服务器端软件包

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

glusterfs-3.4.6-1.el6.x86_64.rpm
    glusterfs-api-3.4.6-1.el6.x86_64.rpm
    glusterfs-cli-3.4.6-1.el6.x86_64.rpm
    glusterfs-fuse-3.4.6-1.el6.x86_64.rpm
    glusterfs-libs-3.4.6-1.el6.x86_64.rpm
    glusterfs-server-3.4.6-1.el6.x86_64.rpm

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

安装客户端软件包

glusterfs-3.4.6-1.el6.x86_64.rpm
    glusterfs-fuse-3.4.6-1.el6.x86_64.rpm
    glusterfs-libs-3.4.6-1.el6.x86_64.rpm

**磁盘配置**

磁盘分区：

EXT4格式化工具目前最大支持16TB

使用parted进行磁盘分区

分区格式化  
每个服务器上面都需要至少添加一块硬盘 我们这边是用的1块8G硬盘 现在每台都格式化

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# mkfs.ext4 -L /brick1 /dev/sdb
    mke2fs 1.41.12 (17-May-2010)
    /dev/sdb is entire device, not just one partition!
    Proceed anyway? (y,n) y
    Filesystem label=/brick1
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=0 blocks, Stripe width=0 blocks
    524288 inodes, 2097152 blocks
    104857 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=2147483648
    64 block groups
    32768 blocks per group, 32768 fragments per group
    8192 inodes per group
    Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
    
    Writing inode tables: done                            
    Creating journal (32768 blocks): done
    Writing superblocks and filesystem accounting information: done
    
    This filesystem will be automatically checked every 39 mounts or
    180 days, whichever comes first.  Use tune2fs -c or -i to override.

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

**自动挂载分区**

# vim /etc/fstab 
    LABEL=/brick1  /brick1 ext4  defaults 0 0

**mount文件系统**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# mkdir /brick1
    # mount -L /brick1 /brick1/
    # df -kh
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VolGroup-lv_root
                           47G  759M   44G   2% /
    tmpfs                 491M     0  491M   0% /dev/shm
    /dev/sda1             477M   28M  425M   6% /boot
    /dev/sdb              7.8G   18M  7.4G   1% /brick1

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

基础测试包安装 性能测试包

工具软件  
Atop,iperf,sysstat

dd,lozone,fio,postmark

工具安装

\#rpm -ivh http://mirrors.ustc.edu.cn/fedora/epel/6/x86\_64/epel-release-6-8.noarch.rpm

yum install sysstat

Gcc -o postmark postmark-1.52.c

有些可以用yum安装

**安装GlusterFS软件包**

每台创建存放软件的目录  我们目前安装的是3.4.6的版本

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

#mdkir /home/tools/3.4.6 -p
    #cd /home/tools/3.4.6
    # ll
    total 2164
    -rw-r--r-- 1 root root 1027320 Oct  4 11:13 glusterfs-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root   66680 Oct  4 11:13 glusterfs-api-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root  108872 Oct  4 11:14 glusterfs-cli-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root   88452 Oct  4 11:14 glusterfs-fuse-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root  124288 Oct  4 11:14 glusterfs-geo-replication-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root  227900 Oct  4 11:14 glusterfs-libs-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root   50128 Oct  4 11:14 glusterfs-rdma-3.4.6-1.el6.x86_64.rpm
    -rw-r--r-- 1 root root  504744 Oct  4 11:14 glusterfs-server-3.4.6-1.el6.x86_64.rpm

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

直接rpm安装

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# rpm -ivh *.rpm
    warning: glusterfs-3.4.6-1.el6.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 4ab22bb3: NOKEY
    Preparing...                ########################################### [100%]
       1:glusterfs-libs         ########################################### [ 13%]
       2:glusterfs              ########################################### [ 25%]
       3:glusterfs-fuse         ########################################### [ 38%]
       4:glusterfs-cli          ########################################### [ 50%]
       5:glusterfs-server       ########################################### [ 63%]
       6:glusterfs-geo-replicati########################################### [ 75%]
       7:glusterfs-api          ########################################### [ 88%]
       8:glusterfs-rdma         ########################################### [100%]

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

**安装成功启动GlusterFS**

# service glusterd start
    Starting glusterd:                                         [  OK  ]
    # chkconfig glusterd on

**组建集群扩展**

在GlusterFS-01服务器上面操作 选择10.20.0.200 作为当前集群节点

添加另外两台服务器

# gluster peer probe 10.20.0.201
    peer probe: success
    # gluster peer probe 10.20.0.202
    peer probe: success

**GlusterFS-01服务器查看添加状态**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# gluster peer status
    Number of Peers: 2
    
    Hostname: 10.20.0.201
    Port: 24007
    Uuid: c4d9edfd-d6ae-4628-8ab3-5a9fbb5f18fc
    State: Peer in Cluster (Connected)
    
    Hostname: 10.20.0.202
    Port: 24007
    Uuid: c0b425bb-a7b4-4dae-bc19-ea6031bf02b9
    State: Peer in Cluster (Connected)

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

**GlusterFS-02服务器查看添加状态**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# gluster peer status
    Number of Peers: 2
    
    Hostname: 10.20.0.200
    Port: 24007
    Uuid: 7ad4db05-ee75-44e6-8213-e58072b62d33
    State: Peer in Cluster (Connected)
    
    Hostname: 10.20.0.202
    Uuid: c0b425bb-a7b4-4dae-bc19-ea6031bf02b9
    State: Peer in Cluster (Connected)

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

**GlusterFS-03服务器查看添加状态**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# gluster peer status
    Number of Peers: 2
    
    Hostname: 10.20.0.200
    Port: 24007
    Uuid: 7ad4db05-ee75-44e6-8213-e58072b62d33
    State: Peer in Cluster (Connected)
    
    Hostname: 10.20.0.201
    Uuid: c4d9edfd-d6ae-4628-8ab3-5a9fbb5f18fc
    State: Peer in Cluster (Connected)

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

表示成功

如果添加不成功 排除错误：

1，每台服务器是否能否Ping通

2，防火墙是否关闭iptables selinux

3，glusterd服务是否启动成功

**创建卷**

*  确定创建卷的类型
 *  确定创建卷的brick列表
 *  确实创建卷的网络类型（TCP/RDMA）

Gluster volume create创建卷

在GlusterFS-01服务器上面操作 可以任选一台

# gluster volume create dht-vol1 10.20.0.200:/brick1/b1 10.20.0.201:/brick1/b2 10.20.0.202:/brick1/b
    3
    volume create: dht-vol1: success: please start the volume to access data

(1)distribute volume：分布式卷，文件通过hash算法分布到brick server上，这种卷是glusterfs的基础和最大特点；

(2)stripe volume：条带卷，类似RAID0，条带数=brick server数量，文件分成数据块以Round Robin方式分布到brick server上，并发粒度是数据块，大文件性能高；

(3)replica volume：镜像卷，类似RAID1，镜像数=brick server数量，所以brick server上文件数据相同，构成n-way镜像，可用性高；

(4)distribute stripe volume：分布式条带卷，brick server数量是条带数的倍数，兼具distribute和stripe卷的特点；

(5)distribute replica volume：分布式镜像卷，brick server数量是镜像数的倍数，兼具distribute和replica卷的特点

**查看卷状态信息**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# gluster volume info
     
    Volume Name: dht-vol1
    Type: Distribute
    Volume ID: 7d947cf1-c743-477e-a28d-7d030572b655
    Status: Created
    Number of Bricks: 3
    Transport-type: tcp
    Bricks:
    Brick1: 10.20.0.200:/brick1/b1
    Brick2: 10.20.0.201:/brick1/b2
    Brick3: 10.20.0.202:/brick1/b3

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

**启动卷**

# gluster volume start dht-vol1
    volume start: dht-vol1: success

**客户端挂载卷**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# mkdir /data
    
    # mount -t glusterfs 10.20.0.200:/dth-vol1 /data/
    
    # df -kh
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/VolGroup-lv_root
    47G 1.1G 44G 3% /
    tmpfs 491M 0 491M 0% /dev/shm
    /dev/sda1 477M 28M 425M 6% /boot
    /dev/sdb 7.8G 19M 7.4G 1% /brick1
    10.20.0.200:/dth-vol1
    24G 55M 22G 1% /data

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

注:这个默认卷类型 是3块盘 加起来的容量

**测试 创建replic 复制卷**

#gluster volume create rep1  replica 3 10.20.0.200:/brick1/b20 10.20.0.201:/brick1/b20 10.20.0.202:/brick1/b20

**客户端挂载**

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# mkdir /data1
    # mount -t glusterfs 10.20.0.200:/rep1 /data1
    # df -kh
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/VolGroup-lv_root
                           47G  1.1G   44G   3% /
    tmpfs                 491M     0  491M   0% /dev/shm
    /dev/sda1             477M   28M  425M   6% /boot
    /dev/sdb              7.8G   19M  7.4G   1% /brick1
    10.20.0.200:/dth-vol1
                           24G   55M   22G   1% /data
    10.20.0.200:/rep1     7.8G   19M  7.4G   1% /data1

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

注:replic类型 是复制模式 相当于raid1  一块盘的容量

**常规命令**：

**添加磁盘**

gluster volume add-brick dht-vol1 10.20.0.200:/brick1/b1
    
    rebalance 同步数据
    gluster volume rebalance dht-vol1 start

**删除磁盘**

gluster volume remove-brick dht-vol1 10.20.0.200:/brick1/b1 
    gluster volume rebalance dht-vol1 start

**删除卷**

先停止 在删除

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

# gluster volume stop dht-vol1
    Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
    volume stop: dht-vol1: success
    
    # gluster volume delete dht-vol1
    Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
    volume delete: dht-vol1: success
    
    # gluster volume info
    No volumes present

![复制代码][48304ba5e6f9fe08f3fa1abda7d326ab.png]

集群扩展  
增加节点  
gluster peer probe IP/主机名

删除节点  
gluster peer detach IP/主机名

节点状态  
gluster peer status

GlusterFS典型故障处理：

1，恢复节点配置信息

故障现象:其中一个节点配置信息不正确

故障模拟:

删除server2部分配置信息

配置信息位置:/var/lib/glusterd/

修复方法

触发自修复:通过Gluster工具同步配置信息