Ubuntu下安装GlusterFS

r囧r小猫 2022-09-29 15:47 510阅读 0赞

# 1 安装 #

## 1.1 配置要求 ##

**1 硬件要求**

GlusterFS支持大多数的符合商业标准的x86-64bit硬件平台。存储可以是直接附加的存储，RAID盘，以SATA/SAS/FC盘做后端的FC/Infiniband/iSCSI SAN盘。

**2 网路要求**

GlusterFS支持千兆网，百兆网和Infiniband（光纤）。

**3 操作系统要求**

GlusterFS可以支持很多POSIX兼容的系统，如GNU/Linux, FreeBSD, OpenSolaris, Solaris, 和Mac OS X。Gluster建议采用Ext3, Ext4或者ZFS作为子系统磁盘的格式。Gluster对GNU/Linux, Mac OS X, FreeBSD和OpenSolaris有内嵌的支持。

## 1.2 Ubuntu下安装 ##

Ubuntu 14: install software-properties-common:

sudo apt-get install software-properties-common

Then add the community GlusterFS PPA:

sudo add-apt-repository ppa:gluster/glusterfs-3.8
        sudo apt-get update

Finally, install the packages:

sudo apt-get install glusterfs-server

## 1.3 gluster安装包下载 ##

https://download.gluster.org/pub/gluster/glusterfs/

## 1.4 源码安装 ##

### 1.4.1 安装 ###

./configure
    make
    make install

### 1.4.2 问题 ###

# ./configure 
    configure: error: no acceptable C compiler found in $PATH

解决：apt-get install build-essential

configure: error: Flex or lex required to build glusterfs.

解决：apt-get install flex

configure: error: GNU Bison required to build glusterfs.

解决：apt-get install bison

# ./configure 
    configure: error: OpenSSL crypto library is required to build glusterfs

解决：apt-get install libssl-dev

error: Support for POSIX ACLs is required

解决：apt-get install libacl1 libacl1-dev

checking for SQLITE... configure: error: pass --disable-tiering to build without sqlite

解决：./configure --disable-tiering

configure: error: liburcu-bp not found

解决：apt-get install liburcu1 liburcu-dev

# /usr/local/sbin/gluster
    /usr/local/sbin/gluster: error while loading shared libraries: libglusterfs.so.0: cannot open shared object file: No such file or directory

解决：执行命令ldconfig

### 1.4.3 卸载 ###

执行make命令：

make uninstall-recursive
    make uninstall
    make mostlyclean-recursive
    make clean-recursive
    make distclean-recursive

再执行：

# for i in `find / -name "gluster*"`; do rm -rf $i; done

# 2 安装 #

## 2.1 部署拓扑 ##

3台存储服务器: 192.168.4.132、192.168.4.133、192.168.4.134

root@block2:/glusterfs# cat /etc/hosts
    127.0.0.1	localhost
    192.168.4.130	controller
    192.168.4.131	network
    192.168.4.132	compute
    192.168.4.133	block1
    192.168.4.134	block2

## 2.2 安装 ##

在所有存储服务器上执行安装。

2.2.1 如果出现错误：

W: Failed to fetch http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu/dists/trusty/main/binary-amd64/Packages  Hash Sum mismatch
    
    W: Failed to fetch http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu/dists/trusty/main/binary-i386/Packages  Hash Sum mismatch
    
    W: Failed to fetch http://ppa.launchpad.net/gluster/glusterfs-3.8/ubuntu/dists/trusty/main/i18n/Translation-en  Hash Sum mismatch
    
    E: Some index files failed to download. They have been ignored, or old ones used instead.

是因为没有把这3个文件取到/var/lib/apt/lists目录下。正常情况下：

root@block1:~# ls /var/lib/apt/lists/ppa*
    /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_InRelease
    /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_main_binary-amd64_Packages
    /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_main_binary-i386_Packages
    /var/lib/apt/lists/ppa.launchpad.net_gluster_glusterfs-3.8_ubuntu_dists_trusty_main_i18n_Translation-en

解决方法：  
将/var/lib/apt/lists/partial/下的所有文件删除，但是这可以解决大部分问题，个别源不能解决，  
注：经过长时间的实践，这个解决方法不是万能的。

2.2.2 安装glusterfs-server时，会安装glusterfs-client

root@block2:/glusterfs# apt-get install glusterfs-server
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following extra packages will be installed:
      attr glusterfs-client glusterfs-common liblvm2app2.2 liburcu1
    The following NEW packages will be installed:
      attr glusterfs-client glusterfs-common glusterfs-server liblvm2app2.2
      liburcu1
    0 upgraded, 6 newly installed, 0 to remove and 4 not upgraded.
    Need to get 3,389 kB of archives.
    After this operation, 16.0 MB of additional disk space will be used.
    Do you want to continue? [Y/n]

安装成功后，gluster会运行。查看版本：

root@block2:/glusterfs# glusterfs --version
    glusterfs 3.8.10

root@block2:/glusterfs# ps -ef | grep gluster
    root     17920     1  0 17:58 ?        00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid

## 2.3 查看运行端口 ##

root@compute: # netstat -pan | grep glus
    tcp        0      0 0.0.0.0:24007           0.0.0.0:*               LISTEN      5323/glusterd   
    tcp        0      0 192.168.4.132:24007     192.168.4.133:1023      ESTABLISHED 5323/glusterd   
    tcp        0      0 192.168.4.132:1023      192.168.4.134:24007     ESTABLISHED 5323/glusterd   
    tcp        0      0 192.168.4.132:24007     192.168.4.134:1020      ESTABLISHED 5323/glusterd   
    tcp        0      0 192.168.4.132:1021      192.168.4.133:24007     ESTABLISHED 5323/glusterd

## 2.4 服务启停 ##

使用命令：service glusterfs-server start/stop

root@compute:~# ps -ef | grep gluster
    root     10929     1  0 Mar16 ?        00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
    root     12215  1650  0 10:22 pts/0    00:00:00 grep --color=auto gluster
    root@compute:~# ls /etc/init.d/glusterfs*
    /etc/init.d/glusterfs-server
    root@compute:~# service glusterfs-server stop
    glusterfs-server stop/waiting
    root@compute:~# ps -ef | grep gluster
    root     12225  1650  0 10:23 pts/0    00:00:00 grep --color=auto gluster
    root@compute:~# service glusterfs-server start
    glusterfs-server start/running, process 12233
    root@compute:~# ps -ef | grep gluster
    root     12233     1  0 10:23 ?        00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid
    root     12467  1650  0 10:23 pts/0    00:00:00 grep --color=auto gluster

# 3 集群关联 #

# 3.1创建GlusterFS集群 #

$ gluster peer probe SERVER     SERVER表示存储服务器的hostname。

在服务器192.168.4.132上执行：

root@compute:~#  gluster peer probe 
    Usage: peer probe <HOSTNAME>
    root@compute:~#  gluster peer probe block1
    peer probe: success
    root@compute:~# gluster peer probe block2
    peer probe: success

注意：在构建 GlusterFS 集群时，需要在其中任意一台存储服务器上依次将其他存储服务器添加到集群中。

**3.2  查看集群信息**

指令：gluster peer status

root@compute:/etc/glusterfs# gluster peer status
    Number of Peers: 2
    
    Hostname: block1
    Port: 24007
    Uuid: ac468b50-3d66-4290-aa8d-cb01279e9bf2
    State: Peer in Cluster (Connected)
    
    Hostname: block2
    Port: 24007
    Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
    State: Peer in Cluster (Connected)
    
    root@block1:/etc/glusterfs# gluster peer status
    Number of Peers: 2
    
    Hostname: 192.168.4.132
    Port: 24007
    Uuid: 9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d
    State: Peer in Cluster (Connected)
    
    Hostname: block2
    Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
    State: Peer in Cluster (Connected)
    root@block1:/etc/glusterfs# gluster volume info
    No volumes present

Connected,说明集群节点添加成功，但其中compute是以192.168.4.132这个ip加入集群的，如果需要换成hostname，登陆集群中的集群中的另一个节点，执行：

root@block1:~# gluster peer detach 192.168.4.132
    peer detach: success
    root@block1:~# 
    root@block1:~# gluster peer status
    Number of Peers: 1
    
    Hostname: block2
    Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
    State: Peer in Cluster (Connected)
    root@block1:~# 
    root@block1:~#  gluster peer probe compute
    peer probe: success. 
    root@block1:~# 
    root@block1:~# gluster peer status
    Number of Peers: 2
    
    Hostname: block2
    Uuid: 06288da0-1b18-42fb-a7b8-cd9ca24d5f26
    State: Peer in Cluster (Connected)
    
    Hostname: compute
    Uuid: 9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d
    State: Peer in Cluster (Connected)

# 4 Volume卷操作 #

基本卷：

(1)  distribute volume：分布式卷

文件通过hash算法分布到所有brick server上，这种卷是glusterfs的基础和最大特点。优点是容量大，缺点是没冗余。

(2)  stripe volume：条带卷

类似RAID0，文件分成数据块以Round Robin方式分布到brick server上，并发粒度是数据块，支持超大文件，大文件性能高；优点是分布式读写，性能整体较好。缺点是没冗余，分片随机读写可能会导致硬盘IOPS饱和。

(3)  replica volume：复制卷

文件同步复制到多个brick上，文件级RAID 1，具有容错能力，写性能下降，读性能提升。缺点是磁盘利用率低。

复合卷：

(4)  distribute stripe volume：分布式条带卷

brickserver数量是条带数的倍数，兼具distribute和stripe卷的特点；

(5)  distribute replica volume：分布式复制卷

brickserver数量是镜像数的倍数，兼具distribute和replica卷的特点,可以在2个或多个节点之间复制数据。

(6) stripe replica volume：条带复制卷

类似RAID 10

同时具有条带卷和复制卷的特点

(7) distribute stripe replicavolume：分布式条带复制卷

三种基本卷的复合卷

通常用于类Map Reduce应用

(8) 冗余卷（Dispersed volume）

近似于raid5，文件分片存储在各个硬盘上，但有部分硬盘用于冗余用途，数量可以指定。比如一共10块硬盘，2块盘用于冗余，那么就可以承受同时损坏两块硬盘，总容量是8块盘。

优点是在冗余和性能之间取得平衡

## 4.1．创建volume ##

a. 单磁盘，调试环境推荐  
sudo gluster volume create vol\_name 172.17.0.2:/d/disk0  
  
b. 多磁盘，无raid，试验、测试环境推荐。  
sudo gluster volume create vol\_name 172.17.0.2:/d/disk0 172.17.0.3:/d/disk0 172.17.0.4:/d/disk0 172.17.0.5:/d/disk0  
  
c. 多磁盘，有raid1。线上高并发环境推荐。

Usage: volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]

注意：以上命令中，磁盘数量必须为复制份数的整数倍。

实验命令是在根目录下创建目录/glusterfs，要在所有的3台服务器上都创建这个目录，否则会出错。

root@compute:~# gluster volume create zhai-volume replica 2 192.168.4.132:/glusterfs/brick1 block1:/glusterfs/brick1  block2:/glusterfs/brick1
    number of bricks is not a multiple of replica count
    Usage: volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
    root@compute:~# 
    root@compute:~# gluster volume create zhai-volume replica 2 compute:/glusterfs/brick1 compute:/glusterfs/brick2 block1:/glusterfs/brick1 block1:/glusterfs/brick2 block2:/glusterfs/brick1 block2:/glusterfs/brick2
    volume create: zhai-volume: failed: The brick compute:/glusterfs/brick1 is being created in the root partition. It is recommended that you don't use the system's root partition for storage backend. Or use 'force' at the end of the command if you want to override this behavior.
    root@compute:~# 
    root@compute:~# ls /glusterfs/
    root@compute:~# 
    root@compute:~# gluster volume create zhai-volume replica 2 compute:/glusterfs/brick1 compute:/glusterfs/brick2 block1:/glusterfs/brick1 block1:/glusterfs/brick2 block2:/glusterfs/brick1 block2:/glusterfs/brick2 force
    volume create: zhai-volume: failed: Staging failed on block2. Error: Failed to create brick directory for brick block2:/glusterfs/brick1. Reason : No such file or directory 
    Staging failed on block1. Error: Failed to create brick directory for brick block1:/glusterfs/brick1. Reason : No such file or directory 
    root@compute:~# 
    root@compute:~# gluster volume create zhai-volume replica 2 compute:/glusterfs/brick1 compute:/glusterfs/brick2 block1:/glusterfs/brick1 block1:/glusterfs/brick2 block2:/glusterfs/brick1 block2:/glusterfs/brick2 force
    volume create: zhai-volume: success: please start the volume to access data
    root@compute:~# 
    root@compute:~# gluster volume info
     
    Volume Name: zhai-volume
    Type: Distributed-Replicate
    Volume ID: 779a7440-e127-43c6-a13f-7420ab8baec8
    Status: Created
    Snapshot Count: 0
    Number of Bricks: 3 x 2 = 6
    Transport-type: tcp
    Bricks:
    Brick1: compute:/glusterfs/brick1
    Brick2: compute:/glusterfs/brick2
    Brick3: block1:/glusterfs/brick1
    Brick4: block1:/glusterfs/brick2
    Brick5: block2:/glusterfs/brick1
    Brick6: block2:/glusterfs/brick2
    Options Reconfigured:
    transport.address-family: inet

Note : The number of bricks should be a multiple of the replica count for a distributed replicated volume. Also, the order in which bricks are specified has a great effect on data protection. Each replica\_count consecutive bricks in the list you give will form a replica set, with all replica sets combined into a volume-wide distribute set.

volume创建成功后，会在每台服务器的/glusterfs目录下生成brick1和brick2目录。  
查看日志：

root@compute:~# tail -20 /var/log/glusterfs/cli.log
    [2017-03-17 03:01:38.518626] I [cli.c:728:main] 0-cli: Started running gluster with version 3.8.10
    [2017-03-17 03:01:38.608896] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
    [2017-03-17 03:01:38.608958] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
    [2017-03-17 03:01:38.619870] I [cli-rpc-ops.c:1117:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
    [2017-03-17 03:01:38.619953] I [input.c:31:cli_batch] 0-: Exiting with: -1
    [2017-03-17 03:02:34.746102] I [cli.c:728:main] 0-cli: Started running gluster with version 3.8.10
    [2017-03-17 03:02:34.837934] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
    [2017-03-17 03:02:34.838008] I [socket.c:2403:socket_event_handler] 0-transport: disconnecting now
    [2017-03-17 03:02:36.111778] I [cli-rpc-ops.c:1117:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
    [2017-03-17 03:02:36.111888] I [input.c:31:cli_batch] 0-: Exiting with: 0

## 4.2．启动volume ##

刚创建好的volume还没有运行，需要执行运行命令方可使用。 在其中一台服务器上执行。  
sudo gluster volume start vol\_name

root@compute:~# ps -ef | grep volume
    root     12663  1650  0 11:16 pts/0    00:00:00 grep --color=auto volume
    root@compute:~# gluster volume start zhai-volume
    volume start: zhai-volume: success
    root@compute:~# ps -ef | grep volume
    root     12683     1  0 11:16 ?        00:00:00 /usr/sbin/glusterfsd -s compute --volfile-id zhai-volume.compute.glusterfs-brick1 -p /var/lib/glusterd/vols/zhai-volume/run/compute-glusterfs-brick1.pid -S /var/run/gluster/233b16b60a6dacf67c1b711bb1e7e89c.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d --brick-port 49152 --xlator-option zhai-volume-server.listen-port=49152
    root     12702     1  0 11:16 ?        00:00:00 /usr/sbin/glusterfsd -s compute --volfile-id zhai-volume.compute.glusterfs-brick2 -p /var/lib/glusterd/vols/zhai-volume/run/compute-glusterfs-brick2.pid -S /var/run/gluster/55ba583ae215e8215759c8d1aca9548c.socket --brick-name /glusterfs/brick2 -l /var/log/glusterfs/bricks/glusterfs-brick2.log --xlator-option *-posix.glusterd-uuid=9fdd15f2-c7e1-44e1-9315-d62e1dcfb44d --brick-port 49153 --xlator-option zhai-volume-server.listen-port=49153
    root     12729  1650  0 11:16 pts/0    00:00:00 grep --color=auto volume

root@block1:~# ps -ef | grep zhai-volume
    root     26213     1  0 11:16 ?        00:00:00 /usr/sbin/glusterfsd -s block1 --volfile-id zhai-volume.block1.glusterfs-brick1 -p /var/lib/glusterd/vols/zhai-volume/run/block1-glusterfs-brick1.pid -S /var/run/gluster/98664ae15cd31af2fd3394f1a239d85b.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=ac468b50-3d66-4290-aa8d-cb01279e9bf2 --brick-port 49152 --xlator-option zhai-volume-server.listen-port=49152
    root     26232     1  0 11:16 ?        00:00:00 /usr/sbin/glusterfsd -s block1 --volfile-id zhai-volume.block1.glusterfs-brick2 -p /var/lib/glusterd/vols/zhai-volume/run/block1-glusterfs-brick2.pid -S /var/run/gluster/e4bf4eb272d50def1259ad279e81bc61.socket --brick-name /glusterfs/brick2 -l /var/log/glusterfs/bricks/glusterfs-brick2.log --xlator-option *-posix.glusterd-uuid=ac468b50-3d66-4290-aa8d-cb01279e9bf2 --brick-port 49153 --xlator-option zhai-volume-server.listen-port=49153

root@block2:~# ps -ef | grep zhai-volume
    root     24570     1  0 11:16 ?        00:00:00 /usr/sbin/glusterfsd -s block2 --volfile-id zhai-volume.block2.glusterfs-brick1 -p /var/lib/glusterd/vols/zhai-volume/run/block2-glusterfs-brick1.pid -S /var/run/gluster/b5c757abb3485edb13b25f9aaac3c387.socket --brick-name /glusterfs/brick1 -l /var/log/glusterfs/bricks/glusterfs-brick1.log --xlator-option *-posix.glusterd-uuid=06288da0-1b18-42fb-a7b8-cd9ca24d5f26 --brick-port 49152 --xlator-option zhai-volume-server.listen-port=49152
    root     24589     1  0 11:16 ?        00:00:00 /usr/sbin/glusterfsd -s block2 --volfile-id zhai-volume.block2.glusterfs-brick2 -p /var/lib/glusterd/vols/zhai-volume/run/block2-glusterfs-brick2.pid -S /var/run/gluster/0acc5e5ccc951203695b9654ef4dbd67.socket --brick-name /glusterfs/brick2 -l /var/log/glusterfs/bricks/glusterfs-brick2.log --xlator-option *-posix.glusterd-uuid=06288da0-1b18-42fb-a7b8-cd9ca24d5f26 --brick-port 49153 --xlator-option zhai-volume-server.listen-port=49153

## 4.3．挂载volume ##

### 4.3.1  在3台服务器中任一服务器上挂载都可以。  ###

For this step, we will use one of the servers to mount the volume. Typically, you would do this from an external machine, known as a "client". Since using this method would require additional packages to be installed on the client machine, we willuse one of the servers as a simple place to test first, as if it were that "client".

\# 挂载任一节点即可(推荐)

root@compute:~# mount -t glusterfs 192.168.4.132:/zhai-volume /mnt
    root@compute:~# df -h
    Filesystem                  Size  Used Avail Use% Mounted on
    udev                        3.9G   12K  3.9G   1% /dev
    tmpfs                       788M  660K  787M   1% /run
    /dev/dm-0                   909G  2.4G  860G   1% /
    none                        4.0K     0  4.0K   0% /sys/fs/cgroup
    none                        5.0M     0  5.0M   0% /run/lock
    none                        3.9G     0  3.9G   0% /run/shm
    none                        100M     0  100M   0% /run/user
    /dev/sda2                   237M   91M  134M  41% /boot
    /dev/sda1                   511M  3.4M  508M   1% /boot/efi
    192.168.4.132:/zhai-volume  2.7T  6.0G  2.6T   1% /mnt

### 4.3.2 在3台服务器之外的服务器上挂载 ###

需要安装glusterfs-client包  
否则，出错：

root@network:~# mount -t glusterfs 192.168.4.132:/zhai-volume /mnt
    mount: unknown filesystem type 'glusterfs'

安装

sudo  apt-get  install  glusterfs-client

安装后，执行挂载

# mount -t glusterfs 10.10.30.4:/zhai-volume /mnt
    Mount failed. Please check the log file for more details.

原因： 没有在/etc/hosts中添加glusterfs集群中各个服务器的信息

### 4.3.3 开机挂载 ###

\# 若需要开机自动挂载，请更新/etc/fstab

## 4.4．使用GlusterFS ##

a. 挂载了GlusterFS的某个卷后，就可以将其当做本地文件访问，代码中只需使用原生的文件api即可。这种方式使用不一定需要root权限，只要拥有对应目录或文件的权限即可。  
b. 直接API方式，这种方式需要root权限才能使用，并且java、python、ruby的api包装目前都不够完整，一般情况不推荐使用。  
到

**测试1：**

到/mnt目录下创建文件：

root@compute:/mnt# uname -a >info
    root@compute:/mnt# ls
    info
    root@compute:/mnt# cat info 
    Linux compute 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

在3台服务器上查看， 在block2服务器上生成文件，在block1和compute结点上没有生成文件：

root@block2:~# ls /glusterfs/brick1/
    info
    root@block2:~# ls /glusterfs/brick2
    info
    root@block2:~# cat /glusterfs/brick1/info 
    Linux compute 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

**测试2：**

mount -t glusterfs server1:/gv0 /mnt
          for i in `seq -w 1 100`; do cp -rp /var/log/messages /mnt/copy-test-$i; done

First, check the mount point:

ls -lA /mnt | wc -l

You should see 100 files returned. Next, check the GlusterFS mount points on each server:

ls -lA /data/brick1/gv0

You should see 100 files on each server using the method we listed here. Without replication, in a distribute only volume (not detailed here), you should see about 50 files on each one.

## 4.5．卸载volume ##

卸载与挂载操作是一对。虽然没有卸载也可以停止volume，但是这样做是会出问题，如果集群较大，可能导致后面volume启动失败。  
sudo umount /mnt

## 4.6．停止volume ##

停止与启动操作是一对。停止前最好先卸载所有客户端。  
sudo gluster volume stop vol\_name

## 4.7．删除volume ##

删除与创建操作是一对。删除前需要先停止volume。在生产上一般不会删除volume  
sudo gluster volume delete vol\_name

# 5 使用独立存储 #

使用独立/dev/sdb

## 5.1 分区 ##

fdisk /dev/sdb //创建分区，输入,n,p,w，创建新主分区并写入分区表

## 5.2 Format and mount the bricks ##

(on both nodes): Note: These examples are going to assume the brick is going to reside on /dev/sdb1.

mkfs.xfs -i size=512 /dev/sdb1
        mkdir -p /data/brick1
        echo '/dev/sdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
        mount -a && mount

You should now see sdb1 mounted at /data/brick1

root@block1:~# mkfs.xfs -i size=512 /dev/sdb1
    root@compute:~# echo '/dev/sdb1 /glusterfs xfs defaults 1 2' >> /etc/fstab
    root@compute:~# mount -a && mount
    /dev/mapper/compute--vg-root on / type ext4 (rw,errors=remount-ro)
    proc on /proc type proc (rw,noexec,nosuid,nodev)
    sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
    none on /sys/fs/cgroup type tmpfs (rw)
    none on /sys/fs/fuse/connections type fusectl (rw)
    none on /sys/kernel/debug type debugfs (rw)
    none on /sys/kernel/security type securityfs (rw)
    none on /sys/firmware/efi/efivars type efivarfs (rw)
    udev on /dev type devtmpfs (rw,mode=0755)
    devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
    tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
    none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
    none on /run/shm type tmpfs (rw,nosuid,nodev)
    none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
    none on /sys/fs/pstore type pstore (rw)
    tracefs on /var/lib/ureadahead/debugfs/tracing type tracefs (rw,relatime)
    /dev/sda2 on /boot type ext2 (rw)
    /dev/sda1 on /boot/efi type vfat (rw)
    systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
    rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw)
    192.168.4.132:/zhai-volume on /mnt type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
    /dev/sdb1 on /glusterfs type xfs (rw)
    
    
    root@compute:~# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            3.9G   12K  3.9G   1% /dev
    tmpfs           788M  660K  787M   1% /run
    /dev/dm-0       909G  2.4G  860G   1% /
    none            4.0K     0  4.0K   0% /sys/fs/cgroup
    none            5.0M     0  5.0M   0% /run/lock
    none            3.9G     0  3.9G   0% /run/shm
    none            100M     0  100M   0% /run/user
    /dev/sda2       237M   91M  134M  41% /boot
    /dev/sda1       511M  3.4M  508M   1% /boot/efi
    /dev/sdb1       932G   34M  932G   1% /glusterfs

mount之后再安装glusterfs

5.3 其它命令

#apt-get install xfsprogs //安装xfs文件系统（如果没有的话），更安全可靠
    
    #lshw -short	//查看硬件信息，主要是查看硬盘信息，以决定哪块硬盘作为gluster来使用
    或使用lsblk来查看块设备
    #lsblk

# 6 相关问题 #

1 Q: 创建/启用/停用/删除volume最终都干了些什么？

(1) 创建volume能看得到的结果是: glusterd在工作目录(默认为/var/lib/glusterd)的vols目录下，创建以volume名称为名字的目录，并在这个目录下创建相关文件记录volume的相关信息；看不到的结果是glusterd程序中也记录了volume的相关信息。

(2) 启用volume的结果是在指定brick的节点上启动glusterfsd和glusterfs进程(注意：如果多个brick在同一节点下，会有多个glusterfsd进程，但只有一个glusterfs进程)，同时在日志文件中更新记录的volume相关信息。

(3) 停用volume的结果是结束glusterfsd和glusterfs进程。

(4)删除volume则是将创建volume的相关文件信息全部清除。

2 错误

root@compute:~# gluster volume create volume-test block1:/glusterfs/repbrick block2:/glusterfs/repbrick
    volume create: volume-test: failed: Staging failed on block2. Error: Brick: block2:/glusterfs/repbrick not available. Brick may be containing or be contained by an existing brick
    Staging failed on block1. Error: Brick: block1:/glusterfs/repbrick not available. Brick may be containing or be contained by an existing brick

这是因为创建volume时，在brick对应的存储目录上，增加了"trusted.glusterfs.volume-id"和"trusted.gfid"两个扩展属性，在删除volume时并未移除这两个属性，再次创建时，对brick的存储目录进行校验，发现已经有了扩展属性，因此会有brick已经是volume的一部分的提示。

通过getfattr命令可以查看这些存储路径的扩展属性

root@block2:~# getfattr -d -m ".*" /glusterfs/repbrick
    getfattr: Removing leading '/' from absolute path names
    # file: glusterfs/repbrick
    trusted.glusterfs.volume-id=0sS29pmfEgRDyMNCX3YAbwsQ==

设置属性

setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/volume-down-fault/info  | cut -d= -f2 | sed 's/-//g') /glusterfs/brick2

3 可执行程序

glusterfs安装后，会有gluster，glusterd，glusterfs，glusterfsd这么几个可执行程序，其作用分别为：

gluster：Glusterfs控制台管理程序（Gluster Console Manager），可以以命令形式或者交互式形式对glusterfs中的volume，brick，集群节点等信息进行查看及操作（增，删，改）。

glusterd：软链接指向glusterfsd，Glusterfs的管理进程，负责处理来自gluster的命令。

glusterfs：软链接指向glusterfsd，Glusterfs自带的客户端

glusterfsd：Glusterfs服务端程序

注：glusterd,glusterfs,glusterfsd为最终是运行同一个程序(glusterfsd)，程序内部会根据可执行程序名称加以区别。

staticuint8_t gf_get_process_mode (char*exec_name)
    {
        char*dup_execname = NULL, *base = NULL;
        uint8_t ret = 0;
     
        dup_execname = gf_strdup (exec_name);
        base = basename (dup_execname);
         
        if(!strncmp (base, "glusterfsd", 10)) {
            ret = GF_SERVER_PROCESS;
        }else if (!strncmp(base, "glusterd", 8)) {
            ret = GF_GLUSTERD_PROCESS;
        }else {
            ret = GF_CLIENT_PROCESS;
        }
        GF_FREE (dup_execname);
        returnret;
    }

4  客户端依靠nfs协议（或gluster native协议）动态的从服务端获取卷的信息。

5 如果建立的是复制卷，gluster客户端会并发的向几个服务器写入\\读取数据，而不是通过服务器之间进行数据的传输。