Moosefs 分布式存储

我就是我 2022-05-30 06:27 254阅读 0赞

**主机环境:RHEL6.5 selinux and iptables disabled**  
**Master:172.25.254.4:server4**  
**Chunkserver: 172.25.254.5:server5 172.25.254.6:server6**  
**Client: 172.25.254.56**  
![这里写图片描述][SouthEast]

主控服务器 Master server 安装:
    yum install -y mfs-master-1.6.26-1.x86_64.rpm mfs-cgi-1.6.26-1.x86_64.rpm moosefs-cgiserv-3.0.97-1.rhsysv.x86_64.rpm
    开启服务，查看端口是否开启
    [root@server4 etc]# /etc/init.d/moosefs-master start
    Starting mfsmaster:                                        [  OK  ]
    启动 MooseFS cgi server用于使用web端查看监控
    [root@server4 etc]# /etc/init.d/moosefs-cgiserv start
    Starting mfscgiserv:                                       [  OK  ]
    [root@server4 etc]# netstat -antlp
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
    tcp        0      0 0.0.0.0:9419                0.0.0.0:*                   LISTEN      1101/mfsmaster      
    tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      915/redis-server 0. 
    tcp        0      0 0.0.0.0:9420                0.0.0.0:*                   LISTEN      1101/mfsmaster      
    tcp        0      0 0.0.0.0:9421                0.0.0.0:*                   LISTEN      1101/mfsmaster      
    tcp        0      0 0.0.0.0:9425                0.0.0.0:*                   LISTEN      1108/python         
    tcp        0      0 172.25.254.4:58201          172.25.254.4:6379           TIME_WAIT   -

![这里写图片描述][SouthEast 1]

出现上面这个问题是因为解析没有做好，需要在你的master主机上做好解析：
    [root@server4 etc]# vim /etc/hosts
    172.25.254.4 server4 mfsmaster

![这里写图片描述][SouthEast 2]

在server5 6安装chunker
    yum install moosefs-chunkserver-3.0.97-1.rhsysv.x86_64.rpm -y
    为server5新加一块磁盘，并做文件系统。
    [root@server5 ~]# fdisk /dev/vdb 
    [root@server5 ~]# mkfs.ext4 /dev/vdb1 
    [root@server5 ~]# mkdir /mnt/chunk1
    [root@server5 ~]# mount /dev/vdb1 /mnt/chunk1/
    [root@server5 ~]# vim /etc/fstab 
    /dev/vdb1               /mnt/chunk1             ext4    defaults        0 0
    [root@server5 ~]# mount -a
    [root@server5 ~]# df
    /dev/vdb1                      8256888 149492   7687972   2% /mnt/chunk1
    [root@server5 ~]# cd /mnt/
    [root@server5 mnt]# ll
    total 4
    drwxr-xr-x 3 root root 4096 Mar  6 19:41 chunk1
    [root@server5 mnt]# chown mfs.mfs chunk1/
    [root@server5 mnt]# ll
    total 4
    drwxr-xr-x 3 mfs mfs 4096 Mar  6 19:41 chunk1
    [root@server5 mnt]# cd /etc/mfs/
    [root@server5 mfs]# vim mfshdd.cfg #定义 mfs 共享点
     35 /mnt/chunk1
    [root@server5 mfs]# /etc/init.d/moosefs-chunkserver start
    Starting mfschunkserver:                                   [  OK  ]
    [root@server5 mfs]# cd /mnt/chunk1/
    [root@server5 chunk1]# ls
    00  0F  1E  2D  3C  4B  5A  69  78  87  96  A5  B4  C3  D2  E1  F0  FF
    ......
    0E  1D  2C  3B  4A  59  68  77  86  95  A4  B3  C2  D1  E0  EF  FE

![这里写图片描述][SouthEast 3]

**server6操作同server5**

**安装chunk–创建共享分区并为其创建文件系统–挂载分区–修改文件权限–编写配置文件–给mfsmaster加上地址解析–开启chunk服务**

![这里写图片描述][SouthEast 4]

安装moosefs-client
    [root@foundation56 3.0.97]# ym install moosefs-client-3.0.97-1.rhsystemd.x86_64.rpm 
    [root@foundation56 ~]# cd /etc/mfs/
    [root@foundation56 mfs]# vim mfsmount.cfg #定义客户端默认挂载
     16 /mnt/mfs
    [root@foundation56 mfs]# mkdir /mnt/mfs
    [root@foundation56 mfs]# mfsmount 
    [root@foundation56 mfs]# df
    mfsmaster:9421  15674880   825280  14849600   6% /mnt/mfs
    
    # 在 MFS 挂载点下创建两个目录,并设置其文件存储份数:
    [root@foundation56 3.0.97]# cd /mnt/mfs
    [root@foundation56 mfs]# mkdir dir{1..2}
    [root@foundation56 mfs]# ls
    dir1  dir2
    [root@foundation56 mfs]# mfsgetgoal dir1/
    dir1/: 2
    [root@foundation56 mfs]# mfsgetgoal dir2/
    dir2/: 2
    
    对一个目录设定 “ goal”,此目录下的新创建文件和子目录均会继承此目录的设
    定,但不会改变已经存在的文件及目录的 copy 份数。但使用-r 选项可以更改已
    经存在的 copy 份数。
    [root@foundation56 mfs]# mfssetgoal -r 1 dir1/ #设置在 dir1 中文件存储份数为一个,默认是两个
    dir1/:
     inodes with goal changed:                       1
     inodes with goal not changed:                   0
     inodes with permission denied:                  0
    [root@foundation56 mfs]# mfsgetgoal dir1/
    dir1/: 1
    [root@foundation56 mfs]# cp /etc/shadow dir1/
    [root@foundation56 mfs]# cp /etc/shadow dir2/
    [root@foundation56 mfs]# mfsfileinfo dir1/shadow #查看文件信息
    dir1/shadow:
        chunk 0: 0000000000000002_00000001 / (id:2 ver:1)
            copy 1: 172.25.254.6:9422 (status:VALID)
    [root@foundation56 mfs]# mfsfileinfo dir2/shadow 
    dir2/shadow:
        chunk 0: 0000000000000003_00000001 / (id:3 ver:1)
            copy 1: 172.25.254.5:9422 (status:VALID)
            copy 2: 172.25.254.6:9422 (status:VALID)
    关闭 mfschunkserver2 后再查看文件信息
    [root@server6 ~]# /etc/init.d/moosefs-chunkserver stop
    Stopping mfschunkserver:                                   [  OK  ]
    原本dir1中的shadow文件显示为chunkserver2上面的，这个时候将chunkserver2
    关掉，发现不能访问文件。查看文件信息后发现：dir1下面的shadow文件不存在了，
    原本dir2下面存储两份的文件只剩下了一份。

![这里写图片描述][SouthEast 5]

[root@server6 ~]# /etc/init.d/moosefs-chunkserver start
    Starting mfschunkserver:                                   [  OK  ]
    # 数据存储方式，多地存储
    [root@foundation56 dir2]# dd if=/dev/zero of=bigfile bs=1M count=100
    100+0 records in
    100+0 records out
    104857600 bytes (105 MB) copied, 0.477157 s, 220 MB/s
    [root@foundation56 dir2]# mfsfileinfo bigfile 
    bigfile:
        chunk 0: 000000000000000B_00000001 / (id:11 ver:1)
            copy 1: 172.25.254.5:9422 (status:VALID)
            copy 2: 172.25.254.6:9422 (status:VALID)
        chunk 1: 000000000000000C_00000001 / (id:12 ver:1)
            copy 1: 172.25.254.5:9422 (status:VALID)
            copy 2: 172.25.254.6:9422 (status:VALID)

# 数据恢复
    [root@foundation56 mfs]# rm -rf dir1/passwd 
    [root@foundation56 mfs]# mfsgettrashtime .
    .: 86400
    [root@foundation56 mfs]# mkdir /mnt/mfsmeta
    [root@foundation56 mnt]# mfsmount -m /mnt/mfsmeta/
    mfsmaster accepted connection with parameters: read-write,restricted_ip
    [root@foundation56 mnt]# mount #可以使用mount命令查看到挂载情况
    mfsmaster:9421 on /mnt/mfsmeta type fuse.mfsmeta (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    [root@foundation56 trash]# pwd
    /mnt/mfsmeta/trash
    [root@foundation56 trash]# find -name *passwd*
    ./00F/0000000F|dir1|passwd
    [root@foundation56 trash]# mv ./00F/0000000F\|dir1\|passwd undel/

# master端因为数据文件不存在，moosefs-master服务无法开启
    断网、杀掉 MFS的 master 服务对MFS 系统无影响。
    断电可能会出现以下的情况:moosefs-master缺少数据文件，无法启动服务。
    [root@server4 ~]# /etc/init.d/moosefs-master start
    Starting mfsmaster:                                        [FAILED]
    解决方式：查看日志
    [root@server4 ~]# less /var/log/messages 
    ......
    Mar  7 14:48:50 server4 mfsmaster[1106]: topology file has been loaded
    Mar  7 14:48:50 server4 mfsmaster[1106]: can't find metadata.mfs - try using option '-a'  #提示使用-a选项，ok
    Mar  7 14:48:50 server4 mfsmaster[1106]: init: metadata manager failed !!!
    Mar  7 14:48:50 server4 mfsmaster[1106]: exititng ...
    Mar  7 14:48:50 server4 mfsmaster[1106]: process exited successfully (status:1)
    [root@server4 ~]# mfsmetarestore -a #1.7以前的版本使用
    mfsmetarestore has been removed in version 1.7, use mfsmaster -a instead  
    [root@server4 ~]# mfsmaster -a #1.7以后的版本使用
    #恢复完数据后，先关闭服务，然后重新开启就正常了。
    [root@server4 ~]# /etc/init.d/moosefs-master stop 
    Stopping mfsmaster:                                        [  OK  ]
    [root@server4 ~]# /etc/init.d/moosefs-master start
    Starting mfsmaster:                                        [  OK  ]

[SouthEast]: /images/20220530/3065348f731246f78e9a514b6548d445.png
[SouthEast 1]: /images/20220530/bae00128215243f192e97bb5b8f54267.png
[SouthEast 2]: /images/20220530/51c7a418f8c043a39b9f239d29bf5003.png
[SouthEast 3]: /images/20220530/05b403abe3b4426097119dbff838c59a.png
[SouthEast 4]: /images/20220530/3ef427a740554d62a921901fbcb444df.png
[SouthEast 5]: /images/20220530/edf65524143b4ce98fa95e397a28309b.png