glusterfs引起的split brain问题

迷南。 2022-05-28 08:52 170阅读 0赞

# 1 问题描述 #

在openstack环境中的一个vm无故停止了，使用界面和命令都启动不了，查看日志：

2018-04-02 03:22:53.463 14655 INFO nova.virt.libvirt.driver [-] [instance: 0498a9d1-3958-4972-943e-fe6a44490192] Instance destroyed successfully.
    2018-04-02 03:22:54.285 INFO nova.virt.libvirt.imagebackend [req-694c00cb-56f5-47b2-bd0d-50a7d34edabf 1d94aa6ff73b471b8d39edb1823fca60 a20101a00a7c4943bc8eaae2c6dd792e] base: /var/lib/nova/instances/_base/1cd4a201979d6b4edc01d6920f6015b2d0e19627
    2018-04-02 03:22:55.242 ERROR nova.openstack.common.rpc.amqp [req-694c00cb-56f5-47b2-bd0d-50a7d34edabf 1d94aa6ff73b471b8d39edb1823fca60 a20101a00a7c4943bc8eaae2c6dd792e] Exception during message handling
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt, version, method, **args)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     temp_level, payload)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     pass
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     self.gen.next()
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1454, in start_instance
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     block_device_info)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1429, in power_on
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     self._hard_reboot(context, instance, network_info, block_device_info)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1390, in _hard_reboot
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     disk_info_json)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3319, in _create_images_and_backing
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     size=info['virt_disk_size'])
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 159, in cache
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     *args, **kwargs)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 261, in create_image
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     prepare_template(target=base, *args, **kwargs)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 221, in inner
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     with lock:
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 83, in __enter__
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     self.trylock()
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 118, in trylock
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp     fcntl.lockf(self.lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
    2018-04-02 03:22:55.242 14655 TRACE nova.openstack.common.rpc.amqp IOError: [Errno 2] No such file or directory

日志显示没有文件，但是我查看/var/lib/nova/instance目录后，确认文件存在。考虑可能是glusterfs的问题。

查看glusterfs的日志：

[2018-04-02 01:48:49.746336] E [afr-transaction.c:820:afr_changelog_pre_op_cbk] 1-dv-grizzly-ins-replicate-0: xattrop failed on child dv-grizzly-ins-client-1: Success
    [2018-04-02 01:48:49.747353] W [client-rpc-fops.c:1822:client3_3_fxattrop_cbk] 1-dv-grizzly-ins-client-1: remote operation failed: Success
    [2018-04-02 01:48:49.747382] E [afr-transaction.c:820:afr_changelog_pre_op_cbk] 1-dv-grizzly-ins-replicate-0: xattrop failed on child dv-grizzly-ins-client-1: Success
    [2018-04-02 01:48:50.011437] W [client-rpc-fops.c:1822:client3_3_fxattrop_cbk] 1-dv-grizzly-ins-client-1: remote operation failed: Success
    [2018-04-02 01:48:50.011486] E [afr-transaction.c:820:afr_changelog_pre_op_cbk] 1-dv-grizzly-ins-replicate-0: xattrop failed on child dv-grizzly-ins-client-1: Success
    [2018-04-02 01:48:50.011688] W [client-rpc-fops.c:1822:client3_3_fxattrop_cbk] 1-dv-grizzly-ins-client-1: remote operation failed: Success
    [2018-04-02 01:48:50.011753] E [afr-transaction.c:820:afr_changelog_pre_op_cbk] 1-dv-grizzly-ins-replicate-0: xattrop failed on child dv-grizzly-ins-client-1: Success
    [2018-04-02 01:48:50.012787] W [client-rpc-fops.c:1822:client3_3_fxattrop_cbk] 1-dv-grizzly-ins-client-1: remote operation failed: Success
    [2018-04-02 01:48:50.012819] E [afr-transaction.c:820:afr_changelog_pre_op_cbk] 1-dv-grizzly-ins-replicate-0: xattrop failed on child dv-grizzly-ins-client-1: Success
    [2018-04-02 01:48:50.147697] W [client-rpc-fops.c:1822:client3_3_fxattrop_cbk] 1-dv-grizzly-ins-client-1: remote operation failed: Success
    [2018-04-02 01:48:50.147790] E [afr-transaction.c:820:afr_changelog_pre_op_cbk] 1-dv-grizzly-ins-replicate-0: xattrop failed on child dv-grizzly-ins-client-1: Success

产生了几十G的日志文件

glusterfs集群是70和78两台服务器组成。使用命令gluster volume heal gluster\_vol info 查看发现已经脑裂。

# 2 方法 #

参考：http://blog.51cto.com/nosmoking/1718157

一、现象  
1、ovirt在升级glusterfs的版本时，测试未停止ovirt和gluster服务，直接升级gluster软件包然后重启gluster服务，结果发现其中一个vm的数据盘出现异常，无法启动  
2、查看glustefs日志，可能是存在脑裂  
3、原因是因为作为测试用途，仅用2台物理机做的glustefs做为ovirt的数据域，且关闭了 quorum 相关选项，这种场景容易导致脑裂，ovirt在增加数据域的时候有明显的提示。  
   
二、处理过程  
1、查看gluster的状态  
\--------- split-brain的报告 ---------  
gluster> volume heal ovirt-data info  
Brick n33.test:/data/ovirt/data/  
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54 - Is in split-brain  
   
Number of entries: 1  
   
Brick n34.test:/data/ovirt/data/  
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54 - Is in split-brain  
   
Number of entries: 1  
   
   
2、数据对比  
\--------- 异常的状态对比 ---------  
\[root@n33 1d04adc8-9921-4e1c-b192-84fba64224db\]\# getfattr -d -m . -e hex bd3c9ecf-8100-42db-9db5-038dd95a4d54  
\# file: bd3c9ecf-8100-42db-9db5-038dd95a4d54  
trusted.afr.ovirt-data-client-0=0x000000000000000000000000  
trusted.afr.ovirt-data-client-1=0x0000000a0000000000000000  
trusted.gfid=0xaa32a1b65bf14655aafbeb0f242e61e0  
   
\[root@n34 1d04adc8-9921-4e1c-b192-84fba64224db\]\# getfattr -d -m . -e hex bd3c9ecf-8100-42db-9db5-038dd95a4d54  
\# file: bd3c9ecf-8100-42db-9db5-038dd95a4d54  
trusted.afr.ovirt-data-client-0=0x0000000f0000000000000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
trusted.gfid=0xaa32a1b65bf14655aafbeb0f242e61e0  
   
\--------- 随便找一个正常的数据文件的状态做对比 ---------  
\[root@n33 560a7b8c-fbfe-4495-b315-e46593b4523e\]\# getfattr -d -m . -e hex 52a0c2aa-778a-4964-98c6-03cdb591cfa8  
\# file: 52a0c2aa-778a-4964-98c6-03cdb591cfa8  
trusted.afr.ovirt-data-client-0=0x000000000000000000000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
trusted.gfid=0xa36f8c0d2e014dc1ba19600aa62a20ec  
   
\[root@n34 560a7b8c-fbfe-4495-b315-e46593b4523e\]\# getfattr -d -m . -e hex 52a0c2aa-778a-4964-98c6-03cdb591cfa8  
\# file: 52a0c2aa-778a-4964-98c6-03cdb591cfa8  
trusted.afr.ovirt-data-client-0=0x000000000000000000000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
trusted.gfid=0xa36f8c0d2e014dc1ba19600aa62a20ec  
\--------- 上面是正常的现象，2个节点的状态一致 ---------  
   
   
   
3、解释  
trusted.afr.ovirt-data-client-0=0x000000000000000000000000  
trusted.afr.ovirt-data-client-1=0x0000000a0000000000000000  
对比  
trusted.afr.ovirt-data-client-0=0x0000000f0000000000000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
   
说明：  
0x 0x0000000a 00000000 00000000  
        |      |       |  
        |      |        \\\_ changelog of directory entries  
        |       \\\_ changelog of metadata  
         \\ \_ changelog of data  
            
结论：client-0认为client1的数据有问题，反之亦然  
   
   
4、处理  
\[root@n33 1d04adc8-9921-4e1c-b192-84fba64224db\]\# stat bd3c9ecf-8100-42db-9db5-038dd95a4d54  
  File: \`bd3c9ecf-8100-42db-9db5-038dd95a4d54'  
  Size: 85899345920     Blocks: 27881072   IO Block: 4096   regular file  
Device: fd01h/64769d    Inode: 2147485390  Links: 2  
Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)  
Access: 2015-11-27 19:02:02.947525264 +0800  
Modify: 2015-11-27 19:02:14.020777092 +0800  
Change: 2015-11-30 09:25:41.425588695 +0800  
\[root@n34 1d04adc8-9921-4e1c-b192-84fba64224db\]\# stat bd3c9ecf-8100-42db-9db5-038dd95a4d54  
  File: \`bd3c9ecf-8100-42db-9db5-038dd95a4d54'  
  Size: 85899345920     Blocks: 27881896   IO Block: 4096   regular file  
Device: fd01h/64769d    Inode: 1974        Links: 2  
Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)  
Access: 2015-11-27 18:59:45.987653357 +0800  
Modify: 2015-11-27 18:59:57.051164636 +0800  
Change: 2015-11-30 09:22:22.977259770 +0800  
   
根据文件的状态内容来选择1个可靠的副本，丢弃另一个。  
   
ok，我先打算保留时间更新的一个副本，也就是n33上的，则我们打算丢弃n34上的副本  
\[root@n34 1d04adc8-9921-4e1c-b192-84fba64224db\]\# setfattr -n trusted.afr.ovirt-data-client-0 -v 0x000000000000000100000000 bd3c9ecf-8100-42db-9db5-038dd95a4d54  
\[root@n34 1d04adc8-9921-4e1c-b192-84fba64224db\]\# getfattr -d -m . -e hex bd3c9ecf-8100-42db-9db5-038dd95a4d54  
\# file: bd3c9ecf-8100-42db-9db5-038dd95a4d54  
trusted.afr.ovirt-data-client-0=0x000000000000000100000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
trusted.gfid=0xaa32a1b65bf14655aafbeb0f242e61e0  
   
   
\[root@n33 1d04adc8-9921-4e1c-b192-84fba64224db\]\# gluster  
gluster> volume heal ovirt-data info  
Brick n33.test:/data/ovirt/data/  
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54 - Possibly undergoing heal  
   
Number of entries: 1  
   
Brick n34.test:/data/ovirt/data/  
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54 - Possibly undergoing heal  
   
Number of entries: 1  
   
大概5分钟后：  
gluster> volume heal ovirt-data info  
Brick n33.test:/data/ovirt/data/  
/09cb8372-a68d-47dc-962e-70b5225be6bc/images/1d04adc8-9921-4e1c-b192-84fba64224db/bd3c9ecf-8100-42db-9db5-038dd95a4d54 - Possibly undergoing heal  
   
Number of entries: 1  
   
Brick n34.test:/data/ovirt/data/  
Number of entries: 0  
   
大概半小时后：  
gluster> volume heal ovirt-data info  
Brick n33.test:/data/ovirt/data/  
Number of entries: 0  
   
Brick n34.test:/data/ovirt/data/  
Number of entries: 0  
   
   
再次对比：  
\[root@n33 1d04adc8-9921-4e1c-b192-84fba64224db\]\# getfattr -d -m . -e hex bd3c9ecf-8100-42db-9db5-038dd95a4d54  
\# file: bd3c9ecf-8100-42db-9db5-038dd95a4d54  
trusted.afr.dirty=0x000000000000000000000000  
trusted.afr.ovirt-data-client-0=0x000000000000000000000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
trusted.gfid=0xaa32a1b65bf14655aafbeb0f242e61e0  
   
\[root@n34 1d04adc8-9921-4e1c-b192-84fba64224db\]\# getfattr -d -m . -e hex bd3c9ecf-8100-42db-9db5-038dd95a4d54  
\# file: bd3c9ecf-8100-42db-9db5-038dd95a4d54  
trusted.afr.dirty=0x000000000000000000000000  
trusted.afr.ovirt-data-client-0=0x000000000000000000000000  
trusted.afr.ovirt-data-client-1=0x000000000000000000000000  
trusted.gfid=0xaa32a1b65bf14655aafbeb0f242e61e0  
   
   
三、验证  
启动ovirt中对应的vm，系统是windows2008r2，可以启动，有提示异常关闭，类似突然断电，在系统中运行的服务无明显异常。  
符合预期。  
   
四、小结  
建议开启 server 和 client 的 quorum 相关选项来预防脑裂，详情请参考后附redhat的文档链接。

# 参考 #

1、Managing Split-brain  
https://access.redhat.com/documentation/en-US/Red\_Hat\_Storage/3/html/Administration\_Guide/sect-Managing\_Split-brain.html

2  https://blog.csdn.net/lblxyh/article/details/40894635

3 https://joejulian.name/post/what-is-this-new-glusterfs-directory-in-33/