RabbitMQ cluster

朱雀 2022-05-29 08:16 93阅读 0赞

**RabbitMQ概念及环境搭建（三）RabbitMQ cluster**

测试环境：VMS00781 VMS00782 VMS00386 ([CentOS][]5.8)  
1.先在三台机器上分别安装RabbitMQ Server

2.读取其中一个节点的cookie，并复制到其他节点（节点间通过cookie确定相互是否可通信）  
两者之一均可：  
sudo vim /var/lib/rabbitmq/.erlang.cookie  
sudo vim $HOME/.erlang.cookie

3.逐个启动节点  
sudo service rabbitmq-server start

4.查看各节点中的RabbitMQ brokers  
sudo rabbitmqctl cluster\_status

5.建集群  
分别在VMS00386、VMS00782 上执行  
sudo rabbitmqctl stop\_app  
sudo rabbitmqctl join\_cluster --ram rabbit@VMS00781  
sudo rabbitmqctl start\_app  
sudo rabbitmqctl stop\_app  
sudo rabbitmqctl join\_cluster rabbit@VMS00781  
sudo rabbitmqctl start\_app

6.排错  
建集群过程中碰到如下错误：  
sudo rabbitmqctl join\_cluster --ram rabbit@VMS00386  
Clustering node rabbit@VMS00782 with rabbit@VMS00386 ...  
Error: unable to connect to nodes \[rabbit@VMS00386\]: nodedown  
DIAGNOSTICS  
===========  
attempted to contact: \[rabbit@VMS00386\]  
rabbit@VMS00386:  
  \* unable to connect to epmd (port 4369) on VMS00386: nxdomain (non-existing domain)  
current node details:  
\- node name: 'rabbitmqctl-8666@VMS00782'  
\- home dir: /var/lib/rabbitmq  
\- cookie hash: 50YO3zK+HJHos0tab1vHjg==  
解决之道：  
集群节点间需能互相访问，故每个集群节点的hosts文件应包含集群内所有节点的信息以保证互相解析  
vim /etc/hosts  
781's IP  VMS00781  
782's IP  VMS00782  
386's IP  vms00386  
之后重启各节点中的rabbitmq

7.其他问题  
Error: mnesia\_unexpectedly\_running  
原因：忘记先停止stop\_app  
解决：sudo rabbitmqctl stop\_app

若rabbitmq-server第一次启动后hostname不能被解析或者发生了更改则会导致启动失败  
需执行如下操作  
sudo rm -rf /var/lib/rabbitmq/mnesia（因为相关信息会记录在此数据库）  
重装RabbitMQ Server

\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
RabbitMQ cluster 管理  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
1.查看集群状态  
可分别在集群中各个节点执行  
sudo rabbitmqctl cluster\_status

2.更改节点类型（内存型或磁盘型）  
sudo rabbitmqctl stop\_app  
sudo rabbitmqctl change\_cluster\_node\_type disc  
或  
sudo rabbitmqctl change\_cluster\_node\_type ram  
sudo rabbitmqctl start\_app

3.重启cluster中的节点  
停止某个节点或者节点down掉剩余节点不受影响  
\[op1@vms00386 ~\]$ sudo rabbitmqctl stop  
Stopping and halting node rabbit@vms00386 ...

\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00781 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00782,rabbit@VMS00781\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

\[op1@VMS00782 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00782 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00781,rabbit@VMS00782\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

\[op1@VMS00782 ~\]$ sudo rabbitmqctl stop  
Stopping and halting node rabbit@VMS00782 ...

\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00781 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00781\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

待节点重启后自动追上其他节点  
\[op1@vms00386 ~\]$ sudo service rabbitmq-server start  
Starting rabbitmq-server: SUCCESS  
rabbitmq-server.

\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00781 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@vms00386,rabbit@VMS00781\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

\[op1@VMS00782 ~\]$ sudo service rabbitmq-server start  
Starting rabbitmq-server: SUCCESS  
rabbitmq-server.

\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00781 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00782,rabbit@vms00386,rabbit@VMS00781\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

\[op1@VMS00782 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00782 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00781,rabbit@vms00386,rabbit@VMS00782\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

\[op1@vms00386 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@vms00386 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782,rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00782,rabbit@VMS00781,rabbit@vms00386\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

几点注意：  
保证集群中至少有一个磁盘类型的节点以防数据丢失，在更改节点类型时尤其要注意。  
若整个集群被停掉了，应保证最后一个down掉的节点被最先启动，若不能则要使用forget\_cluster\_node命令将其移出集群  
若集群中节点几乎同时以不可控的方式down了此时在其中一个节点使用force\_boot命令重启节点

4.从集群移除节点  
\[op1@vms00386 ~\]$ sudo rabbitmqctl stop\_app  
Stopping node rabbit@vms00386 ...  
\[op1@vms00386 ~\]$ sudo rabbitmqctl reset  
Resetting node rabbit@vms00386 ...  
\[op1@vms00386 ~\]$ sudo rabbitmqctl start\_app  
Starting node rabbit@vms00386 ...

\[op1@vms00386 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@vms00386 ...  
\[\{nodes,\[\{disc,\[rabbit@vms00386\]\}\]\},  
 \{running\_nodes,\[rabbit@vms00386\]\},  
 \{cluster\_name,<<"rabbit@vms00386">>\},  
 \{partitions,\[\]\}\]

\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00781 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00782,rabbit@VMS00781\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

\[op1@VMS00782 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00782 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781,rabbit@VMS00782\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00781,rabbit@VMS00782\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]  
可见rabbit@vms00386成为了独立的节点，原集群只剩rabbit@VMS00781,rabbit@VMS00782了

也可在某个节点移除集群中其他节点  
如继续在rabbit@VMS00781上移除rabbit@VMS00782  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl forget\_cluster\_node rabbit@VMS00782  
Removing node rabbit@VMS00782 from cluster ...

\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status  
Cluster status of node rabbit@VMS00781 ...  
\[\{nodes,\[\{disc,\[rabbit@VMS00781\]\}\]\},  
 \{running\_nodes,\[rabbit@VMS00781\]\},  
 \{cluster\_name,<<"rabbit@VMS00781">>\},  
 \{partitions,\[\]\}\]

可见集群只剩rabbit@VMS00781一个节点了

这里有个问题，在远程其他节点中被移除的节点会自认为仍属于集群

\[op1@VMS00782 ~\]$ sudo rabbitmqctl start\_app  
Starting node rabbit@VMS00782 ...  
BOOT FAILED  
===========  
Error description:  
  \{error,\{inconsistent\_cluster,"Node rabbit@VMS00782 thinks it's clustered with node rabbit@VMS00781, but rabbit@VMS00781 disagrees"\}\}  
Log files (may contain more information):  
  /var/log/rabbitmq/rabbit@VMS00782.log  
  /var/log/rabbitmq/rabbit@VMS00782-sasl.log  
Stack trace:  
  \[\{rabbit\_mnesia,check\_cluster\_consistency,0\},  
    \{rabbit,'-start/0-fun-0-',0\},  
    \{rabbit,start\_it,1\},  
    \{rpc,'-handle\_call\_call/6-fun-0-',5\}\]  
Error: \{rabbit,failure\_during\_boot,  
          \{error,  
              \{inconsistent\_cluster,  
                  "Node rabbit@VMS00782 thinks it's clustered with node rabbit@VMS00781, but rabbit@VMS00781 disagrees"\}\}\}  
需要重置一下  
\[op1@VMS00782 ~\]$ sudo rabbitmqctl reset  
Resetting node rabbit@VMS00782 ...  
\[op1@VMS00782 ~\]$ sudo rabbitmqctl start\_app  
Starting node rabbit@VMS00782 ...

此时三个节点均已成为独立的节点  
其中rabbit@vms00386、rabbit@VMS00782均被重置为了新的RabbitMQ broker而rabbit@VMS00781还保留着原cluster的残留状态可通过如下步骤重置  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl stop\_app  
Stopping node rabbit@VMS00781 ...  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl reset  
Resetting node rabbit@VMS00781 ...  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl start\_app  
Starting node rabbit@VMS00781 ...

5.自动配置cluster  
显然，这是通过配置文件而非命令行工具进行的  
首先重置各节点  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl stop\_app  
Stopping node rabbit@VMS00781 ...  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl reset  
Resetting node rabbit@VMS00781 ...  
...  
其次调整配置文件  
\[\{rabbit,  
  \[\{cluster\_nodes, \{\['rabbit@VMS00781', 'rabbit@VMS00782', 'rabbit@vms00386'\], disc\}\}\]\}\].  
...  
之后启动各节点  
\[op1@VMS00781 ~\]$ sudo service rabbitmq-server start  
Starting rabbitmq-server: SUCCESS  
rabbitmq-server.

查看集群状态  
\[op1@VMS00781 ~\]$ sudo rabbitmqctl cluster\_status

几点注意：  
无论通过命令行还是通过配置文件配置，请确保各节点上Erlang和RabbitMQ版本一致  
配置文件仅对新鲜节点有效，也即被reset或者第一次启动的节点。因此在重启节点后自动化集群过程并不会发生。也以为这通过rabbitmq进行的改变优先于自动化集群配置。

在一台机器上部署集群，一般用户测试集群特性  
这里的关键是已不同的端口可节点名称启动多个rabbitmq-server实例，其余过程同多机器上部署集群类似

其他注意事项：  
如防火墙策略等

参考：

http://www.rabbitmq.com/clustering.html

https://www.linuxidc.com/Linux/2014-12/110449p3.htm

openstack官方：https://docs.openstack.org/ha-guide/shared-messaging.html

问题参考：https://stackoverflow.com/questions/15720540/ubuntu-rabbitmq-error-unable-to-connect-to-node-rabbitsomename-nodedown

[CentOS]: https://www.linuxidc.com/topicnews.aspx?tid=14