Mysql MHA搭建以及故障切换测试

柔光的暖阳◎ 2022-06-17 06:43 396阅读 0赞

测试环境信息：  
OS:Centos6.8  
192.168.180.55  manager  
192.168.180.56  master  
192.168.180.57  slave1  
192.168.180.58  slave2  
  
1.架构  
从架构上来说，MHA分为如下两大部分：  
（1） Node  
我们知道，MHA是基于MySQL Replication环境的，在该环境中，不管是Master角色，还是Slave角色，都称为Node，是被监控管理的对象节点。  
Node服务器上需要安装MHA Node包。  
（2） Manager  
Manager为MHA架构中的管理者，建议部署在一台独立的服务器上，当然也可部署在某个Slave上，但该Slave永远不要被选择成为新的Master，否则故障切换后的MHA架构就失去了高可用性。  
Manager服务器需要安装MHA Manager包，并完善一个主配置文件。  
一个Manager可管理多套MySQL Replication环境。

2.工作原理：

相较于其它HA软件，MHA的目的在于维持MySQL Replication中Master库的高可用性，其最大特点是可以修复多个Slave之间的差异日志，最终使所有Slave保持数据一致，然后从中选择一个充当新的Master，并将其它Slave指向它。  
基本工作流程大致如下：  
（1） Manager定期监控Master，监控时间间隔由参数ping\_interval决定，缺省为3秒钟一次；可利用其自身的监控功能，也可调用第三方软件来监控；MHA自身提供了两种监控方式：SELECT（执行SELECT 1）和CONNECT（创建连接/断开连接），由于参数ping\_type决定，缺省为SELECT方式。  
（2） 当监测到Master故障时，调用SSH脚本对所有Node执行一次检查，包括如下几个方面：  
――MySQL实例是否可以连接；  
――Master服务器是否可以SSH连通；  
――检查SQL Thread的状态；  
――检查哪些Server死掉了，哪些Server是活动的，以及活动的Slave实例；  
――检查Slave实例的配置及复制过滤规则；  
――最后退出监控脚本并返回代表特殊意义代码。  
（3） 开始Master故障切换，包括如下几个子阶段：  
――Phase 1: Configuration Check Phase  
在这个阶段，若某个Slave实例的SQL Thread停止了，则会自动启动它；并再次确认活动的Servers及Slaves。  
――Phase 2: Dead Master Shutdown Phase  
在这个阶段，首先调用master\_ip\_failover\_script，若HA是基于VIP实现的，则关闭VIP，若是基于目录数据库实现的，则修改映射记录。  
然后调用shutdown\_script脚本强制关闭主机，以避免服务重启时，发生脑裂。  
――Phase 3: Master Recovery Phase  
又包括如下3个子阶段：  
Phase 3.1: Getting Latest Slaves Phase  
检查各个Slave，获取最近的和最旧的binary log file和position，并检查各个Slave成为Master的优先级，依赖于candidate\_master、no\_master、\[server\_xxx\]顺序、binary log差异量等因素。  
Phase 3.2: Saving Dead Master's Binlog Phase  
若dead master所在服务器依然可以通过SSH连通，则提取dead master的binary log，提取日志的起点就是上一步获取的最新的binary log file和position，直到最后一条事件日志，并在dead master本地的工作目录（由参数remote\_workdir决定）中创建文件保存这些提取到的日志，然后将该文件拷贝到Manager服务器的工作目录下（由参数manager\_workdir决定）。  
当然，若dead master系统就无法连接，也就不存在差异的binary log了。  
另外，MHA还要对各个Slave节点进行健康检查，主要是SSH连通性。  
Phase 3.3: Determining New Master Phase  
接下来调用apply\_diff\_relay\_logs命令恢复Slave的差异日志，这个差异日志指的是各个Slave之间的relay log。  
恢复完成后，所有的Slave数据是一致的，此时就可以根据优先级选择New Master了。  
Phase 3.3: New Master Diff Log Generation Phase  
这里是生成dead master和new master之间的差异日志，即将Phase 3.2保存的binary log拷贝到New Master的工作目录中（remote\_workdir）。  
Phase 3.4: Master Log Apply Phase  
将上一步拷贝的差异日志恢复到New Master上，若发生错误，也可手动恢复。  
然后获取New Master的binlog name和position，以便其它Slave从这个新的binlog name和position开始复制。  
最后会开启New Master的写权限，即将read\_only参数设置为0。  
――Phase 4: Slaves Recovery Phase  
Phase 4.1: Starting Parallel Slave Diff Log Generation Phase  
生成Slave与New Slave之间的差异日志，并将该日志拷贝到各Slave的工作目录下，这部分日志dead master和new master之间差异的那部分日志，因为各个Slave在Phase 3.3阶段已经同步了。  
Phase 4.2: Starting Parallel Slave Log Apply Phase  
在各个Slave上应用这部分差异日志，然后通过CHANGE MASTER TO命令将这些Slave指向新的New Master，最后开始复制(start slave)。  
――Phase 5: New master cleanup phase  
清理New Master其实就是重置slave info，即取消原来的Slave信息。  
至此整个Master故障切换过程完成。  
3.功能  
从官方网站的介绍来看，MHA具有如下几个功能：  
（1） Master自动监控和故障转移  
基于现有的MySQL主从复制环境，MHA可以监控Master，当发现其故障时，自动进行切换。  
在多个Slave环境中，如果个别Slave没有接受到最新的relay log events，MHA则会自动从最新的那个Slave上查找差异的relay log events，并将这些差异事件应用到有延迟的Slave上，最终保持所有的Slave数据一致。通常情况下，MHA可在9-12秒内监测到Master故障，7-10秒内关闭主机以避免脑裂，然后花费几秒时间应用差异的relay log，整个过程通常只需10－30秒即可完成。  
既然MHA可以自动修复多个Slaves之间的差异日志，所以不用担心数据一致性问题。当Master故障时，MHA会从多个Slave中随机选择一个充当新的Master；当然，也可在配置文件中指定某一个Slave优先成为Master。  
（2） 互动（手动）Master故障转移  
可以只使用MHA的故障转移功能，而不监控Master，当其故障时，手动调用MHA来进行故障切换。  
（3）非交互式自动故障转移  
MHA还支持非交互式的Master故障切换（不监控Master，但实现自动故障切换），这个特性其实是将Master的监控和VIP接管交给第三方工具来做，比如Heartbeat，MHA只负责Master故障转移和Slave之间的数据同步。  
（4） 在线切换Master到不同的主机

在有些情况下，比如更换Raid控制器，升级主机硬件等，则需要将其上的Master实例迁移到其它主机上，MHA支持快速的Master切换和短暂的写操作阻塞，通常只需0.5－2秒的downtime，这是可以接受的，也方便了DBA的操作

4.搭建环境：  
系统环境：  
os：centos6.8  
192.168.180.55  manager  
192.168.180.56  master  
192.168.180.57  slave1  
192.168.180.58  slave2  
  
  
a）安装mysql  
  
  
这里将不再赘述，详细请参考 http://blog.csdn.net/shiyu1157758655/article/details/70226036  
  
  
b）配置SSH等效连接  
  
  
这里也不再赘述，详细请参考 http://blog.csdn.net/shiyu1157758655/article/details/56838603  
  
  
c）安装MHA  
  
  
HA的安装包也包括Manager和Node两部分，其中Node包不但要在所有的Node节点上安装，而且还需在Manager节点上安装，因为Manager模块内部依赖Node模块，  
Manager包则只需在Manager节点安装即可。  
MHA是采用perl语言编写的一个脚本管理工具，所以需要安装一系列perl依赖包。  
（1）在所有节点上安装perl语言包  
   rpm -ivh MySQL-shared-compat-5.6.14-1.rhel5.x86\_64.rpm   
   rpm -ivh perl-DBI-1.609-4.el6.x86\_64.rpm  
   rpm -ivh perl-DBD-MySQL-4.013-3.el6.x86\_64.rpm   
  
  
（2）以下依赖包为manager 节点需要  
   rpm -ivh perl-Params-Validate-0.95-1.el6.rfx.x86\_64.rpm   
   rpm -ivh perl-Log-Dispatch-2.26-1.el6.rf.noarch.rpm  
   rpm -ivh perl-Config-Tiny-2.12-7.1.el6.noarch.rpm  
   rpm -ivh perl-Parallel-ForkManager-0.7.5-2.2.el6.rf.noarch.rpm  
（3）在所有节点以及manager上安装node包  
   rpm  -ivh mha4mysql-node-0.54-0.el6.noarch.rpm  
（4）在manager节点上安装manager包  
   rpm  -ivh mha4mysql-manager-0.55-0.el6.noarch.rpm  
  成功安装后，会在/usr/bin 目录下生成如下一系列命令工具  
 \[root@manager bin\]\# cd /usr/bin/  
\[root@manager bin\]\# ll masterha\_\*  
\-rwxr-xr-x 1 root root 1995 Dec 13  2012 masterha\_check\_repl  
\-rwxr-xr-x 1 root root 1779 Dec 13  2012 masterha\_check\_ssh  
\-rwxr-xr-x 1 root root 1865 Dec 13  2012 masterha\_check\_status  
\-rwxr-xr-x 1 root root 3201 Dec 13  2012 masterha\_conf\_host  
\-rwxr-xr-x 1 root root 2517 Dec 13  2012 masterha\_manager  
\-rwxr-xr-x 1 root root 2165 Dec 13  2012 masterha\_master\_monitor  
\-rwxr-xr-x 1 root root 2373 Dec 13  2012 masterha\_master\_switch  
\-rwxr-xr-x 1 root root 3879 Dec 13  2012 masterha\_secondary\_check  
\-rwxr-xr-x 1 root root 1739 Dec 13  2012 masterha\_stop  
d）配置MHA  
下来就可以配置MHA配置文件了，只需在Manager服务器上操作；  
RPM包安装时，缺省不会生成该配置文件，可手动生成，也可从MHA Manager源码安装包中查找配置文件模板，及一系列调用脚本  
这里使用manager源码安装包的配置文件模板  
\[root@manager ~\]\# mkdir /etc/masterha/  
\[root@manager master\]\# unzip mha4mysql-manager-master.zip   
解压后查看  
\[root@manager conf\]\# pwd  
/data/src/master/mha4mysql-manager-master/samples/conf  
\[root@manager conf\]\# ll  
total 8  
\-rw-r--r-- 1 root root 257 Nov  6  2014 app1.cnf  
\-rw-r--r-- 1 root root 475 Nov  6  2014 masterha\_default.cnf  
\[root@manager conf\]\#   
将app1.cnf复制到/etc/masterha/下  
\[root@manager conf\]\# cp app1.cnf /etc/masterha/  
编辑app1.cnf  
\[root@manager conf\]\# vi /etc/masterha/app1.cnf   
内容如下：  
\[server default\]  
user=mha         \#mha管理的用户名  
password=mysql   
manager\_workdir=/data/mha  
manager\_log=/data/mha/manager.log  
remote\_workdir=/data/mha  
ssh\_user=root    \#上面双机互相的用户  
repl\_user=mysql      \#mysql复制（同步）的用户  
repl\_password=mysql   
ping\_interval=1  
secondary\_check\_script= masterha\_secondary\_check -s 192.168.180.56 -s 192.168.180.57 -s 192.168.180.58  
\#report\_script=/data/mha/scripts/send\_report      \#设置发生切换后发送的报警的脚本  
\#master\_ip\_online\_change\_script=/data/mha/scripts/master\_ip\_online\_change \#设置手动切换时候的切换脚本  
注意：这里的send\_report，master\_ip\_online\_change 可以在刚才解压的文件里找到，可以copy到相应的位置  
\[server1\]  
hostname=192.168.180.56    \#写ip或主机名都可以  
ssh\_port=22  
master\_binlog\_dir=/data/mysql  \#设置master 保存binlog的位置，以便MHA可以找到master的日志  
candidate\_master=1  
\[server2\]  
hostname=192.168.180.57  
ssh\_port=22  
master\_binlog\_dir=/data/mysql  
candidate\_master=1  
  
  
\[server3\]  
hostname=192.168.180.58  
ssh\_port=22  
master\_binlog\_dir=/data/mysql  
no\_master=1  
~               
e）验证ssh通讯  
  
  
root@manager bin\]\# masterha\_check\_ssh --conf=/etc/masterha/app1.cnf   
Wed Apr 19 13:06:39 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 13:06:39 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 13:06:39 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 13:06:39 2017 - \[info\] Starting SSH connection tests..  
Wed Apr 19 13:06:40 2017 - \[debug\]   
Wed Apr 19 13:06:39 2017 - \[debug\]  Connecting via SSH from root@192.168.180.56(192.168.180.56:22) to root@192.168.180.57(192.168.180.57:22)..  
Wed Apr 19 13:06:39 2017 - \[debug\]   ok.  
Wed Apr 19 13:06:39 2017 - \[debug\]  Connecting via SSH from root@192.168.180.56(192.168.180.56:22) to root@192.168.180.58(192.168.180.58:22)..  
Wed Apr 19 13:06:39 2017 - \[debug\]   ok.  
Wed Apr 19 13:06:40 2017 - \[debug\]   
Wed Apr 19 13:06:40 2017 - \[debug\]  Connecting via SSH from root@192.168.180.57(192.168.180.57:22) to root@192.168.180.56(192.168.180.56:22)..  
Wed Apr 19 13:06:40 2017 - \[debug\]   ok.  
Wed Apr 19 13:06:40 2017 - \[debug\]  Connecting via SSH from root@192.168.180.57(192.168.180.57:22) to root@192.168.180.58(192.168.180.58:22)..  
Wed Apr 19 13:06:40 2017 - \[debug\]   ok.  
Wed Apr 19 13:06:41 2017 - \[debug\]   
Wed Apr 19 13:06:40 2017 - \[debug\]  Connecting via SSH from root@192.168.180.58(192.168.180.58:22) to root@192.168.180.56(192.168.180.56:22)..  
Wed Apr 19 13:06:40 2017 - \[debug\]   ok.  
Wed Apr 19 13:06:40 2017 - \[debug\]  Connecting via SSH from root@192.168.180.58(192.168.180.58:22) to root@192.168.180.57(192.168.180.57:22)..  
Wed Apr 19 13:06:40 2017 - \[debug\]   ok.  
Wed Apr 19 13:06:41 2017 - \[info\] All SSH connection tests passed successfully  
  
  
f）验证mysql复制，mysql必须都启动  
  
  
\[root@manager ~\]\# masterha\_check\_repl --conf=/etc/masterha/app1.cnf   
Wed Apr 19 15:21:53 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 15:21:53 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 15:21:53 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 15:21:53 2017 - \[info\] MHA::MasterMonitor version 0.55.  
Wed Apr 19 15:21:53 2017 - \[info\] Dead Servers:  
Wed Apr 19 15:21:53 2017 - \[info\] Alive Servers:  
Wed Apr 19 15:21:53 2017 - \[info\]   192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:21:53 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 15:21:53 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  
Wed Apr 19 15:21:53 2017 - \[info\] Alive Slaves:  
Wed Apr 19 15:21:53 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:21:53 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:21:53 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 15:21:53 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:21:53 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:21:53 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 15:21:53 2017 - \[info\] Current Alive Master: 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:21:53 2017 - \[info\] Checking slave configurations..  
Wed Apr 19 15:21:53 2017 - \[info\]  read\_only=1 is not set on slave 192.168.180.57(192.168.180.57:3306).  
Wed Apr 19 15:21:53 2017 - \[info\] Checking replication filtering settings..  
Wed Apr 19 15:21:53 2017 - \[info\]  binlog\_do\_db= , binlog\_ignore\_db=   
Wed Apr 19 15:21:53 2017 - \[info\]  Replication filtering check ok.  
Wed Apr 19 15:21:53 2017 - \[info\] Starting SSH connection tests..  
Wed Apr 19 15:21:55 2017 - \[info\] All SSH connection tests passed successfully.  
Wed Apr 19 15:21:55 2017 - \[info\] Checking MHA Node version..  
Wed Apr 19 15:21:55 2017 - \[info\]  Version check ok.  
Wed Apr 19 15:21:55 2017 - \[info\] Checking SSH publickey authentication settings on the current master..  
Wed Apr 19 15:21:55 2017 - \[info\] HealthCheck: SSH to 192.168.180.56 is reachable.  
Wed Apr 19 15:21:55 2017 - \[info\] Master MHA Node version is 0.54.  
Wed Apr 19 15:21:55 2017 - \[info\] Checking recovery script configurations on the current master..  
Wed Apr 19 15:21:55 2017 - \[info\]   Executing command: save\_binary\_logs --command=test --start\_pos=4 --binlog\_dir=/data/mysql --output\_file=/data/mha/save\_binary\_logs\_test --manager\_version=0.55 --start\_file=mysql-bin.000001   
Wed Apr 19 15:21:55 2017 - \[info\]   Connecting to root@192.168.180.56(192.168.180.56)..   
  Creating /data/mha if not exists..    ok.  
  Checking output directory is accessible or not..  
   ok.  
  Binlog found at /data/mysql, up to mysql-bin.000001  
Wed Apr 19 15:21:55 2017 - \[info\] Master setting check done.  
Wed Apr 19 15:21:55 2017 - \[info\] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..  
Wed Apr 19 15:21:55 2017 - \[info\]   Executing command : apply\_diff\_relay\_logs --command=test --slave\_user='mha' --slave\_host=192.168.180.57 --slave\_ip=192.168.180.57 --slave\_port=3306 --workdir=/data/mha --target\_version=5.7.15-log --manager\_version=0.55 --relay\_log\_info=/data/mysql/relay-log.info  --relay\_dir=/data/mysql/  --slave\_pass=xxx  
Wed Apr 19 15:21:55 2017 - \[info\]   Connecting to root@192.168.180.57(192.168.180.57:22)..   
  Checking slave recovery environment settings..  
    Opening /data/mysql/relay-log.info ... ok.  
    Relay log found at /data/mysql, up to slave1-relay-bin.000002  
    Temporary relay log file is /data/mysql/slave1-relay-bin.000002  
    Testing mysql connection and privileges..mysql: \[Warning\] Using a password on the command line interface can be insecure.  
 done.  
    Testing mysqlbinlog output.. done.  
    Cleaning up test file(s).. done.  
Wed Apr 19 15:21:56 2017 - \[info\]   Executing command : apply\_diff\_relay\_logs --command=test --slave\_user='mha' --slave\_host=192.168.180.58 --slave\_ip=192.168.180.58 --slave\_port=3306 --workdir=/data/mha --target\_version=5.7.15-log --manager\_version=0.55 --relay\_log\_info=/data/mysql/relay-log.info  --relay\_dir=/data/mysql/  --slave\_pass=xxx  
Wed Apr 19 15:21:56 2017 - \[info\]   Connecting to root@192.168.180.58(192.168.180.58:22)..   
  Checking slave recovery environment settings..  
    Opening /data/mysql/relay-log.info ... ok.  
    Relay log found at /data/mysql, up to mysql-relay-bin.000002  
    Temporary relay log file is /data/mysql/mysql-relay-bin.000002  
    Testing mysql connection and privileges..mysql: \[Warning\] Using a password on the command line interface can be insecure.  
 done.  
    Testing mysqlbinlog output.. done.  
    Cleaning up test file(s).. done.  
Wed Apr 19 15:21:56 2017 - \[info\] Slaves settings check done.  
Wed Apr 19 15:21:56 2017 - \[info\]   
192.168.180.56 (current master)  
 +--192.168.180.57  
 +--192.168.180.58  
  
  
Wed Apr 19 15:21:56 2017 - \[info\] Checking replication health on 192.168.180.57..  
Wed Apr 19 15:21:56 2017 - \[info\]  ok.  
Wed Apr 19 15:21:56 2017 - \[info\] Checking replication health on 192.168.180.58..  
Wed Apr 19 15:21:56 2017 - \[info\]  ok.  
Wed Apr 19 15:21:56 2017 - \[warning\] master\_ip\_failover\_script is not defined.  
Wed Apr 19 15:21:56 2017 - \[warning\] shutdown\_script is not defined.  
Wed Apr 19 15:21:56 2017 - \[info\] Got exit code 0 (Not master dead).  
  
  
MySQL Replication Health is OK.  
  
  
如果有报错，请根据上面的报错排查  
  
  
g）启动MHA  
\[root@manager ~\]\# nohup masterha\_manager --conf=/etc/masterha/app1.cnf </dev/null >/data/mha/mag.log 2>&1 &  
查看：  
\[root@manager ~\]\# masterha\_check\_status --conf=/etc/masterha/app1.cnf   
app1 (pid:5181) is running(0:PING\_OK), master:192.168.180.56  
\[root@manager ~\]\#   
  
  
\[root@manager scripts\]\# tail -f /data/mha/manager.log   
192.168.180.56 (current master)  
 +--192.168.180.57  
 +--192.168.180.58  
  
  
Wed Apr 19 14:12:45 2017 - \[warning\] master\_ip\_failover\_script is not defined.  
Wed Apr 19 14:12:45 2017 - \[warning\] shutdown\_script is not defined.  
Wed Apr 19 14:12:45 2017 - \[info\] Set master ping interval 1 seconds.  
Wed Apr 19 14:12:45 2017 - \[info\] Set secondary check script: masterha\_secondary\_check -s 192.168.180.56 -s 192.168.180.57 -s 192.168.180.58  
Wed Apr 19 14:12:45 2017 - \[info\] Starting ping health check on 192.168.180.56(192.168.180.56:3306)..  
Wed Apr 19 14:12:45 2017 - \[info\] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..  
  
  
5.测试master(56)宕机后，是否会自动切换  
\[root@master src\]\# /etc/init.d/mysql status  
MySQL running (3629)                                       \[  OK  \]  
\[root@master src\]\# /etc/init.d/mysql stop  
Shutting down MySQL............                            \[  OK  \]  
\[root@master src\]\#   
  
  
查看manager日志  
  
  
\[root@manager scripts\]\# tail -f /data/mha/manager.log   
192.168.180.56 (current master)  
 +--192.168.180.57  
 +--192.168.180.58  
  
  
Wed Apr 19 14:12:45 2017 - \[warning\] master\_ip\_failover\_script is not defined.  
Wed Apr 19 14:12:45 2017 - \[warning\] shutdown\_script is not defined.  
Wed Apr 19 14:12:45 2017 - \[info\] Set master ping interval 1 seconds.  
Wed Apr 19 14:12:45 2017 - \[info\] Set secondary check script: masterha\_secondary\_check -s 192.168.180.56 -s 192.168.180.57 -s 192.168.180.58  
Wed Apr 19 14:12:45 2017 - \[info\] Starting ping health check on 192.168.180.56(192.168.180.56:3306)..  
Wed Apr 19 14:12:45 2017 - \[info\] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..  
Wed Apr 19 15:45:30 2017 - \[warning\] Got error on MySQL select ping: 2006 (MySQL server has gone away)  
Wed Apr 19 15:45:30 2017 - \[info\] Executing seconary network check script: masterha\_secondary\_check -s 192.168.180.56 -s 192.168.180.57 -s 192.168.180.58  --user=root  --master\_host=192.168.180.56  --master\_ip=192.168.180.56  --master\_port=3306  
Wed Apr 19 15:45:30 2017 - \[info\] Executing SSH check script: save\_binary\_logs --command=test --start\_pos=4 --binlog\_dir=/data/mysql --output\_file=/data/mha/save\_binary\_logs\_test --manager\_version=0.55 --binlog\_prefix=mysql-bin  
Wed Apr 19 15:45:30 2017 - \[info\] HealthCheck: SSH to 192.168.180.56 is reachable.  
Monitoring server 192.168.180.56 is reachable, Master is not reachable from 192.168.180.56. OK.  
Monitoring server 192.168.180.57 is reachable, Master is not reachable from 192.168.180.57. OK.  
Monitoring server 192.168.180.58 is reachable, Master is not reachable from 192.168.180.58. OK.  
Wed Apr 19 15:45:31 2017 - \[info\] Master is not reachable from all other monitoring servers. Failover should start.  
Wed Apr 19 15:45:31 2017 - \[warning\] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)  
Wed Apr 19 15:45:31 2017 - \[warning\] Connection failed 1 time(s)..  
Wed Apr 19 15:45:32 2017 - \[warning\] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)  
Wed Apr 19 15:45:32 2017 - \[warning\] Connection failed 2 time(s)..  
Wed Apr 19 15:45:33 2017 - \[warning\] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)  
Wed Apr 19 15:45:33 2017 - \[warning\] Connection failed 3 time(s)..  
Wed Apr 19 15:45:33 2017 - \[warning\] Master is not reachable from health checker!  
Wed Apr 19 15:45:33 2017 - \[warning\] Master 192.168.180.56(192.168.180.56:3306) is not reachable!  
Wed Apr 19 15:45:33 2017 - \[warning\] SSH is reachable.  
Wed Apr 19 15:45:33 2017 - \[info\] Connecting to a master server failed. Reading configuration file /etc/masterha\_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..  
Wed Apr 19 15:45:33 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 15:45:33 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 15:45:33 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 15:45:33 2017 - \[info\] Dead Servers:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\] Alive Servers:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  
Wed Apr 19 15:45:33 2017 - \[info\] Alive Slaves:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\] Checking slave configurations..  
Wed Apr 19 15:45:33 2017 - \[info\]  read\_only=1 is not set on slave 192.168.180.57(192.168.180.57:3306).  
Wed Apr 19 15:45:33 2017 - \[info\] Checking replication filtering settings..  
Wed Apr 19 15:45:33 2017 - \[info\]  Replication filtering check ok.  
Wed Apr 19 15:45:33 2017 - \[info\] Master is down!  
Wed Apr 19 15:45:33 2017 - \[info\] Terminating monitoring script.  
Wed Apr 19 15:45:33 2017 - \[info\] Got exit code 20 (Master dead).  
Wed Apr 19 15:45:33 2017 - \[info\] MHA::MasterFailover version 0.55.  
Wed Apr 19 15:45:33 2017 - \[info\] Starting master failover.  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] \* Phase 1: Configuration Check Phase..  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] Dead Servers:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\] Checking master reachability via mysql(double check)..  
Wed Apr 19 15:45:33 2017 - \[info\]  ok.  
Wed Apr 19 15:45:33 2017 - \[info\] Alive Servers:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  
Wed Apr 19 15:45:33 2017 - \[info\] Alive Slaves:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\] \*\* Phase 1: Configuration Check Phase completed.  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] \* Phase 2: Dead Master Shutdown Phase..  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] Forcing shutdown so that applications never connect to the current master..  
Wed Apr 19 15:45:33 2017 - \[warning\] master\_ip\_failover\_script is not set. Skipping invalidating dead master ip address.  
Wed Apr 19 15:45:33 2017 - \[warning\] shutdown\_script is not set. Skipping explicit shutting down of the dead master.  
Wed Apr 19 15:45:33 2017 - \[info\] \* Phase 2: Dead Master Shutdown Phase completed.  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] \* Phase 3: Master Recovery Phase..  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] \* Phase 3.1: Getting Latest Slaves Phase..  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] The latest binary log file/position on all slaves is mysql-bin.000001:1299  
Wed Apr 19 15:45:33 2017 - \[info\] Latest slaves (Slaves that received relay log files to the latest):  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\] The oldest binary log file/position on all slaves is mysql-bin.000001:1299  
Wed Apr 19 15:45:33 2017 - \[info\] Oldest slaves:  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:33 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:33 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] \* Phase 3.2: Saving Dead Master's Binlog Phase..  
Wed Apr 19 15:45:33 2017 - \[info\]   
Wed Apr 19 15:45:33 2017 - \[info\] Fetching dead master's binary logs..  
Wed Apr 19 15:45:33 2017 - \[info\] Executing command on the dead master 192.168.180.56(192.168.180.56:3306): save\_binary\_logs --command=save --start\_file=mysql-bin.000001  --start\_pos=1299 --binlog\_dir=/data/mysql --output\_file=/data/mha/saved\_master\_binlog\_from\_192.168.180.56\_3306\_20170419154533.binlog --handle\_raw\_binlog=1 --disable\_log\_bin=0 --manager\_version=0.55  
  Creating /data/mha if not exists..    ok.  
 Concat binary/relay logs from mysql-bin.000001 pos 1299 to mysql-bin.000001 EOF into /data/mha/saved\_master\_binlog\_from\_192.168.180.56\_3306\_20170419154533.binlog ..  
  Dumping binlog format description event, from position 0 to 123.. ok.  
  No need to dump effective binlog data from /data/mysql/mysql-bin.000001 (pos starts 1299, filesize 1299). Skipping.  
 /data/mha/saved\_master\_binlog\_from\_192.168.180.56\_3306\_20170419154533.binlog has no effective data events.  
Event not exists.  
Wed Apr 19 15:45:34 2017 - \[info\] Additional events were not found from the orig master. No need to save.  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 3.3: Determining New Master Phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] Finding the latest slave that has all relay logs for recovering other slaves..  
Wed Apr 19 15:45:34 2017 - \[info\] All slaves received relay logs to the same position. No need to resync each other.  
Wed Apr 19 15:45:34 2017 - \[info\] Searching new master from slaves..  
Wed Apr 19 15:45:34 2017 - \[info\]  Candidate masters from the configuration file:  
Wed Apr 19 15:45:34 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:34 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:34 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 15:45:34 2017 - \[info\]  Non-candidate masters:  
Wed Apr 19 15:45:34 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 15:45:34 2017 - \[info\]     Replicating from 192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 15:45:34 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 15:45:34 2017 - \[info\]  Searching from candidate\_master slaves which have received the latest relay log events..  
Wed Apr 19 15:45:34 2017 - \[info\] New master is 192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 15:45:34 2017 - \[info\] Starting master failover..  
Wed Apr 19 15:45:34 2017 - \[info\]   
From:  
192.168.180.56 (current master)  
 +--192.168.180.57  
 +--192.168.180.58  
  
  
To:  
192.168.180.57 (new master)  
 +--192.168.180.58  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 3.3: New Master Diff Log Generation Phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\]  This server has all relay logs. No need to generate diff files from the latest slave.  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 3.4: Master Log Apply Phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \*NOTICE: If any error happens from this phase, manual recovery is needed.  
Wed Apr 19 15:45:34 2017 - \[info\] Starting recovery on 192.168.180.57(192.168.180.57:3306)..  
Wed Apr 19 15:45:34 2017 - \[info\]  This server has all relay logs. Waiting all logs to be applied..   
Wed Apr 19 15:45:34 2017 - \[info\]   done.  
Wed Apr 19 15:45:34 2017 - \[info\]  All relay logs were successfully applied.  
Wed Apr 19 15:45:34 2017 - \[info\] Getting new master's binlog name and position..  
Wed Apr 19 15:45:34 2017 - \[info\]  mysql-bin.000002:893  
Wed Apr 19 15:45:34 2017 - \[info\]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER\_HOST='192.168.180.57', MASTER\_PORT=3306, MASTER\_LOG\_FILE='mysql-bin.000002', MASTER\_LOG\_POS=893, MASTER\_USER='mysql', MASTER\_PASSWORD='xxx';  
Wed Apr 19 15:45:34 2017 - \[warning\] master\_ip\_failover\_script is not set. Skipping taking over new master ip address.  
Wed Apr 19 15:45:34 2017 - \[info\] \*\* Finished master recovery successfully.  
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 3: Master Recovery Phase completed.  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 4: Slaves Recovery Phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] -- Slave diff file generation on host 192.168.180.58(192.168.180.58:3306) started, pid: 11789. Check tmp log /data/mha/192.168.180.58\_3306\_20170419154533.log if it takes time..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] Log messages from 192.168.180.58 ...  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\]  This server has all relay logs. No need to generate diff files from the latest slave.  
Wed Apr 19 15:45:34 2017 - \[info\] End of log messages from 192.168.180.58.  
Wed Apr 19 15:45:34 2017 - \[info\] -- 192.168.180.58(192.168.180.58:3306) has the latest relay log events.  
Wed Apr 19 15:45:34 2017 - \[info\] Generating relay diff files from the latest slave succeeded.  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 4.2: Starting Parallel Slave Log Apply Phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] -- Slave recovery on host 192.168.180.58(192.168.180.58:3306) started, pid: 11791. Check tmp log /data/mha/192.168.180.58\_3306\_20170419154533.log if it takes time..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] Log messages from 192.168.180.58 ...  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] Starting recovery on 192.168.180.58(192.168.180.58:3306)..  
Wed Apr 19 15:45:34 2017 - \[info\]  This server has all relay logs. Waiting all logs to be applied..   
Wed Apr 19 15:45:34 2017 - \[info\]   done.  
Wed Apr 19 15:45:34 2017 - \[info\]  All relay logs were successfully applied.  
Wed Apr 19 15:45:34 2017 - \[info\]  Resetting slave 192.168.180.58(192.168.180.58:3306) and starting replication from the new master 192.168.180.57(192.168.180.57:3306)..  
Wed Apr 19 15:45:34 2017 - \[info\]  Executed CHANGE MASTER.  
Wed Apr 19 15:45:34 2017 - \[info\]  Slave started.  
Wed Apr 19 15:45:34 2017 - \[info\] End of log messages from 192.168.180.58.  
Wed Apr 19 15:45:34 2017 - \[info\] -- Slave recovery on host 192.168.180.58(192.168.180.58:3306) succeeded.  
Wed Apr 19 15:45:34 2017 - \[info\] All new slave servers recovered successfully.  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] \* Phase 5: New master cleanup phase..  
Wed Apr 19 15:45:34 2017 - \[info\]   
Wed Apr 19 15:45:34 2017 - \[info\] Resetting slave info on the new master..  
Wed Apr 19 15:45:34 2017 - \[info\]  192.168.180.57: Resetting slave info succeeded.  
Wed Apr 19 15:45:34 2017 - \[info\] Master failover to 192.168.180.57(192.168.180.57:3306) completed successfully.  
Wed Apr 19 15:45:34 2017 - \[info\]   
  
  
\----- Failover Report -----  
  
  
app1: MySQL Master failover 192.168.180.56 to 192.168.180.57 succeeded  
  
  
Master 192.168.180.56 is down!  
  
  
Check MHA Manager logs at manager:/data/mha/manager.log for details.  
  
  
Started automated(non-interactive) failover.  
The latest slave 192.168.180.57(192.168.180.57:3306) has all relay logs for recovery.  
Selected 192.168.180.57 as a new master.  
192.168.180.57: OK: Applying all logs succeeded.  
192.168.180.58: This host has the latest relay log events.  
Generating relay diff files from the latest slave succeeded.  
192.168.180.58: OK: Applying all logs succeeded. Slave started, replicating from 192.168.180.57.  
192.168.180.57: Resetting slave info succeeded.

Master failover to 192.168.180.57(192.168.180.57:3306) completed successfully.

切换主机后，检查replication状态  
  
  
\[root@manager ~\]\# masterha\_check\_repl --conf /etc/masterha/app1.cnf   
Wed Apr 19 16:07:29 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 16:07:29 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:07:29 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:07:29 2017 - \[info\] MHA::MasterMonitor version 0.55.  
Wed Apr 19 16:07:29 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/ServerManager.pm, ln604\] There are 2 non-slave servers! MHA manages at most one non-slave server. Check configurations.  
Wed Apr 19 16:07:29 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm, ln386\] Error happend on checking configurations.  at /usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm line 300  
Wed Apr 19 16:07:29 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm, ln482\] Error happened on monitoring servers.  
Wed Apr 19 16:07:29 2017 - \[info\] Got exit code 1 (Not master dead).  
  
  
MySQL Replication Health is NOT OK!  
\[root@manager ~\]\# masterha\_check\_repl --conf /etc/masterha/app1.cnf   
Wed Apr 19 16:08:41 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 16:08:41 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:08:41 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:08:41 2017 - \[info\] MHA::MasterMonitor version 0.55.  
Wed Apr 19 16:08:42 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/ServerManager.pm, ln604\] There are 2 non-slave servers! MHA manages at most one non-slave server. Check configurations.  
Wed Apr 19 16:08:42 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm, ln386\] Error happend on checking configurations.  at /usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm line 300  
Wed Apr 19 16:08:42 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm, ln482\] Error happened on monitoring servers.  
Wed Apr 19 16:08:42 2017 - \[info\] Got exit code 1 (Not master dead).  
  
  
MySQL Replication Health is NOT OK!  
\[root@manager ~\]\# masterha\_check\_repl --conf /etc/masterha/app1.cnf   
Wed Apr 19 16:14:17 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 16:14:17 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:14:17 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:14:17 2017 - \[info\] MHA::MasterMonitor version 0.55.  
Wed Apr 19 16:14:18 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/ServerManager.pm, ln604\] There are 2 non-slave servers! MHA manages at most one non-slave server. Check configurations.  
Wed Apr 19 16:14:18 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm, ln386\] Error happend on checking configurations.  at /usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm line 300  
Wed Apr 19 16:14:18 2017 - \[error\]\[/usr/share/perl5/vendor\_perl/MHA/MasterMonitor.pm, ln482\] Error happened on monitoring servers.  
Wed Apr 19 16:14:18 2017 - \[info\] Got exit code 1 (Not master dead).  
  
  
MySQL Replication Health is NOT OK!  
  
  
关于这个报错相信请参考http://blog.csdn.net/shiyu1157758655/article/details/70242249  
  
  
解决之后再次执行  
\[root@manager ~\]\# masterha\_check\_repl --conf /etc/masterha/app1.cnf   
Wed Apr 19 16:22:37 2017 - \[warning\] Global configuration file /etc/masterha\_default.cnf not found. Skipping.  
Wed Apr 19 16:22:37 2017 - \[info\] Reading application default configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:22:37 2017 - \[info\] Reading server configurations from /etc/masterha/app1.cnf..  
Wed Apr 19 16:22:37 2017 - \[info\] MHA::MasterMonitor version 0.55.  
Wed Apr 19 16:22:37 2017 - \[info\] Dead Servers:  
Wed Apr 19 16:22:37 2017 - \[info\] Alive Servers:  
Wed Apr 19 16:22:37 2017 - \[info\]   192.168.180.56(192.168.180.56:3306)  
Wed Apr 19 16:22:37 2017 - \[info\]   192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 16:22:37 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  
Wed Apr 19 16:22:37 2017 - \[info\] Alive Slaves:  
Wed Apr 19 16:22:37 2017 - \[info\]   192.168.180.56(192.168.180.56:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 16:22:37 2017 - \[info\]     Replicating from 192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 16:22:37 2017 - \[info\]     Primary candidate for the new Master (candidate\_master is set)  
Wed Apr 19 16:22:37 2017 - \[info\]   192.168.180.58(192.168.180.58:3306)  Version=5.7.15-log (oldest major version between slaves) log-bin:enabled  
Wed Apr 19 16:22:37 2017 - \[info\]     Replicating from 192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 16:22:37 2017 - \[info\]     Not candidate for the new Master (no\_master is set)  
Wed Apr 19 16:22:37 2017 - \[info\] Current Alive Master: 192.168.180.57(192.168.180.57:3306)  
Wed Apr 19 16:22:37 2017 - \[info\] Checking slave configurations..  
Wed Apr 19 16:22:37 2017 - \[info\]  read\_only=1 is not set on slave 192.168.180.56(192.168.180.56:3306).  
Wed Apr 19 16:22:37 2017 - \[warning\]  relay\_log\_purge=0 is not set on slave 192.168.180.56(192.168.180.56:3306).  
Wed Apr 19 16:22:37 2017 - \[info\] Checking replication filtering settings..  
Wed Apr 19 16:22:37 2017 - \[info\]  binlog\_do\_db= , binlog\_ignore\_db=   
Wed Apr 19 16:22:37 2017 - \[info\]  Replication filtering check ok.  
Wed Apr 19 16:22:37 2017 - \[info\] Starting SSH connection tests..  
Wed Apr 19 16:22:38 2017 - \[info\] All SSH connection tests passed successfully.  
Wed Apr 19 16:22:38 2017 - \[info\] Checking MHA Node version..  
Wed Apr 19 16:22:39 2017 - \[info\]  Version check ok.  
Wed Apr 19 16:22:39 2017 - \[info\] Checking SSH publickey authentication settings on the current master..  
Wed Apr 19 16:22:39 2017 - \[info\] HealthCheck: SSH to 192.168.180.57 is reachable.  
Wed Apr 19 16:22:39 2017 - \[info\] Master MHA Node version is 0.54.  
Wed Apr 19 16:22:39 2017 - \[info\] Checking recovery script configurations on the current master..  
Wed Apr 19 16:22:39 2017 - \[info\]   Executing command: save\_binary\_logs --command=test --start\_pos=4 --binlog\_dir=/data/mysql --output\_file=/data/mha/save\_binary\_logs\_test --manager\_version=0.55 --start\_file=mysql-bin.000002   
Wed Apr 19 16:22:39 2017 - \[info\]   Connecting to root@192.168.180.57(192.168.180.57)..   
  Creating /data/mha if not exists..    ok.  
  Checking output directory is accessible or not..  
   ok.  
  Binlog found at /data/mysql, up to mysql-bin.000002  
Wed Apr 19 16:22:39 2017 - \[info\] Master setting check done.  
Wed Apr 19 16:22:39 2017 - \[info\] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..  
Wed Apr 19 16:22:39 2017 - \[info\]   Executing command : apply\_diff\_relay\_logs --command=test --slave\_user='mha' --slave\_host=192.168.180.56 --slave\_ip=192.168.180.56 --slave\_port=3306 --workdir=/data/mha --target\_version=5.7.15-log --manager\_version=0.55 --relay\_log\_info=/data/mysql/relay-log.info  --relay\_dir=/data/mysql/  --slave\_pass=xxx  
Wed Apr 19 16:22:39 2017 - \[info\]   Connecting to root@192.168.180.56(192.168.180.56:22)..   
  Checking slave recovery environment settings..  
    Opening /data/mysql/relay-log.info ... ok.  
    Relay log found at /data/mysql, up to master-relay-bin.000003  
    Temporary relay log file is /data/mysql/master-relay-bin.000003  
    Testing mysql connection and privileges..mysql: \[Warning\] Using a password on the command line interface can be insecure.  
 done.  
    Testing mysqlbinlog output.. done.  
    Cleaning up test file(s).. done.  
Wed Apr 19 16:22:40 2017 - \[info\]   Executing command : apply\_diff\_relay\_logs --command=test --slave\_user='mha' --slave\_host=192.168.180.58 --slave\_ip=192.168.180.58 --slave\_port=3306 --workdir=/data/mha --target\_version=5.7.15-log --manager\_version=0.55 --relay\_log\_info=/data/mysql/relay-log.info  --relay\_dir=/data/mysql/  --slave\_pass=xxx  
Wed Apr 19 16:22:40 2017 - \[info\]   Connecting to root@192.168.180.58(192.168.180.58:22)..   
  Checking slave recovery environment settings..  
    Opening /data/mysql/relay-log.info ... ok.  
    Relay log found at /data/mysql, up to mysql-relay-bin.000002  
    Temporary relay log file is /data/mysql/mysql-relay-bin.000002  
    Testing mysql connection and privileges..mysql: \[Warning\] Using a password on the command line interface can be insecure.  
 done.  
    Testing mysqlbinlog output.. done.  
    Cleaning up test file(s).. done.  
Wed Apr 19 16:22:40 2017 - \[info\] Slaves settings check done.  
Wed Apr 19 16:22:40 2017 - \[info\]   
192.168.180.57 (current master)  
 +--192.168.180.56  
 +--192.168.180.58  
  
  
Wed Apr 19 16:22:40 2017 - \[info\] Checking replication health on 192.168.180.56..  
Wed Apr 19 16:22:40 2017 - \[info\]  ok.  
Wed Apr 19 16:22:40 2017 - \[info\] Checking replication health on 192.168.180.58..  
Wed Apr 19 16:22:40 2017 - \[info\]  ok.  
Wed Apr 19 16:22:40 2017 - \[warning\] master\_ip\_failover\_script is not defined.  
Wed Apr 19 16:22:40 2017 - \[warning\] shutdown\_script is not defined.  
Wed Apr 19 16:22:40 2017 - \[info\] Got exit code 0 (Not master dead).  
  
  
MySQL Replication Health is OK.  
  
  
  
  
  
  
至此Mysql MHA 搭建以及故障切换已经完成