Keepalived实现NAT+LVS+后端健康检测

Love The Way You Lie 2022-05-19 12:48 499阅读 0赞

对于keepalived和lvs，网上多数的文章都是介绍DR（Direct Route）模式的负载均衡策略。DR模式确实有其优点，尤其是对于负载均衡器压力更小。但是NAT模式也不是一无是处，甚至keepalived的官网首推的就是NAT模式，默认的配置文件也是NAT模式。NAT模式是将负载均衡服务器当做路由器来用，装keepalived的服务器至少要有两个网卡，且要分别处于不同的网段。NAT的优点是仅需配置装有keepalived的负载均衡服务器，而后端的real server是0配置；而DR模式下不仅需要配置keepalived的服务器，还需要需要修改后端服务器的arp策略并在lo网卡上配置虚拟IP。而开发工程师可能不喜欢我们在他的real server上设置网络，到时候出了bug就不好区分是谁的锅了。所以当我们的real server比较少，而且流量也不是特别高的情况下，使用NAT模式是一个很好的选择。

但是NAT模式也有缺点，一是当real server很多的时候会成为瓶颈，不过这一点不容易碰到，因为有这么大流量的场合基本都换成硬件负载均衡器了。第二就是要求有两个不同网段的网络或者VLAN，因为NAT的缘故，所以如果我们手上只有一个网段的话，是无法部署成NAT的。

下面以centos7为例介绍如何安装部署keepalived和NAT模式。

首先我们把centos7以最小化安装，然后关闭防火墙并打开ipv4的路由功能，centOS6也类似，只是关闭防火墙的语句不一样

【centos6关闭防火墙】
    service iptables stop
    chkconfig iptables off
    
    【centos7关闭防火墙】
    systemctl status firewalld.service
    systemctl stop firewalld.service
    systemctl disable firewalld.service
    
    vi /etc/sysctl.conf
    net.ipv4.ip_forward=1
    sysctl -p

然后到官网上下载源码包进行编译，下载地址：[http://www.keepalived.org/download.html][http_www.keepalived.org_download.html]

我是下载当前的最新版2.0.6版

为什么不用yum直接安装呢，centos的安装光盘里明明就有keepalived的安装包，在centos1804的安装盘里，版本是1.3.5

root@lvsnat2 ~]# yum list | grep keepalived
    keepalived.x86_64                        1.3.5-1.el7                   @base

这个版本安装上有一个极大的坑，就是不能进行TCP和HTTP的后端检测，在正确的配置文件下，一开keepalived进程就会在/var/log/message中报错，如下

Jul 26 00:28:51 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:28:58 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:06 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:13 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:21 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:28 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:36 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:43 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:51 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:29:58 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:06 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:13 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:21 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:28 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:36 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:43 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:51 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:30:58 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:06 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:13 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:21 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:28 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:36 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:43 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:51 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:31:58 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:32:06 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:32:13 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:32:21 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling
    Jul 26 00:32:28 lvsnat1 Keepalived_healthcheckers[11147]: WEB socket bind failed. Rescheduling

这个问题研究好久，最后感觉可能是程序的问题，然后通过手动编译最新版keepalived后发现该问题解决，是在是centos的一个大坑，centos在出厂前居然没有仔细的测试过。所以我们还是乖乖的编译吧。

编译之前要安装一些依赖包

yum install curl gcc openssl-devel libnl3-devel net-snmp-devel

然后解压源码包

./configure
    make && make install

这样就编译完成了，并且自动加入了系统服务并关联了环境变量

然后我们需要在指定位置下建立配置文件，文件是 /etc/keepalived/keepalived.conf

首先介绍一下我的网络情况和主机情况

<table> 
 <tbody> 
  <tr> 
   <td style="width:338px;">服务器</td> 
   <td style="width:168px;">对外地址</td> 
   <td style="width:256px;">内部地址</td> 
   <td style="width:256px;">虚拟外部地址</td> 
   <td style="width:255px;">虚拟内部地址</td> 
  </tr> 
  <tr> 
   <td style="width:338px;">Keepalived主服务器</td> 
   <td style="width:168px;">192.168.162.158</td> 
   <td style="width:256px;">10.100.100.99</td> 
   <td style="width:256px;">192.168.162.180</td> 
   <td style="width:255px;">10.100.100.100</td> 
  </tr> 
  <tr> 
   <td style="width:338px;">Keepalived备服务器</td> 
   <td style="width:168px;">192.168.162.159</td> 
   <td style="width:256px;">10.100.100.98</td> 
   <td style="width:256px;">同主机</td> 
   <td style="width:255px;">同主机</td> 
  </tr> 
  <tr> 
   <td style="width:338px;">真实HTTP服务1</td> 
   <td style="width:168px;">&nbsp;</td> 
   <td style="width:256px;">10.100.100.101</td> 
   <td style="width:256px;">&nbsp;</td> 
   <td style="width:255px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:338px;">真实HTTP服务2</td> 
   <td style="width:168px;">&nbsp;</td> 
   <td style="width:256px;">10.100.100.102</td> 
   <td style="width:256px;">&nbsp;</td> 
   <td style="width:255px;">&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

其中，keepalived的内部地址可以不设置，只需要有虚拟内部地址即可

在启动keepalived之前，我们还需要将真实服务器的网关设置为keepalived服务器的虚拟内部地址10.100.100.100，另外关闭真实服务器的防火墙（或者开通相关网络策略），方法同上。

下面一次展示各个服务器的网络配置情况

keepalived主服务器

[root@lvsnat1 keepalived-2.0.6]# ifconfig
    ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.162.159  netmask 255.255.255.0  broadcast 192.168.162.255
            inet6 fe80::f1c5:8549:4523:212c  prefixlen 64  scopeid 0x20<link>
            inet6 fe80::7602:aaf3:d259:a375  prefixlen 64  scopeid 0x20<link>
            ether 00:50:56:be:da:eb  txqueuelen 1000  (Ethernet)
            RX packets 164424  bytes 107613403 (102.6 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 19033  bytes 2223408 (2.1 MiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 10.100.100.98  netmask 255.255.255.0  broadcast 10.100.100.255
            inet6 fe80::3888:f507:bebe:5fc6  prefixlen 64  scopeid 0x20<link>
            inet6 fe80::bbb7:585b:a09b:9884  prefixlen 64  scopeid 0x20<link>
            ether 00:50:56:be:dd:aa  txqueuelen 1000  (Ethernet)
            RX packets 1593  bytes 192661 (188.1 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 6464  bytes 420610 (410.7 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1000  (Local Loopback)
            RX packets 52  bytes 4320 (4.2 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 52  bytes 4320 (4.2 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

keepalived备服务器

[root@lvsnat2 ~]# ifconfig
    ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.162.158  netmask 255.255.255.0  broadcast 192.168.162.255
            inet6 fe80::7602:aaf3:d259:a375  prefixlen 64  scopeid 0x20<link>
            ether 00:50:56:be:4e:5b  txqueuelen 1000  (Ethernet)
            RX packets 102080  bytes 9186263 (8.7 MiB)
            RX errors 0  dropped 20  overruns 0  frame 0
            TX packets 8383  bytes 841991 (822.2 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 10.100.100.99  netmask 255.255.255.0  broadcast 10.100.100.255
            inet6 fe80::bbb7:585b:a09b:9884  prefixlen 64  scopeid 0x20<link>
            ether 00:50:56:be:a6:e8  txqueuelen 1000  (Ethernet)
            RX packets 2310  bytes 153133 (149.5 KiB)
            RX errors 0  dropped 20  overruns 0  frame 0
            TX packets 2650  bytes 157579 (153.8 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1000  (Local Loopback)
            RX packets 43  bytes 3624 (3.5 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 43  bytes 3624 (3.5 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

真实HTTP服务器1

eth0      Link encap:Ethernet  HWaddr 00:50:56:BE:FD:EA  
              inet addr:10.100.100.101  Bcast:10.100.100.255  Mask:255.255.255.0
              inet6 addr: fe80::250:56ff:febe:fdea/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:1349 errors:0 dropped:0 overruns:0 frame:0
              TX packets:922 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:113090 (110.4 KiB)  TX bytes:122113 (119.2 KiB)
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:207 errors:0 dropped:0 overruns:0 frame:0
              TX packets:207 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:15432 (15.0 KiB)  TX bytes:15432 (15.0 KiB)

注意要将网关设置为keepalived的虚拟地址，如下

cat /etc/sysconfig/network-scripts/ifcfg-eth0 
    DEVICE=eth0
    TYPE=Ethernet
    UUID=5fad7be7-5f2e-46e5-98ab-4a28e23538af
    ONBOOT=yes
    NM_CONTROLLED=yes
    BOOTPROTO=none
    HWADDR=00:50:56:BE:FD:EA
    IPADDR=10.100.100.101
    PREFIX=24
    GATEWAY=10.100.100.100
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=yes
    IPV6INIT=no
    NAME="System eth0"

真实HTTP服务器2

eth1      Link encap:Ethernet  HWaddr 00:50:56:BE:42:2F  
              inet addr:10.100.100.102  Bcast:10.100.100.255  Mask:255.255.255.0
              inet6 addr: fe80::250:56ff:febe:422f/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:1450 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1082 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:139305 (136.0 KiB)  TX bytes:117017 (114.2 KiB)
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:278 errors:0 dropped:0 overruns:0 frame:0
              TX packets:278 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0 
              RX bytes:20124 (19.6 KiB)  TX bytes:20124 (19.6 KiB)

cat /etc/sysconfig/network-scripts/ifcfg-Auto_eth1 
    HWADDR=00:50:56:BE:42:2F
    TYPE=Ethernet
    BOOTPROTO=none
    IPADDR=10.100.100.102
    PREFIX=24
    GATEWAY=10.100.100.100
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=yes
    IPV6INIT=no
    NAME="Auto eth1"
    UUID=afe05e25-7775-4a39-bacb-e81a9bfc96c2
    ONBOOT=yes
    LAST_CONNECT=1532527205

如此一来，就可以看懂下面的配置文件了

keepalived主服务配置文件

! Configuration File for keepalived
    
    global_defs {
       router_id LVS_MASTER
       vrrp_skip_check_adv_addr
       #vrrp_strict               #不要开启此项，不然NAT会失败，并ping不通虚拟IP
       vrrp_garp_interval 0
       vrrp_gna_interval 0
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface ens192          #此处设置对外网卡
        virtual_router_id 81      #如果局域网里有其他keepalived集群，不要和他们冲突
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.162.180       #此处设置对外的虚拟IP，即用户最终要访问的IP
        }
    }
    
    vrrp_instance LAN_GATEWAY {
        state MASTER                   
        interface ens224          #此处填写对内网卡的名称，也就是与真实服务器同网段的网卡
        virtual_router_id 82      #此处的ID要与上面的不同
        priority 100                  
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            10.100.100.100        #此处的虚拟IP为对内的虚拟IP，也就是真实服务器网关的IP
        }
    }
    
    virtual_server 192.168.162.180 80 {
        delay_loop 15
        lb_algo rr
        lb_kind NAT               #此处类型要填写NAT，不要写DR
        persistence_timeout 50
        protocol TCP              #此处可以填TCP或UDP
    
        real_server 10.100.100.101 8080 {  #真实WEB服务的端口
            weight 1
            HTTP_GET {
                url {
                  path /hello     #用于检测可用性的子页面
                  digest d7f986677d9f563bd1794b09d82206a3   #子页面内容的MD5
    			  status_code 200 #判断返回状态码
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    	
    	real_server 10.100.100.102 8080 {  #第二个web服务器的地址和端口
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest 3dc49f3f365e728446c8f8df2475464d
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    }

keepalived备服务器的配置文件

! Configuration File for keepalived
    
    global_defs {
       router_id LVS_BACKUP            #BACKUP上修改为LVS_BACKUP
    }
    
    vrrp_instance VI_1 {
        state BACKUP                   #BACKUP上修改为BACKUP
        interface ens192
        virtual_router_id 81
        priority 80                    #BACKUP上修改为80
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.162.180
        }
    }
    
    vrrp_instance LAN_GATEWAY {
        state BACKUP                   #BACKUP上修改为LVS_BACKUP
        interface ens224
        virtual_router_id 82
        priority 80                    #BACKUP上修改为80
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            10.100.100.100
        }
    }
    
    virtual_server 192.168.162.180 80 {
        delay_loop 15
        lb_algo rr
        lb_kind NAT
        persistence_timeout 50
        protocol TCP
    
        real_server 10.100.100.101 8080 {
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest d7f986677d9f563bd1794b09d82206a3
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    	
    	real_server 10.100.100.102 8080 {
            weight 1
            HTTP_GET {
                url {
                  path /hello
                  digest 3dc49f3f365e728446c8f8df2475464d
    			  status_code 200
                }
                connect_timeout 3
                retry 3
                delay_before_retry 3
            }
        }
    }

上面的digest是页面的MD5，可以使用genhash命令来获得，如果使用yum安装keepalived就会自动装这个，如果没有的话可以单独安装

# genhash -s 10.100.100.101 -p 8080 -u /hello
    MD5SUM = 62a184e396b02ccd7a5d6d0817b0fc44

\-s后面的参数为真实服务器的ip地址，-p为服务端口 ， -u后面为url路径

启动keepalived

service keepalived start

查看keepalived的启动日志

tail -f /var/log/messages
    
    
    Jul 26 00:01:17 lvsnat1 systemd: Starting LVS and VRRP High Availability Monitor...
    Jul 26 00:01:17 lvsnat1 Keepalived[20592]: Starting Keepalived v2.0.6 (07/23,2018)
    Jul 26 00:01:17 lvsnat1 Keepalived[20592]: Running on Linux 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 (built for Linux 3.10.0)
    Jul 26 00:01:17 lvsnat1 Keepalived[20592]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 26 00:01:17 lvsnat1 Keepalived[20593]: Starting Healthcheck child process, pid=20594
    Jul 26 00:01:17 lvsnat1 systemd: Started LVS and VRRP High Availability Monitor.
    Jul 26 00:01:17 lvsnat1 Keepalived[20593]: Starting VRRP child process, pid=20595
    Jul 26 00:01:17 lvsnat1 Keepalived_healthcheckers[20594]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 26 00:01:17 lvsnat1 Keepalived_healthcheckers[20594]: Gained quorum 1+0=1 <= 2 for VS [192.168.162.180]:tcp:80
    Jul 26 00:01:17 lvsnat1 Keepalived_healthcheckers[20594]: Activating healthchecker for service [10.100.100.101]:tcp:8080 for VS [192.168.162.180]:tcp:80
    Jul 26 00:01:17 lvsnat1 Keepalived_healthcheckers[20594]: Activating healthchecker for service [10.100.100.102]:tcp:8080 for VS [192.168.162.180]:tcp:80
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Registering Kernel netlink reflector
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Registering Kernel netlink command channel
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Assigned address 192.168.162.159 for interface ens192
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Assigned address fe80::f1c5:8549:4523:212c for interface ens192
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Assigned address 10.100.100.98 for interface ens224
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Assigned address fe80::3888:f507:bebe:5fc6 for interface ens224
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: Registering gratuitous ARP shared channel
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: (VI_1) removing VIPs.
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) removing VIPs.
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: (VI_1) Entering BACKUP STATE (init)
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) Entering BACKUP STATE (init)
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(8,9)]
    Jul 26 00:01:17 lvsnat1 Keepalived_vrrp[20595]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(10,11)]
    Jul 26 00:01:18 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) received lower priority (80) advert from 10.100.100.99 - discarding
    Jul 26 00:01:18 lvsnat1 Keepalived_vrrp[20595]: (VI_1) received lower priority (80) advert from 192.168.162.158 - discarding
    Jul 26 00:01:19 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) received lower priority (80) advert from 10.100.100.99 - discarding
    Jul 26 00:01:19 lvsnat1 Keepalived_vrrp[20595]: (VI_1) received lower priority (80) advert from 192.168.162.158 - discarding
    Jul 26 00:01:20 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) received lower priority (80) advert from 10.100.100.99 - discarding
    Jul 26 00:01:20 lvsnat1 Keepalived_vrrp[20595]: (VI_1) received lower priority (80) advert from 192.168.162.158 - discarding
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) Receive advertisement timeout
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) Entering MASTER STATE
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) setting VIPs.
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) Sending/queueing gratuitous ARPs on ens224 for 10.100.100.100
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (VI_1) Receive advertisement timeout
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (VI_1) Entering MASTER STATE
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (VI_1) setting VIPs.
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: (VI_1) Sending/queueing gratuitous ARPs on ens192 for 192.168.162.180
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:21 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: (LAN_GATEWAY) Sending/queueing gratuitous ARPs on ens224 for 10.100.100.100
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: (VI_1) Sending/queueing gratuitous ARPs on ens192 for 192.168.162.180
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 26 00:01:26 lvsnat1 Keepalived_vrrp[20595]: Sending gratuitous ARP on ens192 for 192.168.162.180

从上面的日志我们可以看到 Entering Master State后就进入了Master状态，并且会立即发送ARP广播，通告自己的虚拟IP

来看一下keepalived备机的启动日志

Jul 25 23:56:34 lvsnat1 systemd: Starting LVS and VRRP High Availability Monitor...
    Jul 25 23:56:35 lvsnat1 Keepalived[11054]: Starting Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2
    Jul 25 23:56:35 lvsnat1 Keepalived[11054]: Unable to resolve default script username 'keepalived_script' - ignoring
    Jul 25 23:56:35 lvsnat1 Keepalived[11054]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 25 23:56:35 lvsnat1 systemd: PID file /var/run/keepalived.pid not readable (yet?) after start.
    Jul 25 23:56:35 lvsnat1 Keepalived[11055]: Starting Healthcheck child process, pid=11056
    Jul 25 23:56:35 lvsnat1 Keepalived[11055]: Starting VRRP child process, pid=11057
    Jul 25 23:56:35 lvsnat1 systemd: Started LVS and VRRP High Availability Monitor.
    Jul 25 23:56:35 lvsnat1 Keepalived_healthcheckers[11056]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 25 23:56:35 lvsnat1 Keepalived_healthcheckers[11056]: Activating healthchecker for service [192.168.162.180]:80
    Jul 25 23:56:35 lvsnat1 Keepalived_healthcheckers[11056]: Activating healthchecker for service [192.168.162.180]:80
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: Registering Kernel netlink reflector
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: Registering Kernel netlink command channel
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: Registering gratuitous ARP shared channel
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: Opening file '/etc/keepalived/keepalived.conf'.
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) removing protocol VIPs.
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: Using LinkWatch kernel netlink reflector...
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Entering BACKUP STATE
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
    Jul 25 23:56:35 lvsnat1 Keepalived_vrrp[11057]: VRRP sockpool: [ifindex(3), proto(112), unicast(0), fd(12,13)]

可以看到，备机启动后只会做健康检查，不会通告自己的状态

下面来看一下模拟一台真实web服务器挂掉，看keepalived主机会有什么变化，方法就是修改上面健康监测页面的内容，让它的MD5不一样

Jul 26 00:03:01 lvsnat1 Keepalived_healthcheckers[20594]: MD5 digest error to server [10.100.100.101]:tcp:8080.
    Jul 26 00:03:04 lvsnat1 Keepalived_healthcheckers[20594]: MD5 digest error to server [10.100.100.101]:tcp:8080.
    Jul 26 00:03:07 lvsnat1 Keepalived_healthcheckers[20594]: MD5 digest error to server [10.100.100.101]:tcp:8080.
    Jul 26 00:03:10 lvsnat1 Keepalived_healthcheckers[20594]: MD5 digest error to server [10.100.100.101]:tcp:8080.
    Jul 26 00:03:10 lvsnat1 Keepalived_healthcheckers[20594]: Check on service [10.100.100.101]:tcp:8080 failed.
    Jul 26 00:03:10 lvsnat1 Keepalived_healthcheckers[20594]: Removing service [10.100.100.101]:tcp:8080 to VS [192.168.162.180]:tcp:80

可以看到，keepalived侦测到之后立即把这个服务器拿出了负载均衡列表，客户也就访问不到了

下面再看看把这个页面改回原来的，keepalived主机能不能再把这台真实服务器加回到列表中

Jul 26 00:04:25 lvsnat1 Keepalived_healthcheckers[20594]: MD5 digest success to [10.100.100.101]:tcp:8080 url(1).
    Jul 26 00:04:25 lvsnat1 Keepalived_healthcheckers[20594]: Remote Web server [10.100.100.101]:tcp:8080 succeed on service.
    Jul 26 00:04:25 lvsnat1 Keepalived_healthcheckers[20594]: Adding service [10.100.100.101]:tcp:8080 to VS [192.168.162.180]:tcp:80

此时访问真实IP发现已经又可以看到被拿掉的服务器了

注意keepalived默认是有回话保持措施的，虽然两个真实服务器的weight都是1，但是你在不保留缓存连续刷新的时候，是只能定位到一台服务器的，所以做测试的时候要多拿几个客户端IP去测

下面看一下模拟keepalived主机宕机，备机能否接管虚拟IP并正常提供服务

使用service keepalived stop来停止主机或者直接关机，观察备机上的日志

Jul 25 23:57:55 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Jul 25 23:57:55 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Transition to MASTER STATE
    Jul 25 23:57:56 lvsnat1 Keepalived_healthcheckers[11056]: WEB socket bind failed. Rescheduling
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Entering MASTER STATE
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) setting protocol VIPs.
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens192 for 192.168.162.180
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Entering MASTER STATE
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) setting protocol VIPs.
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Sending/queueing gratuitous ARPs on ens224 for 10.100.100.100
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:57:56 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:57:59 lvsnat1 Keepalived_healthcheckers[11056]: WEB socket bind failed. Rescheduling
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens192 for 192.168.162.180
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens192 for 192.168.162.180
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Sending/queueing gratuitous ARPs on ens224 for 10.100.100.100
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100
    Jul 25 23:58:01 lvsnat1 Keepalived_vrrp[11057]: Sending gratuitous ARP on ens224 for 10.100.100.100

可以看到备机马上就进入了master状态并且ARP广播了自己的MAC地址，此时从客户端方位对外ip 192.168.162.180发现服务正常。

那么我们再把主服务器启动，看看备服务器能不能切换回去

Jul 25 23:58:27 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 80
    Jul 25 23:58:27 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Jul 25 23:58:27 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(VI_1) removing protocol VIPs.
    Jul 25 23:58:27 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Received advert with higher priority 100, ours 80
    Jul 25 23:58:27 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) Entering BACKUP STATE
    Jul 25 23:58:27 lvsnat1 Keepalived_vrrp[11057]: VRRP_Instance(LAN_GATEWAY) removing protocol VIPs.

可以看到，备服务器在网络中发现比自己优先级高的节点后，会自动的进入backup状态，并且把VIP释放

[http_www.keepalived.org_download.html]: http://www.keepalived.org/download.html