keepalived实现Nginx双机热备

爱被打了一巴掌 2023-02-15 03:56 15阅读 0赞

### 前言 ###

首先介绍一下Keepalived，它是一个高性能的服务器高可用或热备解决方案，起初是专为LVS负载均衡软件设计的，Keepalived主要来防止服务器单点故障的发生问题，可以通过其与Nginx的配合实现web服务端的高可用。

Keepalived以VRRP协议为实现基础，VRRP是Virtual Router Redundancy Protocol（虚拟路由冗余协议）的缩写，VRRP协议将两台或多台路由器设备虚拟成一个设备，对外提供虚拟路由器IP（一个或多个）。

VRRP出现的目的就是为了解决静态路由的单点故障问题的，它能保证当个别节点宕机时，整个网络可以不间断地运行。

下面我们介绍一下nginx keepalived高可用方案的部署安装。

### 环境准备 ###

在两台主机上准备如下压缩文件：

*  keepalived-2.0.20.tar.gz
 *  nginx-1.16.1.tar.gz

<table> 
 <thead> 
  <tr> 
   <th>虚拟IP</th> 
   <th>真实IP</th> 
   <th>nginx端口</th> 
   <th>主从</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>192.168.124.20</td> 
   <td>192.168.124.13</td> 
   <td>80</td> 
   <td>MASTER</td> 
  </tr> 
  <tr> 
   <td>192.168.124.20</td> 
   <td>192.168.124.14</td> 
   <td>80</td> 
   <td>BACKUP</td> 
  </tr> 
 </tbody> 
</table>

### 安装nginx ###

![format_png][]新建一个用户：

useradd tianyan

确定安装目录，我这里的安装目录是：/home/tianyan/tianyan\_soft/nginx.install。

在这个目录下分别新建两个目录用于安装nginx和keepalived，解压两个压缩包。

执行安装命令：

./configure --prefix=/home/tianyan/tianyan_soft/nginx.install \
    --sbin-path=/home/tianyan/tianyan_soft/nginx.install/sbin/nginx --conf-path=/home/tianyan/tianyan_soft/nginx.install/conf/nginx.conf \
    --error-log-path=/home/tianyan/tianyan_soft/nginx.install/error.log \
    --http-log-path=/home/tianyan/tianyan_soft/nginx.install/access.log \
    --pid-path=/home/tianyan/tianyan_soft/nginx.install/nginx.pid \
    --lock-path=/home/tianyan/tianyan_soft/nginx.install/nginx.lock \
    --user=tianyan --group=tianyan \
    --with-http_stub_status_module \
    --with-http_gzip_static_module \
    --with-http_ssl_module \
    --with-http_realip_module \
    --with-threads  \
    --with-pcre \
    --http-client-body-temp-path=/home/tianyan/tianyan_soft/nginx.install/client/  \
    --http-proxy-temp-path=/home/tianyan/tianyan_soft/nginx.install/proxy/ \
     --http-fastcgi-temp-path=/home/tianyan/tianyan_soft/nginx.install/fcgi/  \
    --http-uwsgi-temp-path=/home/tianyan/tianyan_soft/nginx.install/uwsgi  \
    --http-scgi-temp-path=/home/tianyan/tianyan_soft/nginx.install/scgi

如果报错，记得安装相关依赖：

yum install gcc gcc-c++

nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)。

注：以非root权限启动时，会出现 nginx: \[emerg\] bind() to 0.0.0.0:80 failed (13: Permission denied) 错误。

原因：Linux只有root用户可以使用1024以下的端口

解决办法：

*  1.按照root权限启动
 *  2.将 /usr/local/nginx/conf/nginx.conf 文件中的80端口改为1024以上。

### 安装keepalived ###

./configure --prefix=/usr/local/keepalived

上述命令执行完毕后继续执行：

make && make install

安装完毕后，目录是这个样子：

![format_png 1][]

将配置文件拷贝到系统对应的目录下

mkdir /etc/keepalived
    cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
    cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived

#### 编辑 master 节点的 keepalived.conf ####

vim /etc/keepalived/keepalived.conf

内容参考如下：

! Configuration File for keepalived
    
    global_defs {
        #一个没重复的名字即可
        router_id hyq_slave
       }
    
    
    #ng是否运行
    vrrp_script chk_nginx {
            script "/etc/keepalived/nginx_check.sh"
            interval 2
            weight -20
    }
    
    vrrp_instance VI_1 {
        state BACKUP      #  必填，可以是MASTER或BACKUP
    
        interface ens33
        virtual_router_id 101
        priority 90
        advert_int 1
    
        # 如果两节点的上联交换机禁用了组播，则采用vrrp单播通告的方式
        # 本机ip
        unicast_src_ip 192.168.124.14
        unicast_peer {
            # 其他机器ip
            192.168.124.13
        }
        # 设置nopreempt防止抢占资源
        nopreempt
    
        authentication {
            auth_type PASS
            auth_pass 1111
        }
    
        # 与上方nginx运行状况检测呼应
        track_script {
            chk_nginx
        }
        virtual_ipaddress {
            192.168.124.20
        }
    }

#### 编辑 slave 节点的 keepalived.conf ####

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
    
    global_defs {
        #一个没重复的名字即可
        router_id hyq_slave
       }
    
    
    #ng是否运行
    vrrp_script chk_nginx {
            script "/etc/keepalived/nginx_check.sh"
            interval 2
            weight -20
    }
    
    vrrp_instance VI_1 {
        state BACKUP      #  必填，可以是MASTER或BACKUP
    
    
        interface ens33
        virtual_router_id 101
        priority 90
        advert_int 1
    
        # 如果两节点的上联交换机禁用了组播，则采用vrrp单播通告的方式
        # 本机ip
        unicast_src_ip 192.168.124.14
        unicast_peer {
            # 其他机器ip
            192.168.124.13
        }
        # 设置nopreempt防止抢占资源
        nopreempt
    
        authentication {
            auth_type PASS
            auth_pass 1111
        }
    
        # 与上方nginx运行状况检测呼应
        track_script {
            chk_nginx
        }
        virtual_ipaddress {
            192.168.124.20
        }
    }

#### 编写nginx\_check.sh脚本 ####

在/etc/keepalived目录下新建nginx\_check.sh脚本

touch nginx_check.sh

编辑其内容为：

#!/bin/sh
    A=`ps -C nginx --no-header |wc -l`
    if [ $A -eq 0 ]
    then
      /usr/sbin/nginx
      sleep 1
      A2=`ps -C nginx --no-header |wc -l`
      if [ $A2 -eq 0 ]
      then
        systemctl stop keepalived
      fi
    fi

含义是：如果 nginx 停止运行，尝试启动，但是如果无法启动，则杀死本机的 keepalived 进程， keepalied将会把虚拟 ip 绑定到 BACKUP 机器上。注意：/usr/sbin/nginx是nginx的启动命令，如果你安装到其他目录，则相应的替换。

### Keepalived的日志 ###

Keepalived日志默认位置是在/var/log/messages目录下。我们将其修改一下。

由于系统是centos7，修改位置为：/lib/systemd/system/keepalived.service

原内容：

EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived
    ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS

修改为：

![format_png 2][]

修改完毕后重新加载service

systemctl daemon-reload

创建命令软连接：

ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived

执行：

keepalived -D -f /etc/keepalived/keepalived.conf
    -D   将日志输出到message日志,默认日志也在message
    -f    是指定配置文件

修改/etc/sysconfig/keepalived

把KEEPALIVED\_OPTIONS="-D" 修改为：KEEPALIVED\_OPTIONS="-D -d -S 0"

![format_png 3][]

在/etc/rsyslog.conf 末尾添加

local0.*                                                /var/log/keepalived.log

![format_png 4][]

最后执行命令：

service rsyslog restart

重启keepalived后就可以看到日志在/var/log/keepalived.log下了。

### 测试验证VIP ###

当keepalived和nginx都启动后，我们来测试一下。要模拟出转移的效果，记得把ng检查脚本中，启动nginx的命令先注释一下。

首先在浏览器里面访问三个地址

*  http://192.168.124.20 （vip）
 *  http://192.168.124.13 （master）
 *  http://192.168.124.14 (slave)

我修改了一下nginx的index.html,可以看到当前vip指向是13的master节点：

![format_png 5][]

然后，我们手动停止13上的nginx，再次访问http://192.168.124.20。

![format_png 6][]

说明安装成功。

其中通过ip address命令可以观察到网卡的变化![format_png 7][]

实验到这里，我们就完成了keepalived + nginx 主从配置的安装部署了。

### 思考：如何开启双主模式 ###

什么是双主模式？

分别介绍一下两种配置

1、Nginx+keepalived 主从配置

![format_png 8][]

这种方案就是上文介绍过的，使用一个vip地址，前端使用2台机器，一台做主，一台做备，但同时只有一台机器工作，另一台备份机器在主机器不出现故障的时候，永远处于浪费状态，仅仅用于灾备，平时都是空闲着的。

2、Nginx+keepalived 双主配置

这种方案，使用两个vip地址，前端使用2台机器，互为主备，同时有两台机器工作，当其中一台机器出现故障，两台机器的请求转移到一台机器负担，如下图：

![format_png 9][]

![format_png 10][]

[可视化界面在线生成JVM参数][JVM]

[java内存溢出问题分析过程二(附MAT超全操作文档)][java_MAT]

[使用Eclipse Memory Analyzer Tool（MAT）分析线上故障(一)][Eclipse Memory Analyzer Tool_MAT]

[一文让你理解什么是shallow heap及retained heap][shallow heap_retained heap]

[内存分析诊断系列-理解heap dump][-_heap_dump]

[记一次服务器被当肉鸡挖矿的经历][Link 1]

[如何编写一个可复用的SpringBoot应用运维脚本][SpringBoot]

[高效率编写Dockerfile需要绕过的一些坑][Dockerfile]

[Mysql百万量级数据高效导入Redis][Mysql_Redis]

[多线程之CountDownLatch的用法及原理笔记][CountDownLatch]

![format_png 11][]

我就知道你“在看”

![format_png 12][]

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4blhPdERaZ2liT0FGZlczQk94cGdDb21xekhWR2RHVDdqN3NyZzRQYzJLc1FOSGd3aWJSRGx0YkZ3LzY0MA?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bnh3RnRXdzFlVEtDOHZwWkpaMXJodlZaeVJ3aWJudHhEUzJTak94YXc4bXdQcnJzdXI4N2d5aWNRLzY0MA?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bkNWc0hUdndMRXgyTWQxbmliYkgxTlBzUVJLT251bDJLaWFPa2ljVWtsMTNpY1VISGVEVUlOQ3B0TlEvNjQw?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bjJIMW1NYmVLU25MalQzR3dTV0ZpYzFRc2NTZHZsaWJweGRTdmVWN0YzaWJXR1BaU0RuNk96R0xVZy82NDA?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4blFQWURwMGNpYldjdnJZT0pwS2JwQ1NBaDJ3aFJ5ZEZjbnYzSHltUG5pY0RuS0dZU29jdmRYemlhdy82NDA?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bnREdTRNdXM3aWFPYVNkUUh1VTU5YnNDeWJMeFhPOEJESE1tRUNoRFp4a2tNaWM2MlhNMXpwOTVBLzY0MA?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4blVLaWJWSkhxMW41RkNHcnU2Q1NodnQ0cTRjajdUMXN0WGx5VWtQTWNBeEFTVUJVb2h5OXdPb0EvNjQw?x-oss-process=image/format,png
[format_png 7]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bnZWelNqdjZNTU5uRjNHT0l0ZUt0RVQwZWV0dzVpYjE3UlpsTHA4cnZCSnRKb2NLcFYxRHVaTHcvNjQw?x-oss-process=image/format,png
[format_png 8]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4blhJNDBpYTJiUmlhNkVRVlo1OEFYOGliSVlpYndJTU0yaWNYNmlhc21XRWxpYkdSS1JDZ0UzaG9VVnRMcUEvNjQw?x-oss-process=image/format,png
[format_png 9]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bkJoUGtBSlN4b1F6Qmd6Q2ljc3BieUlQU3U5cm9ZVHlHTG1Vc29wVlJkYWJEcUw0RlV2Mmh5cmcvNjQw?x-oss-process=image/format,png
[format_png 10]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2dpZi94aWNXWVRTSUN6UnRDYWxzMWRGZFVjZ1lTWDVFMFhQN2liWklsc2FpYzNnMGg5TWtDT2VONkI5aWNuOHJCa3Z0TnBWbmxoc2hGQ1BxNHFXcjFxMnJYdk5HSmcvNjQw?x-oss-process=image/format,png
[JVM]: http://mp.weixin.qq.com/s?__biz=MzU2NDc4MjE2Ng%3D%3D&chksm=fc44f7e6cb337ef0bf114949eec1eb93e404f8ae64988a03cdc53a014119f496172a5de568f4&idx=1&mid=2247484127&scene=21&sn=92e774ee1edc2dd0013c60f1347d0fe6#wechat_redirect
[java_MAT]: https://blog.csdn.net/u012811805/article/details/104770774
[Eclipse Memory Analyzer Tool_MAT]: http://mp.weixin.qq.com/s?__biz=MzU2NDc4MjE2Ng%3D%3D&chksm=fc44f7e8cb337efe3c8a0d65fdcad6021c841cd5c00c47e20b09ff7fa31470a29865cda12a0a&idx=1&mid=2247484113&scene=21&sn=2f603fec4d105d832db0783369b9fe57#wechat_redirect
[shallow heap_retained heap]: https://blog.csdn.net/u012811805/article/details/104737895
[-_heap_dump]: http://mp.weixin.qq.com/s?__biz=MzU2NDc4MjE2Ng%3D%3D&chksm=fc44f794cb337e82fdcd6ad918d1a490d90908b6f453ddb45ad55b4a24a8b749e09baa9cc7a8&idx=1&mid=2247484077&scene=21&sn=5ae695faef094c78f2aa257b58b3034a#wechat_redirect
[Link 1]: http://mp.weixin.qq.com/s?__biz=MzU2NDc4MjE2Ng%3D%3D&chksm=fc44f7a7cb337eb1916c752918a881cd6cc033caa83657876ae3292e9e0639b7040ecfb37a9c&idx=1&mid=2247484062&scene=21&sn=d16d8ad10c5d4d26919276e7973708a5#wechat_redirect
[SpringBoot]: http://mp.weixin.qq.com/s?__biz=MzU2NDc4MjE2Ng%3D%3D&chksm=fc44f7adcb337ebbd0f51405a4502fc8e0ad0afdc562cbab581209e92b040645de7189a96f23&idx=1&mid=2247484052&scene=21&sn=e8e0520c74584721b366085834fd0489#wechat_redirect
[Dockerfile]: https://blog.csdn.net/u012811805/article/details/104638007
[Mysql_Redis]: http://mp.weixin.qq.com/s?__biz=MzU2NDc4MjE2Ng%3D%3D&chksm=fc44f743cb337e55a10570a29831dd060548a10e51988cbe27440c6c92d6b54589bf5accbc0d&idx=1&mid=2247484026&scene=21&sn=e537c94bb695d28f7a82f19f4b688899#wechat_redirect
[CountDownLatch]: https://blog.csdn.net/u012811805/article/details/104203801
[format_png 11]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy94aWNXWVRTSUN6UnZ6aGRLck1pY1NQSWliZzhQRWJPa2E4bmpjOHJFZmFLa09YTHAyd21wckxpY0JRaWNYMWlhdVFzWEdTTDlpYXhreFZTb3RQQ2F2U2hva1NvVHcvNjQw?x-oss-process=image/format,png
[format_png 12]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2dpZi94aWNXWVRTSUN6UnRDYWxzMWRGZFVjZ1lTWDVFMFhQN2liTVhOMUtqMlN0cmptVWoybEpMbHp0WUo0ejZqN1JYd0VGRUtjYUhpYmt4ekt2ODVqMlgwYTJLZy82NDA?x-oss-process=image/format,png