Nginx系列六：Keepalived+Nginx高可用集群（主从模式）

墨蓝 2023-01-03 04:57 81阅读 0赞

一：Nginx高可用集群解决的问题？

在我们负载均衡配置中有一台非常关键的服务器，nginx反向代理服务器，所有的任务分发都是从他这里发出的， 如果这台反向代理服务器挂了，整个流程全部down了，所以需要对负载均衡进行监听，添加一台备份服务器做容灾。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlYl9zbmFpbA_size_16_color_FFFFFF_t_70][]

二：Nginx高可用集群实现思路

1：如上所述我们需要有个程序一直去监听nginx是否down，这里我们使用Keepalived来做

2：当我们发现nginx反向代理服务器down了，我们需要找一台服务器做替代

当nginx主服务故障，使用备份服务器的时候，对外暴露的IP会发生改变，为了减少web端的操作，我们需要统一对外的IP，这就是所谓的虚拟IP， 虚拟IP解决的问题就是当我的nginx故障切换了，对外暴露的IP始终是不变的， 不会对web层有影响。

主要是采用 nginx负载均衡 + 监控(keepalived)来实现，keepalived来监听Nginx是否存活，同时维护 虚拟IP对外访问解决静态路由的单点故障。

三：Keepalived+Nginx高可用集群具体配置

具体架构如下：

192.168.8.203：nginx(主)  Vip(虚拟IP): 192.168.8.222
        192.168.8.200：nginx(备)
        
        192.168.8.201：server1
        192.168.8.202：server2

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlYl9zbmFpbA_size_16_color_FFFFFF_t_70 1][]

3.1 ：主备服务器配置负载均衡

主服务器：192.168.8.203
    vim /usr/local/nginx/conf/nginx.conf
    http {
    	....
        upstream myServer {
            ip_hash;
            server 192.168.8.201:8080 weight=1 max_fails=1 fail_timeout=10;
            server 192.168.8.202:8080 weight=1 max_fails=1 fail_timeout=10;
        }
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'  '$connection $upstream_addr '  'upstream_response_time $upstream_response_time request_time $request_time ';
        access_log  /var/log/nginx/access.log  main;
       ....
    } 
    
    在单个server块添加反向代理
    vim /usr/local/nginx/conf.d/test.conf
        #access_log /var/log/nginx/test/access.log;
        ....
        location / {   # 请求全部跳转
            proxy_pass http://myServer;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
        ....
    }
    
    备份slave服务器配置步骤和主服务器一样
    ps: 配置完记得重启nginx

3.2 ：主备服务器安装配置keepalived

1：安装keepalived

yum install keepalived –y

2：配置主服务器keepalived

vim /etc/keepalived/keepalived.conf
    global_defs {
        notification_email {
    	iwester@163.com
        }
        notification_email_from Alexandre.Cassen@firewall.loc
        smtp_server 192.168.200.1
        smtp_connect_timeout 30
        script_user root
        enable_script_security
        router_id LVS_01
    }
    
    vrrp_script chk_nginx_port {
        script "/usr/local/nginx/src/nginx_check.sh"
        interval 5
        weight -20
    }
    
    vrrp_instance VI_1 {
        state MASTER
        interface ens33
        virtual_router_id 51
        priority 100
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        unicast_src_ip 192.168.8.203
        unicast_peer {
            192.168.8.203
            192.168.8.200
        }
        virtual_ipaddress {
    	192.168.8.222/24
        }  
        track_script
        {
            chk_nginx_port
        }
    }

3：配置备份服务器keepalived

vim /etc/keepalived/keepalived.conf
    global_defs {
        notification_email {
    	iwester@163.com
        }
        notification_email_from Alexandre.Cassen@firewall.loc
        smtp_server 192.168.200.1
        smtp_connect_timeout 30
        script_user root
        enable_script_security
        router_id LVS_02
    }
    
    vrrp_script chk_nginx_port {
        script "/usr/local/nginx/src/nginx_check.sh"
        interval 5
        weight -20
    }
    
    vrrp_instance VI_1 {
        state BACKUP
        interface ens33
        virtual_router_id 51
        priority 90
        advert_int 1
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        unicast_src_ip 192.168.8.200
        unicast_peer {
            192.168.8.203
            192.168.8.200
        }
        virtual_ipaddress {
             192.168.8.222/24
        } 
        track_script
        {
            chk_nginx_port
        }
    }

4：主备库都要创建nginx的监控脚本， 如果不适用keepalived自带的vrrp\_script也可以自己写一个脚本，常驻后台去判断nginx是否存活！

mkdir -p /usr/local/nginx/src
    vim /usr/local/nginx/src/nginx_check.sh
    
    #!/bin/sh
    nginxpid=$(ps -C nginx --no-header|wc -l)
    #判断Nginx是否存活
    if [ $nginxpid -eq 0 ];then
        systemctl stop keepalived
        exit 1
    fi
    
    
    chmod -R 744 /usr/local/nginx/src/nginx_check.sh
    
    配置说明：判断nginx是否存活，如果nginx挂了，就把keepalived停掉，备库的keepalived就会争抢主库的虚拟IP变成主库， 因为keepalived是服务器级别的监控，nginx挂掉并不会有什么反应，所以用脚本关闭keepalived让备库去抢占虚拟IP

5：启动keepalived

systemctl start keepalived

查看主服务器虚拟ip

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlYl9zbmFpbA_size_16_color_FFFFFF_t_70 2][]

查看主服务器广播

sudo tcpdump -i ens33 vrrp -n
    通过keepalived的vrrp通信发布存活信息给备库

![20210113195841641.png][]

6：测试高可用

关闭主服务器nginx
    systemctl stop nginx
    
    监控日志tail -f /var/log/messages
    发现异常：
    1：Unsafe permissions found for script /usr/local/nginx/src/nginx_check.sh
    解决：chmod -R 744 /usr/local/nginx/src/nginx_check.sh
    
    2：default user 'keepalived_script' for script execution does not exist - please create.
    解决：
    global_def{
        ...  
        #添加配置
        script_user root
        enable_script_security
        ...
    }
     3: /usr/local/nginx/src/nginx_check.sh exited with status 1
     解决：一般是由seliunx防火墙导致
     setenforce 0 #临时关闭
     vi /etc/sysconfig/selinux
     SELINUX=disabled
     
     4： /usr/local/nginx/src/nginx_check.sh exited due to signal 15
     解决：vrrp_script{}中的interval时间需大于脚本中的sleep时间

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlYl9zbmFpbA_size_16_color_FFFFFF_t_70]: /images/20221119/dc9f355d199940deba303fdab572b6a6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlYl9zbmFpbA_size_16_color_FFFFFF_t_70 1]: /images/20221119/4f1397313e6a41b0b33b5c0055db9787.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlYl9zbmFpbA_size_16_color_FFFFFF_t_70 2]: /images/20221119/91c2aa36cb9d413db85200ce69ded698.png
[20210113195841641.png]: /images/20221119/f07831e146ef4750a6018a7c07b3718a.png