CentOS7 实现 Keepalived + Nginx 实现高可用 Web 负载均衡

逃离我推掉我的手 2023-01-21 08:28 213阅读 0赞

一、基础环境  
系统版本 nginx版本 keepalived版本 ip 作用  
CentOS Linux release 7.6 nginx/1.18.0 keepalived-2.1.5 192.168.86.135 master  
CentOS Linux release 7.6 nginx/1.18.0 keepalived-2.1.5 192.168.86.136 slave  
VIP:192.168.86.137

二、编译环境安装  
yum -y install make zlib zlib-devel gcc-c++ libtool openssl openssl-devel pcre pcre-devel

挂镜像到系统

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

mv /etc/yum.repos.d/CentOS-Debuginfo.repo /etc/yum.repos.d/CentOS-Debuginfo.repo.bak

mv /etc/yum.repos.d/CentOS-Vault.repo /etc/yum.repos.d/CentOS-Vault.repo.bak

cp /etc/yum.repos.d/CentOS-Media.repo /etc/yum.repos.d/CentOS-Media.repo.bak

sed -i ‘s\#file:///media/cdrom/\#\#g’ /etc/yum.repos.d/CentOS-Media.repo

sed -i ‘s\#file:///media/cdrecorder/\#\#g’ /etc/yum.repos.d/CentOS-Media.repo

sed -i ‘s/^ \*enabled *=.*$/enabled=1/g’ /etc/yum.repos.d/CentOS-Media.repo

sed -i ‘s\#^ \*baseurl *=.*$\#baseurl=file:///mnt/centos/\#g’ /etc/yum.repos.d/CentOS-Media.repo

mount -t iso9660 -o loop /tools/CentOS-7-x86\_64-DVD-1908.iso /mnt/centos

三、安裝nginx  
安裝nignx

\#进入目录  
cd /tools \#上传安装文件，并解压  
tar -zxvf nginx-1.18.0.tar.gz  
\#进入安装目录  
cd nginx-1.18.0  
\#检查配置  
./configure --prefix=/usr/local/nginx --with-stream

make&&make install

修改nginx配置文件

修改 Nginx 欢迎首页内容（用于后面测试， 用于区分两个节点的 Nginx）：

master

echo ‘this is master 135’ > /usr/local/nginx/html/index.html  
slave

echo ‘this is slave 136’ > /usr/local/nginx/html/index.html  
修改nginx.conf

# vi /usr/local/nginx/conf/nginx.conf #

user root;  
worker\_processes 1;  
\#error\_log logs/error.log;  
\#error\_log logs/error.log notice;  
\#error\_log logs/error.log info;  
\#pid logs/nginx.pid;  
events \{  
worker\_connections 1024;  
\}  
http \{  
include mime.types;  
default\_type application/octet-stream;  
\#log\_format main '$remote\_addr -  r e m o t e u s e r \[ remote\_user \[ remoteuser\[time\_local\] “$request” ’

# '$status  b o d y b y t e s s e n t " body\_bytes\_sent " bodybytessent"http\_referer" ’ #

# ‘“ h t t p u s e r a g e n t " " http\_user\_agent" " httpuseragent""http\_x\_forwarded\_for”’; #

\#access\_log logs/access.log main;  
sendfile on;  
\#tcp\_nopush on;  
\#keepalive\_timeout 0;  
keepalive\_timeout 65;  
\#gzip on;  
server \{  
listen 88;  
server\_name localhost;  
\#charset koi8-r;  
\#access\_log logs/host.access.log main;  
location / \{  
root html;  
index index.html index.htm;  
\}  
\#error\_page 404 /404.html;

# redirect server error pages to the static page /50x.html #

error\_page 500 502 503 504 /50x.html;  
location = /50x.html \{  
root html;  
\}  
\}  
\}  
\#user nobody;  
worker\_processes 2;  
error\_log logs/error.log;  
error\_log logs/error.log debug;  
error\_log logs/error.log info;  
\#pid logs/nginx.pid;  
worker\_rlimit\_nofile 65535;  
events \{  
use epoll;  
multi\_accept on;  
accept\_mutex on;  
worker\_connections 65535;  
\}  
stream\{  
upstream tcp21001\{  
server 192.168.10.130:11001 weight=1 max\_fails=2 fail\_timeout=30s;\#负载均衡的主机IP与端口 weight权重 代表分到资源的比例 max\_fails=2 失败超过2次，剔除 fail\_timeout=30s 30秒不会在连接些服务  
\}  
server\{  
\#listen \[::1\]:7777 \#IPV6  
listen 7777;\#监听本机端口号IPV4  
proxy\_pass tcp11001;  
proxy\_connect\_timeout 3s; \#服务不通时 超时时间  
proxy\_timeout 3600s; \#无数据传送，超时时间  
upstream tcp21002\{  
server 192.178.17.130:11002 weight=1 max\_fails=2 fail\_timeout=30s;\#负载均衡的主机IP与端口 weight权重 代表分到资源的比例 max\_fails=2 失败超过2次，剔除 fail\_timeout=30s 30秒不会在连接些服务  
\}  
server\{  
\#listen \[::1\]:8888 \#IPV6  
listen 8888;\#监听本机端口号IPV4  
proxy\_pass tcp11002;  
proxy\_connect\_timeout 3s; \#服务不通时 超时时间  
proxy\_timeout 3600s; \#无数据传送，超时时间  
\}  
\}  
启动nginx

systemctl start nginx  
测试nginx启动

curl localhost  
this is master135

4.CentOS 7设置Nginx开机启动

第一步：在/lib/systemd/system目录下创建nginx.service文件  
第二步：编辑nginx.service文件：  
vi /lib/systemd/system/nginx.service  
\[Unit\]  
Description=nginx  
After=network.target

\[Service\]  
Type=forking  
ExecStart=/usr/local/nginx/sbin/nginx  
ExecReload=/usr/local/nginx/sbin/nginx reload  
ExecStop=/usr/local/nginx/sbin/nginx quit  
PrivateTmp=true

\[Install\]  
WantedBy=multi-user.target  
\[Unit\]:服务的说明  
Description:描述服务  
After:描述服务类别  
\[Service\]服务运行参数的设置  
Type=forking是后台运行的形式  
ExecStart为服务的具体运行命令  
ExecReload为重启命令  
ExecStop为停止命令  
PrivateTmp=True表示给服务分配独立的临时空间  
注意：\[Service\]的启动、重启、停止命令全部要求使用绝对路径  
\[Install\]运行级别下服务安装的相关设置，可设置为多用户，即系统运行级别为3  
\#赋予脚本执行权限  
chmod +x /usr/lib/systemd/system/nginx.service  
第三步：加入开机启动

# systemctl enable nginx.service #

取消开机自启动  
\#systemctl disable nginx.service  
启动nginx服务  
\#systemctl start nginx.service  
停止nginx服务  
\#systemctl stop nginx.service  
重启nginx服务  
\#systemctl restart nginx.service  
查看所有以启动的服务  
\#systemctl list-units --type=service  
查看服务当前状态  
\#systemctl status nginx.service  
遇到的错误  
Warning: nginx.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.  
按照提示执行命令systemctl daemon-reload即可。  
生成系统service管理文件  
vvim /usr/lib/systemd/system/nginx.service

\[Unit\]

Description=nginx-The High-performance HTTP Server  
After=network.target

\[Service\]

Type=forking  
PIDFile= /usr/local/nginx/logs/nginx.pid  
ExecStartPre=/usr/local/nginx/sbin/nginx -t -c /usr/local/nginx/conf/nginx.conf  
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf  
ExecReload=/usr/local/nginx/sbin/nginx -s reload  
ExecStop=/usr/local/nginx/sbin/nginx -s stop  
PrivateTmp=true

\[Install\]

WantedBy=multi-user.target

6.加防火墙

添加

firewall-cmd --zone=public --add-port=88/tcp --permanent

删除

firewall-cmd --zone=public --remove-port=80/tcp --permanent

更新防火墙规则：  
firewall-cmd --reload

查看所有打开的端口：  
firewall-cmd --zone=public --list-ports

四、安装keepalived  
1、 创建依赖环境  
yum -y install libnl libnl-devel gcc openssl-devel libnl3-devel pcre-devel net-snmp-agent-libs libnfnetlink-devel curl  
wget  
2、安装keepalived  
tar -zxvf keepalived-2.1.5.tar.gz  
cd keepalived-2.1.5  
./configure --prefix=/usr/local/keepalived-2.1.5  
make && make install  
3、创建启动文件  
ln -s /usr/local/keepalived-2.1.5 /usr/local/keepalived  
mkdir /etc/keepalived/  
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/  
cp /usr/local/keepalived-2.1.5/etc/sysconfig/keepalived /etc/sysconfig/  
ln -s /usr/local/keepalived-2.1.5/sbin/keepalived /usr/sbin/  
下面文件要从keepalived源码目录复制，安装目录中没有cp  
cp /tools/keepalived-2.1.5/keepalived/keepalived.service /etc/systemd/system/  
cp /tools/keepalived-2.1.5/keepalived/etc/init.d/keepalived /etc/init.d/  
4、创建配置文件  
master

cat > /etc/keepalived/keepalived.conf << EOF  
! Configuration File for keepalived  
global\_defs \{  
\#一个没重复的名字即可  
router\_id xxoo\_master  
\}

# 检测nginx是否运行 #

vrrp\_script chk\_nginx \{  
script “/etc/keepalived/nginx\_check.sh”  
interval 2  
weight -20\}  
vrrp\_instance VI\_1 \{  
\# 此处不设置为MASTER，通过priority来竞争master  
state BACKUP  
\# 网卡名字，文章下方会给出如何获取网卡名字的方法 interface enp0s3  
\# 同一个keepalived集群的virtual\_router\_id相同  
virtual\_router\_id 51  
\# 权重，master要大于slave  
priority 100  
\# 主备通讯时间间隔  
advert\_int 1  
\# 如果两节点的上联交换机禁用了组播，则采用vrrp单播通告的方式  
\# 本机ip  
unicast\_src\_ip 192.168.0.182  
unicast\_peer \{  
\# 其他机器ip 192.168.0.189  
\}  
\# 设置nopreempt防止抢占资源  
nopreempt  
\# 主备保持一致  
authentication \{  
auth\_type PASS  
auth\_pass 1111  
\}  
\# 与上方nginx运行状况检测呼应  
track\_script \{  
chk\_nginx  
\}  
virtual\_ipaddress \{  
\# 虚拟ip地址（VIP，一个尚未占用的内网ip即可） 192.168.0.180  
\}  
\}  
EOF  
slave

cat > /etc/keepalived/keepalived.conf << EOF  
! Configuration File for keepalived  
global\_defs \{  
\#一个没重复的名字即可  
router\_id xxoo\_slave  
\}

# 检测nginx是否运行 #

vrrp\_script chk\_nginx \{  
script “/etc/keepalived/nginx\_check.sh”  
interval 2  
weight -20\}  
vrrp\_instance VI\_1 \{  
\# 此处不设置为MASTER，通过priority来竞争master  
state BACKUP  
\# 网卡名字，文章下方会给出如何获取网卡名字的方法 interface enp0s3  
\# 同一个keepalived集群的virtual\_router\_id相同  
virtual\_router\_id 51  
\# 权重，master要大于slave  
priority 90  
\# 主备通讯时间间隔  
advert\_int 1  
\# 如果两节点的上联交换机禁用了组播，则采用vrrp单播通告的方式  
\# 本机ip  
unicast\_src\_ip 192.168.0.189  
unicast\_peer \{  
\# 其他机器ip 192.168.0.182  
\}  
\# 设置nopreempt防止抢占资源  
nopreempt  
\# 主备保持一致  
authentication \{  
auth\_type PASS  
auth\_pass 1111  
\}  
\# 与上方nginx运行状况检测呼应  
track\_script \{  
chk\_nginx  
\}  
virtual\_ipaddress \{  
\# 虚拟ip地址（VIP，一个尚未占用的内网ip即可） 192.168.0.180  
\}  
\}  
EOF

5、启编写Nginx 状态检测脚本  
编写 Nginx 状态检测脚本 /etc/keepalived/nginx\_check.sh (已在 keepalived.conf 中配置)脚本要求：如果 nginx 停止运行，

尝试启动，如果无法启动则杀死本机的 keepalived 进程， keepalied将虚拟 ip 绑定到 BACKUP 机器上。 内容如下：

cat > /etc/keepalived/nginx\_check.sh << EOF  
\#!/bin/bash  
A=`ps -C nginx --no-header | wc -l`  
if \[ $A -eq 0 \];  
then  
systemctl start nginx

# /usr/local/nginx/sbin/nginx \#尝试重新启动nginx #

sleep 2 \#睡眠2秒  
if \[ `ps -C nginx --no-header | wc -l` \-eq 0 \];  
then  
killall keepalived  
fi  
fi  
保存后，给脚本赋执行权限：

# chmod +x /etc/keepalived/nginx\_check.sh #

每5秒执行一次脚本  
crontab -e

*   *   *   *   *  /etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 5;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 10;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 15;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 20;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 25;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 30;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 35;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 40;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 45;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 50;/etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log
 *   *   *   *   *  sleep 55; /etc/keepalived/nginx\_check.sh >>/etc/keepalived/check\_nginx.log

systemctl reload crond

systemctl restart crond

6、启动keepalived  
systemctl start keepalived  
systemctl enable keepalived  
7、日志处理  
1.编辑配置文件/etc/sysconfig/keepalived，将第14行的KEEPALIVED\_OPTIONS="-D"修改为KEEPALIVED\_OPTIONS="-D -d -S 0"

# 说明： #

# –dump-conf -d 导出配置数据 #

# –log-detail -D 详细日志信息 #

# –log-facility -S 设置本地的syslog设备，编号0-7\# -S 0 表示指定为local0设备 #

2.修改rsyslog的配置文件 vi /etc/rsyslog.conf，在结尾加入如下2行内容，将local0设备的所有日志都记录到/var/log/keepalived.log文件  
echo “local0.\* /var/log/keepalived.log” >> /etc/rsyslog.conf  
在42行第一列尾加入";local0.none"，表示来自local0设备的所有日志信息不再记录于/var/log/messages里  
3.重启rsyslog服务  
systemctl restart rsyslog  
4.测试keepalived日志记录结果。重启keepalived服务，查看日志信息。  
tail -f /var/log/keepalived.log  
systemctl start keepalived.service  
配置日志轮转  
/var/log/keepalived/\*.log \{ \#切分的两个文件名  
daily \#按天切分  
rotate 7 \#保留7份  
create 0644 haproxy haproxy \#创建新文件的权限、用户、用户组  
compress \#压缩旧日志  
delaycompress \#延迟一天压缩  
missingok \#忽略文件不存在的错误  
dateext \#旧日志加上日志后缀  
sharedscripts \#切分后的重启脚本只运行一次  
postrotate \#切分后运行脚本重载rsyslog，让rsyslog向新的日志文件中输出日志  
/bin/kill -HUP $(/bin/cat /var/run/syslogd.pid 2>/dev/null) &>/dev/null  
endscript  
\}  
防止出现脑裂现象（主备同时获取了VIP地址）

复制代码

# 指定keepalived配置的网卡：enp0s3，固定的VRRP广播地址：224.0.0.18 #

firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT  
firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT  
firewall-cmd --reload

# 查看配置的规则 #

firewall-cmd --direct --get-rules ipv4 filter INPUT  
firewall-cmd --direct --get-rules ipv4 filter OUTPUT

# 清除规则 #

firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT  
firewall-cmd --direct --permanent --remove-rule ipv4 filter OUTPUT 0 --out-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

五、测试  
1、在两台服务器上测试  
master

$ curl localhost  
this is master  
root@centos7\[14:46:07\]:~  
$ curl 192.168.86.135  
this is master  
root@centos7\[15:03:29\]:~  
slave

$ curl localhost  
this is slave  
root@centos7\[15:03:59\]:/etc/keepalived  
$ curl 192.168.86.136  
this is master  
2、关闭master的keepalived模仿down机  
master关闭keepalived

$ systemctl stop keepalived  
在slave上面进行测试  
$ curl localhost  
this is slave  
root@centos7\[15:10:29\]:/etc/keepalived  
$ curl 192.168.86.136  
this is slave  
到此keepalived完成

4.NetworkManager-nmcli管理网络  
nmcli general status \#显示NetworkManager的整体状态  
nmcli connection show \#显示所有的连接  
nmcli connection show -a \#显示活动的连接  
nmcli device status \#显示NetworkManager识别的设备列表和它们当前的状态  
nmcli device disconnect/connect eno16777736 \#停止/启动网卡==ifup/down

4、关闭NetworkManager服务

\#停止服务

service NetworkManager stop

\#禁用服务，下次不自动启动  
chkconfig NetworkManager off

问题处理：

【keepalived】关于keepalived配置中的 mcast\_src\_ip 和 unicast\_src\_ip

[https://blog.csdn.net/mofiu/article/details/76644012][https_blog.csdn.net_mofiu_article_details_76644012]

keepalived 配置文件参数详解

找到你要虚拟的网卡对应的域，并且开启vrrp协议即可，命令为

sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent  
\#重载配置  
sudo firewall-cmd --reload

来自 “开源世界 ” ，链接：[http://ym.baisou.ltd/post/518.html][http_ym.baisou.ltd_post_518.html]，如需转载，请注明出处，否则将追究法律责任。

[https_blog.csdn.net_mofiu_article_details_76644012]: http://github.crmeb.net/u/defu
[http_ym.baisou.ltd_post_518.html]: http://ym.baisou.ltd/post/518.html