企业 keepalived 高可用项目实战

谁践踏了优雅 2023-10-06 15:06 4阅读 0赞

#### **1、Keepalived VRRP 介绍** ####

keepalived是什么
         keepalived是集群管理中保证集群高可用的一个服务软件，用来防止单点故障。
     
     keepalived工作原理
         keepalived是以VRRP协议为实现基础的，VRRP全称Virtual Router Redundancy Protocol，即虚拟路由冗余协议。
     
         虚拟路由冗余协议，可以认为是实现路由器高可用的协议，即将N台提供相同功能的路由器组成一个路由器组，这个组里面有一个master和多个backup，master上面有一个对外提供服务的vip（该路由器所在局域网内其他机器的默认路由为该vip），master会发组播，当backup收不到vrrp包时就认为master宕掉了，这时就需要根据VRRP的优先级来选举一个backup当master。这样的话就可以保证路由器的高可用了。
     
     keepalived主要有三个模块，分别是core、check和vrrp。core模块为keepalived的核心，负责主进程的启动、维护以及全局配置文件的加载和解析。check负责健康检查，包括常见的各种检查方式。vrrp模块是来实现VRRP协议的。
     ==============================================
     脑裂：
     Keepalived的BACKUP主机在收到不MASTER主机报文后就会切换成为master，如果是它们之间的通信线路出现问题，无法接收到彼此的组播通知，但是两个节点实际都处于正常工作状态，这时两个节点均为master强行绑定虚拟IP，导致不可预料的后果，这就是脑裂。
     解决方式:
     1、添加更多的检测手段，比如ping对方等等。尽量减少"裂脑"发生机会。(指标不治本，只是提高了检测到的概率)；
     2、爆头，将master停掉。然后检查机器之间的防火墙。网络之间的通信

#### **2、Nginx+keepalived实现七层的负载均衡(同类服务)** ####

Nginx通过Upstream模块实现负载均衡

#### **upstream 支持的负载均衡算法** ####

轮询（默认）:可以通过weight指定轮询的权重，权重越大，被调度的次数越多
     ip_hash：可以实现会话保持，将同一客户的IP调度到同一样后端服务器，可以解决session的问题，不能使用weight
     fair：可以根据请求页面的大小和加载时间长短进行调度，使用第三方的upstream_fair模块
     url_hash：按请求的url的hash进行调度，从而使每个url定向到同一服务器，使用第三方的url_hash模块
     配置安装nginx 所有的机器，关闭防火墙和selinux
     [root@nginx-proxy ~]# cd /etc/yum.repos.d/
     [root@nginx-proxy yum.repos.d]# vim nginx.repo
     [nginx-stable]
     name=nginx stable repo
     baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
     gpgcheck=0
     enabled=1
     [root@nginx-proxy yum.repos.d]# yum install yum-utils -y
     [root@nginx-proxy yum.repos.d]# yum install nginx -y
     调度到不同组后端服务器
     网站分区进行调度
     =================================================================================
     
     拓扑结构
     
                                 [vip: 20.20.20.20]
     
                             [LB1 Nginx]     [LB2 Nginx]
                             192.168.1.2     192.168.1.3
     
             [index]     [milis]      [videos]      [images]       [news]
              1.11        1.21          1.31           1.41         1.51
              1.12        1.22          1.32           1.42         1.52
              1.13        1.23          1.33           1.43         1.53
              ...         ...            ...           ...           ...
              /web     /web/milis    /web/videos     /web/images   /web/news
           index.html  index.html     index.html      index.html   index.html
     一、实施过程 
     1、选择两台nginx服务器作为代理服务器。
     2、给两台代理服务器安装keepalived制作高可用生成VIP
     3、配置nginx的负载均衡
     以上两台nginx服务器配置文件一致
     根据站点分区进行调度
     配置upstream文件
     [root@nginx-proxy ~]# cd /etc/nginx/conf.d/
     [root@nginx-proxy conf.d]# cp default.conf proxy.conf
     [root@nginx-proxy conf.d]# mv default.conf default.conf.bak
     [root@nginx-proxy conf.d]# vim upstream.conf
         upstream index {
             server 192.168.246.162:80 weight=1 max_fails=2 fail_timeout=2;
             server 192.168.246.163:80 weight=2 max_fails=2 fail_timeout=2;
            }       
     [root@nginx-proxy conf.d]# vim proxy.conf
     server {
             listen 80;
             server_name     localhost;
             access_log  /var/log/nginx/host.access.log  main;
             
             location / {
              proxy_pass http://index;
              proxy_redirect default;
              proxy_set_header Host $http_host;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
             }
     }
     将nginx的配置文件拷贝到另一台代理服务器中:
     [root@nginx-proxy conf.d]# scp proxy.conf 192.168.246.161:/etc/nginx/conf.d/ 
     [root@nginx-proxy conf.d]# scp upstream.conf 192.168.246.161:/etc/nginx/conf.d/
     二、Keepalived实现调度器HA
     注：主/备调度器均能够实现正常调度
     1. 主/备调度器安装软件
     [root@nginx-proxy-master ~]# yum install -y keepalived
     [root@nginx-proxy-slave ~]# yum install -y keepalived
     [root@nginx-proxy-master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
     [root@nginx-proxy-master ~]# vim /etc/keepalived/keepalived.conf
     ! Configuration File for keepalived
     
     global_defs {
        router_id directory1   #辅助改为directory2
     }
     
     vrrp_instance VI_1 {
         state MASTER        #定义主还是备
         interface ens33     #VIP绑定接口
         virtual_router_id 80  #整个集群的调度器一致
         priority 100         #back改为50
         advert_int 1         #检查间隔，默认为1s
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.246.16/24
         }
     }
     
     [root@nginx-porxy-slave ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
     [root@nginx-proxy-slave ~]# vim /etc/keepalived/keepalived.conf
     ! Configuration File for keepalived
     
     global_defs {
        router_id directory2
     }
     
     vrrp_instance VI_1 {
         state BACKUP    #设置为backup
         interface ens33
         virtual_router_id 80
         priority 50   #辅助改为50
         advert_int 1
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.246.16/24
         }
     }
     3. 启动KeepAlived（主备均启动）
     [root@nginx-proxy-master ~]# chkconfig keepalived on
     [root@nginx-proxy-master ~]# service keepalived start
     [root@nginx-proxy-master ~]# ip addr
     : lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
         inet 127.0.0.1/8 scope host lo
            valid_lft forever preferred_lft forever
         inet6 ::1/128 scope host 
            valid_lft forever preferred_lft forever
     2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
         link/ether 00:0c:29:48:07:7d brd ff:ff:ff:ff:ff:ff
         inet 192.168.246.169/24 brd 192.168.246.255 scope global dynamic ens33
            valid_lft 1726sec preferred_lft 1726sec
         inet 192.168.246.16/24 scope global ens33
            valid_lft forever preferred_lft forever
         inet6 fe80::23e9:de18:1e67:f152/64 scope link 
            valid_lft forever preferred_lft forever
     到此：
     可以解决心跳故障keepalived
     不能解决Nginx服务故障
     4. 扩展对调度器Nginx健康检查（可选）两台都设置
     思路：
     让Keepalived以一定时间间隔执行一个外部脚本，脚本的功能是当Nginx失败，则关闭本机的Keepalived
     (1) script
     [root@nginx-proxy-master ~]# vim /etc/keepalived/check_nginx_status.sh
     #!/bin/bash                                                     
     /usr/bin/curl -I http://localhost &>/dev/null   
     if [ $? -ne 0 ];then                                            
     #   /etc/init.d/keepalived stop
         systemctl stop keepalived
     fi                                                                      
     [root@nginx-proxy-master ~]# chmod a+x /etc/keepalived/check_nginx_status.sh
     
     (2). keepalived使用script
     ! Configuration File for keepalived
     
     global_defs {
        router_id director1
     }
     vrrp_script check_nginx {
        script "/etc/keepalived/check_nginx_status.sh"
        interval 5
     }
     
     vrrp_instance VI_1 {
         state MASTER
         interface ens33
         virtual_router_id 80
         priority 100
         advert_int 1
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.246.16/24
         }
         track_script {
             check_nginx
         }
     }
     注：必须先启动nginx，再启动keepalived
     
     测试访问：

#### **3、LVS\_Director + KeepAlived** ####

![4420043c13a2481a3b685d4983ca48bc.png][]

LVS_Director + KeepAlived
     
     KeepAlived在该项目中的功能：
     1. 管理IPVS的路由表（包括对RealServer做健康检查）
     2. 实现调度器的HA
     http://www.keepalived.org
     
     Keepalived所执行的外部脚本命令建议使用绝对路径
     =================================================================================
     实施步骤：
     1. 主/备调度器安装软件
     [root@lvs-keepalived-master ~]# yum -y install ipvsadm keepalived 
     [root@lvs-keepalived-slave ~]# yum -y install ipvsadm keepalived
     2. Keepalived
     lvs-master
     [root@ha-proxy-master ~]# vim /etc/keepalived/keepalived.conf
     ! Configuration File for keepalived
     
     global_defs {
        router_id lvs-keepalived-master    #辅助改为lvs-backup
     }
     
     vrrp_instance VI_1 {
         state MASTER
         interface ens33                #VIP绑定接口
         virtual_router_id 80         #VRID 同一组集群，主备一致          
         priority 100            #本节点优先级，辅助改为50
         advert_int 1            #检查间隔，默认为1s
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.246.110/32
         }
     }
     
     virtual_server 192.168.246.110 80 {    #LVS配置
         delay_loop 6   #健康检查rs时间间隔
         lb_algo rr     #LVS调度算法
         lb_kind DR     #LVS集群模式（路由模式）
         protocol TCP      #健康检查使用的协议
         real_server 192.168.246.162 80 {
             weight 1
             inhibit_on_failure   #当该节点失败时，把权重设置为0，而不是从IPVS中删除
             TCP_CHECK {          #健康检查
                 connect_port 80   #检查的端口
                 connect_timeout 3  #连接超时的时间
                 }
             }
         real_server 192.168.246.163 80 {
             weight 1
             inhibit_on_failure
             TCP_CHECK {
                 connect_timeout 3
                 connect_port 80
                 }
             }
     }
     
     [root@lvs-keepalived-slave ~]# vim /etc/keepalived/keepalived.conf
     ! Configuration File for keepalived
     
     global_defs {
        router_id lvs-keepalived-slave
     }
     
     vrrp_instance VI_1 {
         state BACKUP
         interface ens33
         virtual_router_id 80
         priority 50
         advert_int 1
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.246.110/32
         }
     }
     virtual_server 192.168.246.110 80 {
         delay_loop 6
         lb_algo rr
         lb_kind DR
         protocol TCP
         real_server 192.168.246.162 80 {
             weight 1
             inhibit_on_failure
             TCP_CHECK {
                 connect_port 80
                 connect_timeout 3
                 }
             }
         real_server 192.168.246.163 80 {
             weight 1
             inhibit_on_failure
             TCP_CHECK {
                 connect_timeout 3
                 connect_port 80
                 }
             }
     }
     3. 启动KeepAlived（主备均启动）
     [root@lvs-keepalived-master ~]# systemctl start keepalived
     [root@lvs-keepalived-master ~]# systemctl enable keepalived
     
     [root@lvs-keepalived-master ~]# ipvsadm -Ln
     IP Virtual Server version 1.2.1 (size=4096)
     Prot LocalAddress:Port Scheduler Flags
       -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
     TCP  192.168.246.110:80 rr persistent 20
       -> 192.168.246.162:80           Route   1      0          0         
       -> 192.168.246.163:80           Route   0      0          0
     
     4. 所有RS配置(nginx1,nginx2)
     配置好网站服务器，测试所有RS
     [root@test-nginx1 ~]# yum install -y nginx
     [root@test-nginx2 ~]# yum install -y nginx
     [root@test-nginx1 ~]# ip addr add dev lo 192.168.246.110/32
     [root@test-nginx1 ~]# echo "net.ipv4.conf.all.arp_ignore = 1" >> /etc/sysctl.conf
     [root@test-nginx1 ~]# sysctl -p
     [root@test-nginx1 ~]# echo "web1..." >> /usr/share/nginx/html/index.html
     [root@test-nginx1 ~]# systemctl start nginx

**3.测试数据库高可用**

1.两台机器安装mysql
     [root@mysql-keep-master ~]# wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
     [root@mysql-keep-master ~]# rpm -ivh mysql80-community-release-el7-3.noarch.rpm
     [root@mysql-keep-master ~]# yum -y install yum-utils #安装yum工具包
     [root@mysql-keep-master ~]# vim /etc/yum.repos.d/mysql-community.repo

![f2ea954b973dbdad79e9924b5eaa5b1d.png][]

安装mysql
     [root@mysql-keep-master ~]# yum install -y mysql-community-server
     [root@mysql-keep-master ~]# systemctl start mysqld 
     [root@mysql-keep-master ~]# systemctl enable mysqld
     修改密码
     [root@mysql-keep-master ~]# grep pass /var/log/mysqld.log 
     2020-02-18T13:54:25.706635Z 1 [Note] A temporary password is generated for root@localhost: %baWi=4!NsT)
     [root@mysql-keep-master ~]# mysqladmin -uroot -p'%baWi=4!NsT)' password 'QianFeng@123!'
     制作互为主从:
     [root@mysql-keep-master ~]# vim /etc/my.cnf   #编辑配置文件开启gtid
     log-bin=m1.log  
     server-id=1 
     gtid_mode = ON    #开启gtid
     enforce_gtid_consistency=1  #强制gtid移植性
     [root@mysql-keep-master ~]# systemctl restart mysqld 
     [root@mysql-keep-master ~]# mysql -uroot -p'QianFeng@123!'  #登录数据库
     mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'QianFeng@123';  #创建远程登录用户
     mysql> flush privileges;
     mysql> grant replication  slave,reload,super  on *.*  to  'slave'@'%'  identified by  'QianFeng@123';  ---创建主从授权用户
     mysql> flush privileges;  #刷新授权
     mysql> \e   #指定主服务器信息
     CHANGE MASTER TO
     MASTER_HOST='192.168.58.136',      #指定主服务器的ip或者主机名
     MASTER_USER='slave',           #授权用户
     MASTER_PASSWORD='QianFeng@123',  #授权用户密码
     master_auto_position=1
     -> ;
     启动slave
     mysql> start slave;
     [root@mysql-keep-backup ~]# vim /etc/my.cnf
     log-bin=m2.log
     server-id=2
     gtid_mode = ON
     enforce_gtid_consistency=1
     [root@mysql-keep-backup ~]# systemctl restart mysqld 
     [root@mysql-keep-backup ~]# mysql -uroot -p'QianFeng@123!'
     mysql> GRANT ALL ON *.* TO 'root'@'%' IDENTIFIED BY 'QianFeng@123';
     mysql> flush privileges;
     mysql> grant replication  slave,reload,super  on *.*  to  'slave'@'%'  identified by  'QianFeng@123';
     mysql> flush privileges;
     指定主服务器信息
     mysql> \e
     CHANGE MASTER TO
     MASTER_HOST='192.168.58.135',
     MASTER_USER='slave',
     MASTER_PASSWORD='QianFeng@123',
     master_auto_position=1
     -> ;
     启动slave
     mysql> start slave;

两台机器验证是否为yes

mysql> show slave status\G

两台机器安装keepalived

[root@mysql-keep-master ~]# yum -y install keepalived
     [root@mysql-keep-master ~]# cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
     [root@mysql-keep-master ~]# vim /etc/keepalived/keepalived.conf
     ! Configuration File for keepalived
     
     global_defs {
        router_id master1
     }
     
     vrrp_instance VI_1 {
         state MASTER
         interface ens33
         virtual_router_id 80
         priority 100
         advert_int 1
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.58.16/24
         }
     }
     将配置文件拷贝到backu机器
     [root@mysql-keep-backup ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
     [root@mysql-keep-master ~]# scp /etc/keepalived/keepalived.conf 192.168.58.136:/etc/keepalived/
     [root@mysql-keep-backup ~]# vim /etc/keepalived/keepalived.conf  #修改如下
     ! Configuration File for keepalived
     
     global_defs {
        router_id backup1
     }
     
     vrrp_instance VI_1 {
         state BACKUP
         interface ens33
         virtual_router_id 80
         priority 50
         advert_int 1
         authentication {
             auth_type PASS
             auth_pass 1111
         }
         virtual_ipaddress {
             192.168.58.16/24
         }
     }
     两台机器启动keepalived
     [root@mysql-keep-master ~]# systemctl start keepalived
     [root@mysql-keep-backup ~]# systemctl start keepalived
     [root@mysql-keep-master ~]# ip a   #vip在master上面
     1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
         inet 127.0.0.1/8 scope host lo
            valid_lft forever preferred_lft forever
         inet6 ::1/128 scope host 
            valid_lft forever preferred_lft forever
     2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
         link/ether 00:0c:29:92:0a:64 brd ff:ff:ff:ff:ff:ff
         inet 192.168.58.135/24 brd 192.168.58.255 scope global dynamic ens33
            valid_lft 1729sec preferred_lft 1729sec
         inet 192.168.58.16/24 scope global secondary ens33
            valid_lft forever preferred_lft forever
         inet6 fe80::86f4:1d74:ad05:c61c/64 scope link 
            valid_lft forever preferred_lft forever
     测试：
     [root@mysql-keep-master ~]# mysql -uroot -p'QianFeng@123' -h 192.168.58.16  #登录vip
     mysql> create database test1;
     Query OK, 1 row affected (0.07 sec)
     
     mysql> show databases;
     +--------------------+
     | Database           |
     +--------------------+
     | information_schema |
     | mysql              |
     | performance_schema |
     | sys                |
     | test1              |
     +--------------------+
     5 rows in set (0.08 sec)
     验证
     [root@mysql-keep-backup ~]# mysql -uroot -p'QianFeng@123!'
     mysql> show databases;
     +--------------------+
     | Database           |
     +--------------------+
     | information_schema |
     | mysql              |
     | performance_schema |
     | sys                |
     | test1              |
     +--------------------+
     5 rows in set (0.12 sec)
     [root@mysql-keep-master ~]# systemctl stop keepalived #关闭master
     [root@mysql-keep-backup ~]# ip a  #vip切换backup机器
     1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
         link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
         inet 127.0.0.1/8 scope host lo
            valid_lft forever preferred_lft forever
         inet6 ::1/128 scope host 
            valid_lft forever preferred_lft forever
     2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
         link/ether 00:0c:29:a2:0f:5c brd ff:ff:ff:ff:ff:ff
         inet 192.168.58.136/24 brd 192.168.58.255 scope global dynamic ens33
            valid_lft 1547sec preferred_lft 1547sec
         inet 192.168.58.16/24 scope global secondary ens33
            valid_lft forever preferred_lft forever
         inet6 fe80::af6e:2eff:e2d1:99cb/64 scope link 
            valid_lft forever preferred_lft forever
     测试登录
     [root@mysql-keep-master ~]# mysql -uroot -p'QianFeng@123' -h 192.168.58.16

[4420043c13a2481a3b685d4983ca48bc.png]: https://img-blog.csdnimg.cn/img_convert/4420043c13a2481a3b685d4983ca48bc.png
[f2ea954b973dbdad79e9924b5eaa5b1d.png]: https://img-blog.csdnimg.cn/img_convert/f2ea954b973dbdad79e9924b5eaa5b1d.png