LVS集群: linux virtual server

怼烎@ 2022-12-31 07:27 73阅读 0赞

### 文章目录 ###

*  LVS 项目介绍
 *  ipvsadm 功能
 *   *  a, 使用场景
     *  b, 内存和性能
     *  c, 调度算法
     *  d, 使用ipvsadm配置: LVS-DR
     *  e, RS/LVS balancer配置VIP说明
 *  keepalived + ipvsadm
 *   *  配置keepalived 使用VMAC地址通信
     *  MASTER配置
     *   *  生成LVS服务
     *  BACKUP配置

# LVS 项目介绍 #

[http://www.linuxvirtualserver.org/whatis.html][http_www.linuxvirtualserver.org_whatis.html]

# ipvsadm 功能 #

[http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.ipvsadm.html\#using\_ipvsadm][http_www.austintek.com_LVS_LVS-HOWTO_HOWTO_LVS-HOWTO.ipvsadm.html_using_ipvsadm]

注意事项： put the names of the realservers in `/etc/hosts`， ipvsadm by default outputs the names of the realservers rather than the IPs， or you could use `ipvsadm -n` which outputs the IPs of the realservers instead.

功能/用途： （缺点：不能自动移除故障的RS，或添加修改后的RS,可以使用keepalived来支持)

*  添加服务 add services: add a service with weight >0 \[ 0 to (2^24-1) \]  
    shutdown (or quiesce) services: set the weight to 0.  
    This allows current connections to continue, untill they disconnect or expire, but will not allow new connections. When there are no connections remaining, you can bring down the service/realserver.
 *  删除服务delete services: this stops traffic for the service (the connection will hang), but the entry in the connection table is not deleted till it times out. This allows deletion, followed shortly thereafter by adding back the service, to not affect established (but quiescent) connections.
 *  备份/恢复配置  
    $ ipvsadm-sav > ipvsadm.sav  
    and then after reboot, restore the ipvsadm settings, with ipvsadm-restore  
    $ ipvsadm-restore < ipvsadm.sav  
    Both of these commands can be part of an ipvsadm init script.

## a, 使用场景 ##

*  In a system where 700Mbit/s traffic is flowing through it
 *  Concurrent connection number is about 420.000
 *  Our main purpose for using LVS is to direct 80. port requests into number of squid servers (~80 servers)

## b, 内存和性能 ##

*  Each entry takes 128 bytes：the entries for each connection are stored in a hash table
 *  LVS-DR + squid 缓存服务 + 420000并非连接 = 内存消耗【420000 x 128 x \[RTTmin/RTTmin+maxIdleTime\] 】bytes  
    粗算估计：LVS-DR + 4G内存~= 500万并发连接

## c, 调度算法 ##

*  round robin (rr), weighted round robin (wrr)
 *  least connected (lc), weighted least connection (wlc)
 *  DH: destination hash，makes a hash from the target IP and all requests to that IP will be sent to the same realserver
 *  LBLC：a persistent memory algorythm, maybe lblc = dh + wlc.
 *  SH: source hash
 *  persistence：持久连接

## d, 使用ipvsadm配置: LVS-DR ##

[http://www.linuxvirtualserver.org/VS-DRouting.html][http_www.linuxvirtualserver.org_VS-DRouting.html]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]  
TCP报文：收发

*  客户端报文SrvMac->to-> VIP：balancer 接收后选择一个RS并转发给它；
 *  RS拆报文，看到是VIP发过来的，而自己也有一个VIP, 于是封装报文VIP->to -> SrvMac  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]

## e, RS/LVS balancer配置VIP说明 ##

RS和LVS balancer 配置 VIP=172.26.20.110

<table> 
 <thead> 
  <tr> 
   <th align="left">角色</th> 
   <th align="left">执行动作</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="left">LVS balancer:172.26.20.111</td> 
   <td align="left">ifconfig eth0:0 $VIP/24 broadcast $VIP up</td> 
  </tr> 
  <tr> 
   <td align="left"></td> 
   <td align="left">echo 1 &gt; /proc/sys/net/ipv4/ip_forward</td> 
  </tr> 
  <tr> 
   <td align="left"></td> 
   <td align="left">ipvsadm -A -t $VIP:23 -s wlc</td> 
  </tr> 
  <tr> 
   <td align="left"></td> 
   <td align="left">ipvsadm -a -t $VIP:23 -r 172.26.20.112 -g</td> 
  </tr> 
  <tr> 
   <td align="left">RS:172.26.20.112</td> 
   <td align="left">ifconfig lo:0 $VIP/32 broadcast $VIP up #or ip addr add $VIP/32 dev lo label lo:0</td> 
  </tr> 
  <tr> 
   <td align="left"></td> 
   <td align="left">route add -net 172.26.20.0 netmask 255.255.255.0 dev eth0</td> 
  </tr> 
  <tr> 
   <td align="left"></td> 
   <td align="left">route add -host $VIP dev lo:0</td> 
  </tr> 
 </tbody> 
</table>

*  On lo the interface will answer to *all* addresses covered by the netmask. This is how 127.0.0.1/8 on lo ends up answering 127.0.0.0-127.255.255.255.
 *  So if you add 172.16.4.222/16 to eth0 then it will answer 172.16.4.222 and only 172.16.4.222. But if you add the same thing to lo then it will answer 172.16.0.0-172.16.255.255. So you need to use 172.16.4.222/32 instead.

#普通网卡eth0： vip 子网掩码/24 == > 仅响应vip
    ifconfig eth0:0 192.168.10.10 netmask 255.255.255.0 broadcast 192.168.10.255 up
       -> Add 192.168.10.10 to eth0
    
    #回环网卡lo： vip 子网掩码/24 == > 响应0-255个IP
    ifconfig lo:0 192.168.10.10 netmask 255.255.255.0 broadcast 192.168.10.255 up
       -> Add 192.168.10.0 - 192.168.10.255 to lo
    # vip段ip.0 子网掩码/24 == > 响应0-255个IP 
    ifconfig lo:0 192.168.10.0 netmask 255.255.255.0 broadcast 192.168.10.255 up
       -> Same as above, add 192.168.10.0 - 192.168.10.255 to lo
    # vip 子网掩码/32 == > 仅响应vip 
    ifconfig lo:0 192.168.10.10 netmask 255.255.255.255 broadcast 192.168.10.10 up
       -> Add 192.168.10.10 to lo

# keepalived + ipvsadm #

[https://www.keepalived.org/doc/introduction.html][https_www.keepalived.org_doc_introduction.html]  
[http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.failover.html\#keepalived\_vrrpd][http_www.austintek.com_LVS_LVS-HOWTO_HOWTO_LVS-HOWTO.failover.html_keepalived_vrrpd]  
功能/用途

*  Health checking for LVS systems: realserver 故障检查(移除故障的RS, 可以自动添加修复后的RS)
 *  Implementation of the VRRPv2 stack to handle load balancer failover：VIP故障转移

[root@c72 ~]# ip a
    6: vrrp.51@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
        link/ether 00:00:5e:00:01:33 brd ff:ff:ff:ff:ff:ff
        inet 192.168.56.254/32 scope global vrrp.51
           valid_lft forever preferred_lft forever
    
    [root@c72 ~]# tailf /var/log/messages
    Dec 30 11:39:12 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
    Dec 30 11:39:15 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
    Dec 30 11:39:18 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
    Dec 30 11:39:21 c72 Keepalived_healthcheckers[8989]: Error connecting server [192.168.56.7]:80.
    Dec 30 11:39:21 c72 Keepalived_healthcheckers[8989]: Check on service [192.168.56.7]:80 failed after 3 retry.
    Dec 30 11:39:21 c72 Keepalived_healthcheckers[8989]: Removing service [192.168.56.7]:80 from VS [192.168.56.254]:80
    
    
    Dec 30 11:41:33 c72 Keepalived_healthcheckers[8989]: HTTP status code success to [192.168.56.7]:80 url(1).
    Dec 30 11:41:33 c72 Keepalived_healthcheckers[8989]: Remote Web server [192.168.56.7]:80 succeed on service.
    Dec 30 11:41:33 c72 Keepalived_healthcheckers[8989]: Adding service [192.168.56.7]:80 to VS [192.168.56.254]:80

组成部件

*  keepalived daemon:  
    一个主（父）进程： in charge with forked children process monitoring.并重启失败的子进程  
    两个子进程： one responsible for VRRP framework and the other for healthchecking.
 *  genhash utility

#配置virtual_server :HTTP_GET --> digest
    [root@lvs-c6 ~]# grep digest /etc/keepalived/keepalived.conf
               url { 
                  path /index.html
                  #digest 640205b7b0fc66c1ea91c463fac6334c
                  status_code 200
                }
                          
    [root@lvs-c6 ~]# genhash -s 192.168.56.7 -p 80 -u /index.html
    MD5SUM = 3ab3417b4624b998d70adf9ac9fbb8ae

## 配置keepalived 使用VMAC地址通信 ##

[https://www.keepalived.org/doc/software\_design.html\#note-on-using-vrrp-with-virtual-mac-address][https_www.keepalived.org_doc_software_design.html_note-on-using-vrrp-with-virtual-mac-address]

*  To reduce takeover impact, some networking environment would require using VRRP with VMAC address. To reach that goal Keepalived VRRP framework implements VMAC support by the invocation of ‘use\_vmac’ keyword in configuration file

查看系统（MASTER和BACKUP) 是否支持 macvlan

[root@lvs-c6 ~]# lsmod | grep macvlan
    macvlan                10356  0
    
    #配置内核arp参数
    [root@lvs-c6 ~]# tail /etc/sysctl.conf
    net.ipv4.conf.all.arp_ignore = 1
    net.ipv4.conf.all.arp_announce = 1
    net.ipv4.conf.all.arp_filter = 0
    net.ipv4.conf.eth1.arp_filter = 1
    [root@lvs-c6 ~]# sysctl -w

## MASTER配置 ##

global_defs { 
       router_id LVS_DEVEL
    }
    
    vrrp_instance VI_1 { 
        state MASTER
        interface eth1
        virtual_router_id 51
        #use_vmac interface_prefix_name
        use_vmac
           vmac_xmit_base
    
        priority 100
        advert_int 1
        authentication { 
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress { 
            192.168.56.254
        }
    }
    
    virtual_server 192.168.56.254 80 { 
        delay_loop 6
        lb_algo rr
        lb_kind DR
        #使用lvs持久连接：保证session连接同一个主机
        persistence_timeout 5
        protocol TCP
    
        #sorry_server 192.168.200.200 1358
    
        real_server 192.168.56.8 80 { 
            weight 1
            HTTP_GET { 
                url { 
                  path /index.html
                  #digest 640205b7b0fc66c1ea91c463fac6334d
                  status_code 200
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    
        real_server 192.168.56.7 80 { 
            weight 1
            HTTP_GET { 
                url { 
                  path /index.html
                  #digest 640205b7b0fc66c1ea91c463fac6334c
                  status_code 200
                }
                connect_timeout 3
                nb_get_retry 3
                delay_before_retry 3
            }
        }
    }
    
    
    #启动后，生成一个网卡 ,名称为 vrrp.{virtual_router_id}
    [root@lvs-c6 ~]# ip a
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
        link/ether 08:00:27:8c:e4:d6 brd ff:ff:ff:ff:ff:ff
        inet 192.168.56.6/24 brd 192.168.56.255 scope global eth1
        inet6 fe80::a00:27ff:fe8c:e4d6/64 scope link
           valid_lft forever preferred_lft forever
    6: vrrp.51@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
        link/ether 00:00:5e:00:01:33 brd ff:ff:ff:ff:ff:ff
        inet 192.168.56.254/32 scope global vrrp.51
        inet6 fe80::200:5eff:fe00:133/64 scope link
           valid_lft forever preferred_lft forever

### 生成LVS服务 ###

*  当realserver故障时，会自动删除它；恢复后，会自动添加
 *  当keepalived-MASTER故障时，BACKUP自动切换到VIP

[root@lvs-c6 ~]# ipvsadm -ln
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    TCP  192.168.56.254:80 rr persistent 5
      -> 192.168.56.7:80              Route   1      0          0
      -> 192.168.56.8:80              Route   1      0          0

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 2]

## BACKUP配置 ##

同上： 只有以下差别:state , priority

[root@c72 ~]# cat /etc/keepalived/keepalived.conf
    ...
    vrrp_instance VI_1 { 
        state BACKUP
    	...
    	 priority 60
    ...
    
    #启动后，生成一个网卡 ,名称为 vrrp.{virtual_router_id}
    [root@c72 ~]# ip a
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 08:00:27:9b:af:24 brd ff:ff:ff:ff:ff:ff
        inet 192.168.56.77/24 brd 192.168.56.255 scope global noprefixroute eth1
           valid_lft forever preferred_lft forever
        inet6 fe80::a00:27ff:fe9b:af24/64 scope link
           valid_lft forever preferred_lft forever
    6: vrrp.51@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
        link/ether 00:00:5e:00:01:33 brd ff:ff:ff:ff:ff:ff

[http_www.linuxvirtualserver.org_whatis.html]: http://www.linuxvirtualserver.org/whatis.html
[http_www.austintek.com_LVS_LVS-HOWTO_HOWTO_LVS-HOWTO.ipvsadm.html_using_ipvsadm]: http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.ipvsadm.html#using_ipvsadm
[http_www.linuxvirtualserver.org_VS-DRouting.html]: http://www.linuxvirtualserver.org/VS-DRouting.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: /images/20221120/86bcfcbebf7a4b19a5cb28a5cdcac7f4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]: /images/20221120/d8c0fa2c70d746a38445ddd0d9751163.png
[https_www.keepalived.org_doc_introduction.html]: https://www.keepalived.org/doc/introduction.html
[http_www.austintek.com_LVS_LVS-HOWTO_HOWTO_LVS-HOWTO.failover.html_keepalived_vrrpd]: http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.failover.html#keepalived_vrrpd
[https_www.keepalived.org_doc_software_design.html_note-on-using-vrrp-with-virtual-mac-address]: https://www.keepalived.org/doc/software_design.html#note-on-using-vrrp-with-virtual-mac-address
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/202012301123330.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU=,size_16,color_FFFFFF,t_70