LVS+KeepAlived+Nginx高可用实现方案

快来打我* 2023-07-07 11:24 11阅读 0赞

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70][]

# **LVS** #

*  **什么是lvs**  
    **LVS**是**Linux Virtual Server**的简写，意即**Linux虚拟服务器**，是一个虚拟的服务器集群系统。本项目在1998年5月由章文嵩博士成立，**是中国国内最早出现的自由软件项目之一**。
 *  详细设置请参阅拙作： [LVS 负载均衡 https://blog.csdn.net/gaofei0428/article/details/104438533][LVS _ https_blog.csdn.net_gaofei0428_article_details_104438533]

# **KeepAlived** #

![format_png][]

keepalived是一个类似于layer3, 4 & 5交换机制的软件，也就是我们平时说的第3层、第4层和第5层交换。Keepalived是自动完成，不需人工干涉。Keepalived的作用是检测服务器的状态，如果有一台web服务器宕机，或工作出现故障，Keepalived将检测到，并将有故障的服务器从系统中剔除，同时使用其他服务器代替该服务器的工作，当服务器工作正常后Keepalived自动将服务器加入到服务器群中，这些工作全部自动完成，不需要人工干涉，需要人工做的只是修复故障的服务器。

keepalived主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现。

高可用web架构: LVS+keepalived+nginx+apache+php+eaccelerator（+nfs可选 可不选）

![format_png 1][]

Layer3,4,5工作在IP/TCP协议栈的IP层，TCP层，及应用层,原理分别如下：

*  Layer3：Keepalived使用Layer3的方式工作式时，Keepalived会定期向服务器群中的服务器发送一个ICMP的数据包（既我们平时用的Ping程序）,如果发现某台服务的IP地址没有激活，Keepalived便报告这台服务器失效，并将它从服务器群中剔除，这种情况的典型例子是某台服务器被非法关机。Layer3的方式是以服务器的IP地址是否有效作为服务器工作正常与否的标准。
 *  Layer4:如果您理解了Layer3的方式，Layer4就容易了。Layer4主要以TCP端口的状态来决定服务器工作正常与否。如web server的服务端口一般是80，如果Keepalived检测到80端口没有启动，则Keepalived将把这台服务器从服务器群中剔除。
 *  Layer5：Layer5对指定的URL执行HTTP GET。然后使用MD5算法对HTTP GET结果进行求和。如果这个总数与预期值不符，那么测试是错误的，服务器将从服务器池中移除。该模块对同一服务实施多URL获取检查。如果您使用承载多个应用程序服务器的服务器，则此功能很有用。此功能使您能够检查应用程序服务器是否正常工作。MD5摘要是使用genhash实用程序（包含在keepalived软件包中）生成的。

--------------------

*  下面开始 搭建 LVS+KeepAlived+Nginx 
 *  试验环境:
 *  keepalived 服务器 2台 （master + backup）
 *  nginx WEB 服务器 2台
 *  
 *  VIP 192.168.1.168
 *  keepalived master    192.168.1.2  CentOS 7.5   
 *  keepalived backup   192.168.1.3   CentOS 7.5
 *  nginx 1                     192.168.1.10  CentOS 6.5
 *  nginx 2                     192.168.1.11  CentOS 7.5

--------------------

*  ### 首先在 **192.168.1.2 ** 端安装、配置 keepalived  Master ###
 *  前提要配置好 LVS 详细设置请参阅拙作： [LVS 负载均衡 https://blog.csdn.net/gaofei0428/article/details/104438533][LVS _ https_blog.csdn.net_gaofei0428_article_details_104438533]
 *  sudo yum -y install keepalived ipvsadm libnl libnl-devel openssl-devel libnfnetlink-devel net-tools
 *  备份 keepalived 主配置文件
 *  mv /etc/keepalived/keepalived.conf /bak/keepalived.conf.bak  
    vim /etc/keepalived/keepalived.conf
 *  编辑 /etc/keepalived/keepalived.conf 添加以下内容
 *  global\_defs \{                 \#全局配置  
    \#   notification\_email \{  \#指定keepalived在发生切换时是否发送email警告  
    \#   \}  
    \#   smtp\_connect\_timeout 30  
            router\_id LVS\_DEVEL   \#此处注意router\_id为负载均衡标识，在局域网内应该是唯一的  
    \}  
    vrrp\_instance VI\_1 \{    \#虚拟路由的标识符  
            state MASTER    \#指定状态，有MASTER和BACKUP两种**要大写**，MASTER为工作状态，BACKUP是备用状态  
            interface eth0       \#指定 VIP 虚拟IP 对应的 INTERFACE 接口

lvs\_sync\_daemon\_inteface eth0 \#心跳线接口，DR模式用的和上面的接口一样，也可以用机器上的其他网卡，用来防止脑裂

virtual\_router\_id 51    \#虚拟路由的ID号，是虚拟路由MAC的最后一位地址  
        priority 100          \#此节点的优先级，主节点的优先级需要比其他节点高    
        advert\_int 1         \#通告的间隔时间  
        authentication \{    \#认证配置      
                auth\_type PASS  \#认证方式  
                auth\_pass 1111    \#认证密码  
        \}  
        virtual\_ipaddress \{   \#虚拟ip地址,可以有多个地址，每个地址占一行，不需要子网掩码，必须与 lvs 客户端设定的 vip 一致  
                192.168.1.168  
        \}  
\}  
virtual\_server 192.168.1.168 80 \{ \#集群所使用的VIP和端口  
        delay\_loop 6         \#健康检查间隔，单位为秒    
        lb\_algo wrr            \#lvs调度算法rr|wrr|lc|wlc|lblc|sh|dh  
        lb\_kind DR            \#负载均衡转发规则。一般包括DR,NAT,TUN 3种  
        nat\_mask 255.255.255.255   \#VIP掩码  
        persistence\_timeout 50      \#会话保持时间  
        protocol TCP             \#转发协议               
        real\_server 192.168.1.10 80 \{ \#真实服务器，包括IP和端口号  
                weight 1   \#默认为1,0为失效

inhibit\_on\_failure \#在服务器健康检查失效时，将其设为0，而不是直接从ipvs中删除

TCP\_CHECK \{      \#通过tcpcheck判断RealServer的健康状态                
                        connect\_timeout 10    \#连接超时时间  
                        nb\_get\_retry 3    \#重连次数  
                        delay\_before\_retry 3   \#重连间隔时间  
                        connect\_port 80   \#健康检查端口的端口  
                \}  
        \}  
        real\_server 192.168.1.11 80 \{  
                weight 1

inhibit\_on\_failure  
                TCP\_CHECK \{  
                        connect\_timeout 10  
                        nb\_get\_retry 3  
                        delay\_before\_retry 3  
                        connect\_port 80  
                \}  
        \}  
\}

*  确认无误保存退出，暂时不启动 keepalived

--------------------

*  接下来在 **192.168.1.3 **上安装、配置 keeepalived （前提要配置好 LVS）
 *  sudo yum -y install keepalived ipvsadm libnl libnl-devel openssl-devel libnfnetlink-devel net-tools
 *  备份 keepalived 主配置文件
 *  mv /etc/keepalived/keepalived.conf /bak/keepalived.conf.bak  
    vim /etc/keepalived/keepalived.conf
 *  编辑 /etc/keepalived/keepalived.conf 添加以下内容
 *  global\_defs \{                         
    \#   notification\_email \{               
    \#   \}  
    \#   smtp\_connect\_timeout 30  
            router\_id LVS\_DEVEL               
    \}  
    vrrp\_instance VI\_1 \{              
            state BACKUP    \#BACKUP是备用状态  
            interface eth0  
            lvs\_sync\_daemon\_inteface eth0  
            virtual\_router\_id 51          
            priority 99  
            nopreempt   ** \#设置为不抢占 注：这个配置只能设置在backup主机上，而且这个主机优先级要比另外一台高**  
            advert\_int 1             
            authentication \{          
                    auth\_type PASS  
                    auth\_pass 1111  
            \}  
            virtual\_ipaddress \{           
                    192.168.1.168  
            \}  
    \}  
    virtual\_server 192.168.1.168 80 \{  
            delay\_loop 6             
            lb\_algo wrr              
            lb\_kind DR  
            nat\_mask 255.255.255.255     
            persistence\_timeout 50      
            protocol TCP                            
            real\_server 192.168.1.10 80 \{  
                    weight 1  
            inhibit\_on\_failure  
                    TCP\_CHECK \{                       
                            connect\_timeout 10     
                            nb\_get\_retry 3  
                            delay\_before\_retry 3  
                            connect\_port 80  
                    \}  
            \}  
            real\_server 192.168.1.11 80 \{  
                    weight 1  
                   inhibit\_on\_failure  
            TCP\_CHECK \{  
                            connect\_timeout 10  
                            nb\_get\_retry 3  
                            delay\_before\_retry 3  
                            connect\_port 80  
                    \}  
            \}  
    \}
 *  确认无误保存退出，分别在 Master 和 Backup 端启动 keepalived 和 lvs

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 2][]

*  这里为了测试将 persistence\_timeout 50 链接时间改为 0
 *  在另外一台虚拟机上测试

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 3][]

*  模拟故障将 Master 停掉
 *  192.168.1.2 端 systemctl stop keepalived

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 4][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 5][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 7][]

--------------------

*  ### 配置 keepalived + LVS + Nginx 实现互为主从  ###
 *  实验环境：
 *  192.168.2.6     CentOS 7.6   KeepAlived + LVS   VIP : 192.168.2.168
 *  192.168.2.10   CentOS 7.6   KeepAlived + LVS   VIP : 192.168.2.169
 *  192.168.2.4    CentOS 7.6   Nginx + PHP  VIP : 192.168.2.168  +  192.168.2.169
 *  192.168.2.7    CentOS 7.6   Nginx + PHP  VIP : 192.168.2.168  +  192.168.2.169
 *  
 *  **部署思路：**
 *  **两台 KeepAlive 主机各绑定一个 VIP，同时两台主机都为主服务向外提供一组资源，同时也是互为备用服务。**
 *  
 *  **当 192.168.2.6 主服务出现宕机的情况，会把业务迁移到 192.168.2.10 主机上，此时 192.168.2.10 就会绑定两个 VIP，为两组服务提供资源  
    当 192.168.2.10 主服务出现宕机的情况，会把业务迁移到 192.168.2.6 主机上，此时 192.168.2.6 就会绑定两个 VIP，为两组服务提供资源**
 *  
 *  分别在2 台主机上安装 KeepAlive
 *  yum -y install keepalived ipvsadm libnl libnl-devel openssl-devel libnfnetlink-devel net-tools
 *  
 *  systemctl enable keepalived.service ipvsadm.service
 *  
 *  备份 keepalived 主配置文件
 *  cp /etc/keepalived/keepalived.conf\{,.bak\}
 *  编辑 192.168.2.6 上的  /etc/keepalived/keepalived.conf  内容如下：
 *  
 *  **vim /etc/keepalived/keepalived.conf    **                                                                                                  
    ! Configuration File for keepalived
    
    global\_defs \{  
       \#notification\_email \{  
        \# acassen@firewall.loc  
        \# failover@firewall.loc  
        \# sysadmin@firewall.loc  
       \#\}  
       \#notification\_email\_from Alexandre.Cassen@firewall.loc  
       \#smtp\_server 192.168.200.1  
       \#smtp\_connect\_timeout 30  
       router\_id **LVS\_DEVEL**  
       vrrp\_skip\_check\_adv\_addr        \#默认是不跳过检查。检查收到的VRRP通告中的所有地址可能会比较耗时，设置此命令的意思是，如果通告与接收的上一个通告来自相同的master路由器，则不执行检查(跳过检查)。  
       \#vrrp\_strict                                       \#禁止 vrrp 协议强制检查  
       vrrp\_garp\_interval 0.001        \#在一个接口发送的两个免费ARP之间的延迟。可以精确到毫秒级。默认是0.  
       vrrp\_gna\_interval 0  
    \}
    
    vrrp\_instance **DR\_1**\{           \#定义第一组虚拟 VIP   
        **state MASTER                 \#一定要大写**  
        interface ens33                \#注意网卡的名称  
        virtual\_router\_id 51          \#设置组ID,同一组 vrrp\_instance 必须一致  
        lvs\_sync\_daemon\_inteface ens33  
        **priority 100                      \#设置优先级，MASTER 要比 BACKUP 的值大，互为主从时注意另一台的优先级，比如：192.168.2.10 的优先级就要比这个值小，90**  
        advert\_int 1                      \#检测时间  
        authentication \{  
            auth\_type PASS  
            auth\_pass 1111  
        \}  
        virtual\_ipaddress \{              **\#第一组 VIP 的地址**  
            **192.168.2.168**  
        \}  
    \}
    
    vrrp\_instance **DR\_2** \{    \#定义第二组虚拟 VIP  
        **state BACKUP          \#一定要大写，互为主从时，注意名称和优先级**  
        interface ens33  
        **virtual\_router\_id 53   \#注意组 id**  
        lvs\_sync\_daemon\_inteface ens33  
        **priority 100 **   \#**设置优先级，MASTER 要比 BACKUP 的值大，互为主从时注意另一台的优先级，比如：192.168.2.10 的优先级就要比这个值大，为110**  
        advert\_int 1  
        authentication \{  
            auth\_type PASS  
            auth\_pass 1111  
        \}  
        virtual\_ipaddress \{    **\#第二组 VIP 的地址**  
            **192.168.2.169**  
        \}  
    \}
    
    **\#后端真实服务器 rs 设置**  
    virtual\_server 192.168.2.168 80 \{  
        delay\_loop 6  
        lb\_algo rr  
        lb\_kind DR  
        \#persistence\_timeout 50  
        protocol TCP
    
        real\_server 192.168.2.4 80 \{  
            weight 1  
        inhibit\_on\_failure  
        TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
            connect\_port 80      
            \}  
        \}      
        real\_server 192.168.2.7 80 \{  
            weight 1  
        inhibit\_on\_failure  
            TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
                connect\_port 80  
            \}  
        \}  
    \}
    
    virtual\_server 192.168.2.169 80 \{  
        delay\_loop 6  
        lb\_algo rr  
        lb\_kind DR  
        \#persistence\_timeout 50  
        protocol TCP
    
        real\_server 192.168.2.4 80 \{  
            weight 1  
        inhibit\_on\_failure  
            TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
                connect\_port 80  
            \}  
        \}     
        real\_server 192.168.2.7 80 \{  
            weight 1  
        inhibit\_on\_failure  
            TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
                connect\_port 80  
            \}  
        \}  
    \}
 *  ### **修改完暂时不启动 keepalive** ###
 *  ### **将 keepalive 配置文件 scp 到 192.168.2.10** ###
 *  ### **然后修改如下** ###
 *  /etc/keepalived/keepalived.conf  
    ! Configuration File for keepalived
    
    global\_defs \{  
       \#notification\_email \{  
        \# acassen@firewall.loc  
        \# failover@firewall.loc  
        \# sysadmin@firewall.loc  
       \#\}  
       \#notification\_email\_from Alexandre.Cassen@firewall.loc  
       \#smtp\_server 192.168.200.1  
       \#smtp\_connect\_timeout 30  
       router\_id LVS\_DEVEL  
       \#vrrp\_skip\_check\_adv\_addr  
       \#vrrp\_strict  
       \#vrrp\_garp\_interval 0.001  
       \#vrrp\_gna\_interval 0  
    \}
    
    **vrrp\_instance DR\_1 \{  
        state BACKUP      \#第一组 作为 备用**  
        interface ens33  
        virtual\_router\_id 51  
        lvs\_sync\_daemon\_inteface ens33  
        **priority 90        \#优先级比 MASTER 192.168.2.6 的小**  
        advert\_int 1  
        authentication \{  
            auth\_type PASS  
            auth\_pass 1111  
        \}  
        virtual\_ipaddress \{  
            192.168.2.168  
        \}  
    \}
    
    **vrrp\_instance DR\_2 \{  
        state MASTER       \#第二组 要作为 MASTER**  
        interface ens33  
        virtual\_router\_id 53  
        lvs\_sync\_daemon\_inteface ens33  
    **    priority 110             \#优先级要比 192.168.2.6 的大**  
        advert\_int 1  
        authentication \{  
            auth\_type PASS  
            auth\_pass 1111  
        \}  
        virtual\_ipaddress \{  
            192.168.2.169  
        \}  
    \}
    
    **\#其他设置不用动**  
    virtual\_server 192.168.2.168 80 \{  
        delay\_loop 6  
        lb\_algo rr  
        lb\_kind DR  
        \#persistence\_timeout 50  
        protocol TCP
    
        real\_server 192.168.2.4 80 \{  
            weight 1  
        inhibit\_on\_failure  
        TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
            connect\_port 80      
            \}  
        \}      
        real\_server 192.168.2.7 80 \{  
            weight 1  
        inhibit\_on\_failure  
            TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
                connect\_port 80  
            \}  
        \}  
    \}
    
    virtual\_server 192.168.2.169 80 \{  
        delay\_loop 6  
        lb\_algo rr  
        lb\_kind DR  
        \#persistence\_timeout 50  
        protocol TCP
    
        real\_server 192.168.2.4 80 \{  
            weight 1  
        inhibit\_on\_failure  
            TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
                connect\_port 80  
            \}  
        \}     
        real\_server 192.168.2.7 80 \{  
            weight 1  
        inhibit\_on\_failure  
            TCP\_CHECK \{  
                connect\_timeout 3  
                nb\_get\_retry 3  
                delay\_before\_retry 3  
                connect\_port 80  
            \}  
        \}  
    \}
 *  修改完在 RS 上添加 虚拟 VIP

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 9][]

*  依次重启 192.168.2.6 和 192.168.2.10 的  keepalive 服务
 *  
 *  systemctl restart keepalived.service

![20210707104822145.png][]

![20210707105014320.png][]

*  测试
 *  打开另外一台机器分别访问 192.168.2.168 和 2.169

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 10][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 11][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 12][]

*  多访问几次观察 Conns 记录

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 13][]

![20210707105310289.png][]

*  模拟故障
 *  首先停止 192.168.2.6 的 keepalive

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 14][]

*  观察 192.168.2.10

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 15][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 16][]

![20210707105713154.png][]

*  成功漂移
 *  下面停止 192.168.2.10  后 打开 192.168.2..6

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 17][]

![2021070710583688.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 18][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 19][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 20][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 21][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 22][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 23][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 24][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 25][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 26][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 27][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20210706200132479.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[LVS _ https_blog.csdn.net_gaofei0428_article_details_104438533]: https://blog.csdn.net/gaofei0428/article/details/104438533
[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly90aW1nc2EuYmFpZHUuY29tL3RpbWc_aW1hZ2UmcXVhbGl0eT04MCZzaXplPWI5OTk5XzEwMDAwJnNlYz0xNTgyNDMxODg3ODg3JmRpPWQ2YmNjZmM1ZWE3ZTgzNmRiMjE3NmYzZDg5OTU1MTdiJmltZ3R5cGU9anBnJnNyYz1odHRwJTNBJTJGJTJGaW1nMi5pbWd0bi5iZGltZy5jb20lMkZpdCUyRnUlM0QxNjYzNjU4NzIlMkMxNzkzMjk2MTc1JTI2Zm0lM0QyMTQlMjZncCUzRDAuanBn?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9zczMuYmRzdGF0aWMuY29tLzcwY0Z2OFNoX1ExWW54R2twb1dLMUhGNmhoeS9pdC91PTI4OTQ1MjEwNjYsMTczNzQ0NTQ5JmZtPTI2JmdwPTAuanBn?x-oss-process=image/format,png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200224122916937.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020022412322395.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200224124701732.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/2020022412484166.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/2020022412532861.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/2020022412503587.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200224125144815.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20210707104643612.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20210707104622771.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[20210707104822145.png]: https://img-blog.csdnimg.cn/20210707104822145.png
[20210707105014320.png]: https://img-blog.csdnimg.cn/20210707105014320.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20210707105116624.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20210707105140665.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20210707105159574.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20210707105257535.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[20210707105310289.png]: https://img-blog.csdnimg.cn/20210707105310289.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20210707105523159.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20210707105626618.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20210707105644434.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[20210707105713154.png]: https://img-blog.csdnimg.cn/20210707105713154.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20210707105809888.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[2021070710583688.png]: https://img-blog.csdnimg.cn/2021070710583688.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/2021070710590170.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20210707105925437.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20210707110022618.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/20210707110054639.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20210707110126544.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 23]: https://img-blog.csdnimg.cn/20210707110139446.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 24]: https://img-blog.csdnimg.cn/2021070711032663.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 25]: https://img-blog.csdnimg.cn/20210707110453437.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 26]: https://img-blog.csdnimg.cn/20210707110358303.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg_size_16_color_FFFFFF_t_70 27]: https://img-blog.csdnimg.cn/20210707110437965.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2ZlaTA0Mjg=,size_16,color_FFFFFF,t_70