Lvs+keepalived高可用集群(DR) 不念不忘少年蓝@ 2021-07-26 18:24 434阅读 0赞 ### lvs+keepalived ### * 一.lvs+keepalived高可用集群 * * 1.LVS * 2.keepalived * 3.Keepalived实现原理剖析 * 二.LVS+keepalived高可用集群部署 * 1.部署NFS共享服务器 * 2.配置节点服务器 * * 1)配置虚拟IP地址(VIP:192.168.133.100) * 2)调整内核的ARP响应参数 * 3)编辑网页主页 * 3.配置负载调度器 * 1)关闭防火墙,加载ip\_vs模块 * 2)调整proc响应参数 * * 3)配置负载分配策略 * 4)配置keeplived * * 主负载调度器: * 备负载调度器: * 4.客户机访问 * * 将主调度服务器keepalived.service 服务关闭,再测试 * 总结 # 一.lvs+keepalived高可用集群 # ## 1.LVS ## linux虚拟服务器 LVS实际上相当于基于IP地址的虚拟化应用 为基于IP地址和内容请求分发的负载均衡提出了一种高效的解决办法 ## 2.keepalived ## 专为LVS和HA设计的一款健康检查工具 支持故障自动切换 支持节点健康状态检查 * 判断LVS负载调度器、节点服务器的可用性,当master主机出现故障及时切换到backup节点保证业务正常,当 master 故障主机恢复后将其重新加入群集并且业务重新切换回 master 节点。 ## 3.Keepalived实现原理剖析 ## * keepalived采用VRRP热备份协议实现Linux 服务器的多机热备功能 * VRRP(虚拟路由冗余协议)是针对路由器的一种备份解决方案。 * 由多台路由器组成一个热备份组,通过共用的虚拟IP地址对外提供服务 * 每个热备组内同时只有一台主路由器提供服务,其他路由器处于冗余状态 * 若当前在线的路由器失效,则其他路由器会根据设置的优先级自动接替虚拟IP地址,继续提供服务 ## 二.LVS+keepalived高可用集群部署 ## 环境配置: <table> <thead> <tr> <th>主机</th> <th>操作系统</th> <th>IP地址</th> <th>软件/安装包工具</th> </tr> </thead> <tbody> <tr> <td>主负载调度器(Director1)</td> <td>centos7</td> <td>ens33:192.168.133.10</td> <td>ipvsadm、keepalived</td> </tr> <tr> <td>备负载调度器(Director2)</td> <td>centos7</td> <td>ens33:192.168.133.20</td> <td>ipvsadm、keepalived</td> </tr> <tr> <td>web节点服务器1</td> <td>centos7</td> <td>ens33:192.168.133.30 lo:0(VIP):192.168.133.100</td> <td>rpcbind、nfs-utils、httpd</td> </tr> <tr> <td>web节点服务器2</td> <td>centos7</td> <td>ens33:192.168.133.40 lo:0(VIP):192.168.133.100</td> <td>rpcbind、nfs-utils、httpd</td> </tr> <tr> <td>NFS服务器</td> <td>centos7</td> <td>192.168.133.50</td> <td>rpcbind、nfs-utils</td> </tr> <tr> <td>客户机</td> <td>win10</td> <td>192.168.133.60</td> <td></td> </tr> </tbody> </table> ## 1.部署NFS共享服务器 ## NFS服务器:192.168.133.50 systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 yum -y install nfs-utils rpcbind systemctl start rpcbind.service systemctl start nfs.service systemctl enable nfs.service systemctl enable rpcbind.service mkdir /opt/test1 mkdir /opt/test2 chmod 777 /opt/test1 chmod 777 /opt/test2 vim /etc/exports /opt/test1 192.168.163.0/24(rw,sync) /opt/test2 192.168.163.0/24(rw,sync) exportfs -rv ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 1] ![在这里插入图片描述][20210426152207835.png_pic_center] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 2]![在这里插入图片描述][20210426152448456.png_pic_center] ## 2.配置节点服务器 ## web节点服务器1:192.168.133.30 lo:0(VIP):192.168.133.100 web节点服务器1:192.168.133.40 lo:0(VIP):192.168.133.100 systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 3] ### 1)配置虚拟IP地址(VIP:192.168.133.100) ### vim /etc/sysconfig/network-scripts/ifcfg-lo:0 DEVICE=lo:0 IPADDR=192.168.133.100 NETMASK=255.255.255.255 ONBOOT=yes ifup lo:0 ifconfig lo:0 #设置临时的路由,重启失效 route add -host 192.168.133.100 dev lo:0 #开机自动添加路由 vim /etc/rc.local /sbin/route add -host 192.168.133.100 dev lo:0 chmod +x /etc/rc.d/rc.local ![在这里插入图片描述][20210426153736337.png_pic_center] ![在这里插入图片描述][20210426153919704.png_pic_center]![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 4] ### 2)调整内核的ARP响应参数 ### * 阻止更新 VIP 的MAC 地址,避免发生冲突 vim /etc/sysctl.conf ...... net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 sysctl -p yum install -y nfs-utils rpcbind httpd systemctl start rpcbind systemctl enable rpcbind systemctl start httpd.service systemctl enable httpd.service ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 5] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 6]![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 7] ### 3)编辑网页主页 ### * web节点服务器1:192.168.133.30 showmount -e 192.168.133.50 mount.nfs 192.168.133.50:/opt/test1 /var/www/html echo 'this is test1 web!' > /var/www/html/index.html ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 8] * web节点服务器2:192.168.133.40 showmount -e 192.168.133.50 mount.nfs 192.168.133.50:/opt/test2 /var/www/html echo 'this is test2 web!' > /var/www/html/index.html ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 9] ## 3.配置负载调度器 ## * 主负载调度器:192.168.133.10 * 备负载调度器: 192.168.133.20 ## 1)关闭防火墙,加载ip\_vs模块 ## systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 modprobe ip_vs cat /proc/net/ip_vs yum -y install ipvsadm ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 10] ## 2)调整proc响应参数 ## vim /etc/sysctl.conf net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0 sysctl -p ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 11] ![在这里插入图片描述][20210426162602777.png_pic_center] ### 3)配置负载分配策略 ### * 主负载调度器:192.168.133.10 ipvsadm-save > /etc/sysconfig/ipvsadm systemctl start ipvsadm.service ipvsadm -C #这里-t指定的虚拟地址为本机ens33网卡ip ipvsadm -A -t 192.168.133.10:80 -s rr ipvsadm -a -t 192.168.133.10:80 -r 192.168.133.30:80 -g ipvsadm -a -t 192.168.133.10:80 -r 192.168.133.40:80 -g ipvsadm ipvsadm -ln ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 12] * 备负载调度器:192.168.133.20 ipvsadm-save > /etc/sysconfig/ipvsadm systemctl start ipvsadm.service ipvsadm -C ipvsadm -A -t 192.168.133.20:80 -s rr ipvsadm -a -t 192.168.133.20:80 -r 192.168.163.30:80 -g ipvsadm -a -t 192.168.133.20:80 -r 192.168.163.40:80 -g ipvsadm -ln ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 13] ## 4)配置keeplived ## * 负载调度器1:192.168.133.10 * 负载调度器2:192.168.133.20 yum -y install keepalived cd /etc/keepalived/ cp keepalived.conf keepalived.conf.bak ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 14] ![在这里插入图片描述][20210426165816564.png_pic_center] vim keepalived.conf ...... #定义全局参数 global_defs { #10行修改,邮件服务指向本地 smtp_server 127.0.0.1 #12行修改,指定服务器(路由器)的名称,主备服务器名称须不同,主为LVS_01,备为LVS_02 router_id LVS_01 #14行修改;加注释;vrrp_strict:严格遵守VRRP协议。下列情况将会阻止启动Keepalived:1. 没有VIP地址。2. 单播邻居。3. 在VRRP版本2中有IPv6地址。 #vrrp_strict } #定义VRRP热备实例参数 vrrp_instance VI_1 { #20行修改;指定热备状态,主为MASTER,备为BACKUP state MASTER #21行修改;指定承载vip地址的物理接口 interface ens33 #22行修改;指定虚拟路由器的ID号,每个热备组保持一致 virtual_router_id 10 #23行修改;指定优先级,数值越大优先级越高,主为100,备为90 priority 100 #通告间隔秒数(心跳频率) advert_int 1 #定义认证信息,每个热备组保持一致 authentication { #认证类型 auth_type PASS #27行修改,指定验证密码,主备服务器保持一致 auth_pass 123456 } #指定群集vip地址 virtual_ipaddress { 192.168.133.100 } } #34行修改,指定虚拟服务器地址(VIP)、端口,定义虚拟服务器和Web服务器池参数 virtual_server 192.168.133.100 80 { #健康检查的间隔时间(秒) delay_loop 6 #指定调度算法,轮询(rr) lb_algo rr #37行修改,指定群集工作模式,直接路由(DR) lb_kind DR #连接保持时间(秒) persistence_timeout 50 #应用服务采用的是TCP协议 protocol TCP #41行修改,指定第一个Web节点的地址、端口 real_server 192.168.133.30 80 { #节点的权重 weight 1 #43行删除,添加以下健康检查方式 TCP_CHECK { #添加检查的目标端口 connect_port 80 #添加连接超时(秒) connect_timeout 3 #添加重试次数 nb_get_retry 3 #添加重试间隔 delay_before_retry 4 } } #添加第二个 Web节点的地址、端口 real_server 192.168.133.40 80 { weight 1 TCP_CHECK { connect_port 80 connect_timeout 3 nb_get_retry 3 delay_before_retry 4 } } } #删除后面多余的配置 systemctl start keepalived #查看虚拟网卡vip ip addr show dev ens33 ### 主负载调度器: ### ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 15] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 16] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 17] ### 备负载调度器: ### ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 18] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 19] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 20] ## 4.客户机访问 ## 访问192.168.133.100,隔一段时间进行刷新 轮询显示 ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 21] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 22] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 23] ### 将主调度服务器keepalived.service 服务关闭,再测试 ### ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 24] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 25] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 26] ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 27] ### 总结 ### lvs:linux虚拟服务器 keepalived:为lvs和HA健康检查工具 支持故障自动切换 健康状态检查 * keepalived:VRRP热备份协议实现linux服务器多机热备份 * 多台路由器组成热备份组,公用虚拟IP对外提供服务 * 当前主路由器失效,其他路由器会根据设置优先级自动接替虚拟IP地址,继续提供服务 [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center]: /images/20210726/33d44a82910247d2b333b043cd52f0cf.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20210726/30cf4e23d4de4b818187d1ffe624a8c3.png [20210426152207835.png_pic_center]: /images/20210726/5d90617d208a4d89a20d440983e16b13.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/20210426152357937.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp,size_16,color_FFFFFF,t_70#pic_center [20210426152448456.png_pic_center]: /images/20210726/3052f53ce4634fdd9c2064348a1f45e0.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20210726/c371548887de49f89e01b744517859d1.png [20210426153736337.png_pic_center]: /images/20210726/991daff84787405ab22587726524966a.png [20210426153919704.png_pic_center]: https://img-blog.csdnimg.cn/20210426153919704.png#pic_center [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20210726/9a21e1fdadab477a8a8b94a46488027c.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 5]: /images/20210726/3d74fd519295473ca527ce0187b36dd0.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 6]: https://img-blog.csdnimg.cn/20210426155528674.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp,size_16,color_FFFFFF,t_70#pic_center [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 7]: /images/20210726/4cfe54e48a784e1cb75161eeaf5e52e9.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 8]: /images/20210726/6972c3ce3c0c40e0b3a23f05fbbffb49.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 9]: /images/20210726/8e80edc598664431a95ab92cd2b2d46e.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 10]: /images/20210726/c0d696d08e0d471da0201377c5edb6a0.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 11]: /images/20210726/29be3c2635b94a9481e826f7ee544bd3.png [20210426162602777.png_pic_center]: /images/20210726/71f993d3cab7458b8dfa415085c9d75b.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 12]: /images/20210726/0ab58d83f33845859e1fcebcc9e3a90a.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 13]: /images/20210726/e6044cfaff3c46fcbec25ef295382679.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 14]: /images/20210726/329dd166c8d54f8aa3f4c8d487ae9ca4.png [20210426165816564.png_pic_center]: /images/20210726/9fdfa3e9e3524d1295f84be74e9b6d72.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 15]: /images/20210726/ea2e1d84c55e4aaaace4c15561cf5506.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 16]: /images/20210726/5ed18fa29e9f4b4da3e649a499dbcadd.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 17]: /images/20210726/5206c3f92a77432e86c26e42c8254810.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 18]: /images/20210726/400e2c1cc3764cc397294bc3792ee25a.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 19]: /images/20210726/6b7573bac59b4382bac5a9921de2b5d1.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 20]: /images/20210726/dee4e1e5077a48678be76b4b7f07f6c5.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 21]: /images/20210726/bb68470fee72480d8874b01e6e9a1a3e.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 22]: /images/20210726/b0ff5b60b84a4e598588bbd00a907bfd.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 23]: /images/20210726/48b7d13f15294835a1882db54b3c4fde.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 24]: /images/20210726/abf54165104c4c0e9213546f046b0a22.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 25]: /images/20210726/eee962df40fa4974a7fe0b77c86f1e81.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 26]: /images/20210726/3cb5b842aad445eb822e6b9b567b78f8.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1dzeHlp_size_16_color_FFFFFF_t_70_pic_center 27]: /images/20210726/dfa6c28907a64761a27c6edf0296612a.png
还没有评论,来说两句吧...