高可用性之Keepalived+Mysql

傷城~ 2022-06-07 07:28 190阅读 0赞

MySQL高可用性之Keepalived+Mysql（双主热备） 2014-02-23 11:00:07

标签： [mysql][] [ 高可用][Link 1] [ msql双机热备][msql]

原创作品，允许转载，转载时请务必以超链接形式标明文章 [ 原始出处][Link 2] 、作者信息和本声明。否则将追究法律责任。 [http://lizhenliang.blog.51cto.com/7876557/1362313][Link 2]

**环境描述：**

OS：CentOS6.5\_X64

MASTER：192.168.0.202

BACKUP：192.168.0.203

VIP：192.168.0.204

1、配置两台Mysql主主同步

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> 
    <div>
      6 
    </div> 
    <div>
      7 
    </div> 
    <div>
      8 
    </div> 
    <div>
      9 
    </div> 
    <div>
      10 
    </div> 
    <div>
      11 
    </div> 
    <div>
      12 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;yum&nbsp;install&nbsp;mysql-server&nbsp;mysql&nbsp;-y</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;service&nbsp;mysqld&nbsp;start</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;mysqladmin&nbsp;-u&nbsp;root&nbsp;password&nbsp;123.com</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;vi&nbsp;/etc/my.cnf&nbsp;&nbsp;#开启二进制日志，设置id</code> 
     </div> 
     <div> 
      <code>[mysqld]</code> 
     </div> 
     <div> 
      <code>server-</code> 
      <code>id</code>&nbsp; 
      <code>=&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#backup这台设置2</code> 
     </div> 
     <div> 
      <code>log-bin&nbsp;=&nbsp;mysql-bin</code> 
     </div> 
     <div> 
      <code>binlog-ignore-db&nbsp;=&nbsp;mysql,information_schema&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#忽略写入binlog日志的库</code> 
     </div> 
     <div> 
      <code>auto-increment-increment&nbsp;=&nbsp;2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#字段变化增量值</code> 
     </div> 
     <div> 
      <code>auto-increment-offset&nbsp;=&nbsp;1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#初始字段ID为1</code> 
     </div> 
     <div> 
      <code>slave-skip-errors&nbsp;=&nbsp;all&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#忽略所有复制产生的错误&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;service&nbsp;mysqld&nbsp;restart</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**\#先查看下log bin日志和pos值位置**

[![wKiom1MJYSngwysqAADkFn9zcK8686.jpg][]][wKiom1MJYSngwysqAADkFn9zcK8686.jpg]

**master配置如下：**

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> 
    <div>
      6 
    </div> 
    <div>
      7 
    </div> 
    <div>
      8 
    </div> 
    <div>
      9 
    </div> 
    <div>
      10 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@&nbsp;master&nbsp;~]</code> 
      <code>#&nbsp;mysql&nbsp;-u&nbsp;root&nbsp;-p123.com</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;GRANT&nbsp;&nbsp;REPLICATION&nbsp;SLAVE&nbsp;ON&nbsp;*.*&nbsp;TO&nbsp;</code> 
      <code>'replication'</code> 
      <code>@</code> 
      <code>'192.168.0.%'</code>&nbsp; 
      <code>IDENTIFIED&nbsp;&nbsp;BY&nbsp;</code> 
      <code>'replication'</code> 
      <code>;</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;flush&nbsp;&nbsp;privileges;</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;change&nbsp;&nbsp;master&nbsp;to</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_host=</code> 
      <code>'192.168.0.203'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_user=</code> 
      <code>'replication'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_password=</code> 
      <code>'replication'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_log_file=</code> 
      <code>'mysql-bin.000002'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_log_pos=106;&nbsp;&nbsp;</code> 
      <code>#对端状态显示的值</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;start&nbsp;&nbsp;slave;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#启动同步</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**backup配置如下：**

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> 
    <div>
      6 
    </div> 
    <div>
      7 
    </div> 
    <div>
      8 
    </div> 
    <div>
      9 
    </div> 
    <div>
      10 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@backup&nbsp;~]</code> 
      <code>#&nbsp;&nbsp;mysql&nbsp;-u&nbsp;root&nbsp;-p123.com</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;GRANT&nbsp;&nbsp;REPLICATION&nbsp;SLAVE&nbsp;ON&nbsp;*.*&nbsp;TO&nbsp;</code> 
      <code>'replication'</code> 
      <code>@</code> 
      <code>'192.168.0.%'</code>&nbsp; 
      <code>IDENTIFIED&nbsp;&nbsp;BY&nbsp;</code> 
      <code>'replication'</code> 
      <code>;</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;flush&nbsp;&nbsp;privileges;</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;change&nbsp;&nbsp;master&nbsp;to</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_host=</code> 
      <code>'192.168.0.202'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_user=</code> 
      <code>'replication'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_password=</code> 
      <code>'replication'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_log_file=</code> 
      <code>'mysql-bin.000002'</code> 
      <code>,</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>-&gt;&nbsp;&nbsp;master_log_pos=106;</code> 
     </div> 
     <div> 
      <code>mysql&gt;&nbsp;start&nbsp;&nbsp;slave;</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**\#主主同步配置完毕，查看同步状态Slave\_IO和Slave\_SQL是YES说明主主同步成功。**

[![wKioL1MJYQ7QfZfXAAGQt0H1o1c742.jpg][]][wKioL1MJYQ7QfZfXAAGQt0H1o1c742.jpg]

在master插入数据测试下：

[![wKiom1MJYUCC0nlQAAEk4ruZ3ys652.jpg][]][wKiom1MJYUCC0nlQAAEk4ruZ3ys652.jpg]

在backup查看是否同步成功：

[![wKioL1MJYXyyChXUAADPZraUk3Y684.jpg][]][wKioL1MJYXyyChXUAADPZraUk3Y684.jpg]

可以看到已经成功同步过去，同样在backup插入到user表数据，一样同步过去，双主就做成功了。

2、配置keepalived实现热备

**\[root@backup ~\]\# yum install -y pcre-devel openssl-devel popt-devel \#安装依赖包**

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;wget&nbsp;http://www.keepalived.org/software/keepalived-1.2.7.tar.gz</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;tar&nbsp;zxvf&nbsp;keepalived-1.2.7.tar.gz</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;cd&nbsp;keepalived-1.2.7</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#./configure&nbsp;--prefix=/usr/local/keepalived</code> 
     </div> 
     <div> 
      <code>make</code>&nbsp; 
      <code>&amp;&amp;&nbsp;</code> 
      <code>make</code>&nbsp; 
      <code>install</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**\#将keepalived配置成系统服务**

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;cp&nbsp;/usr/local/keepalived/etc/rc.d/init.d/keepalived&nbsp;/etc/init.d/</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;cp&nbsp;/usr/local/keepalived/etc/sysconfig/keepalived&nbsp;/etc/sysconfig/</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;mkdir&nbsp;/etc/keepalived/</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;cp&nbsp;/usr/local/keepalived/etc/keepalived/keepalived.conf&nbsp;/etc/keepalived/</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;cp&nbsp;/usr/local/keepalived/sbin/keepalived&nbsp;/usr/sbin/</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> 
    <div>
      6 
    </div> 
    <div>
      7 
    </div> 
    <div>
      8 
    </div> 
    <div>
      9 
    </div> 
    <div>
      10 
    </div> 
    <div>
      11 
    </div> 
    <div>
      12 
    </div> 
    <div>
      13 
    </div> 
    <div>
      14 
    </div> 
    <div>
      15 
    </div> 
    <div>
      16 
    </div> 
    <div>
      17 
    </div> 
    <div>
      18 
    </div> 
    <div>
      19 
    </div> 
    <div>
      20 
    </div> 
    <div>
      21 
    </div> 
    <div>
      22 
    </div> 
    <div>
      23 
    </div> 
    <div>
      24 
    </div> 
    <div>
      25 
    </div> 
    <div>
      26 
    </div> 
    <div>
      27 
    </div> 
    <div>
      28 
    </div> 
    <div>
      29 
    </div> 
    <div>
      30 
    </div> 
    <div>
      31 
    </div> 
    <div>
      32 
    </div> 
    <div>
      33 
    </div> 
    <div>
      34 
    </div> 
    <div>
      35 
    </div> 
    <div>
      36 
    </div> 
    <div>
      37 
    </div> 
    <div>
      38 
    </div> 
    <div>
      39 
    </div> 
    <div>
      40 
    </div> 
    <div>
      41 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;vi&nbsp;/etc/keepalived/keepalived.conf</code> 
     </div> 
     <div> 
      <code>!&nbsp;Configuration&nbsp;File&nbsp;forkeepalived</code> 
     </div> 
     <div> 
      <code>global_defs&nbsp;{ </code> 
     </div> 
     <div> 
      <code>notification_email&nbsp;{ </code> 
     </div> 
     <div> 
      <code>test</code> 
      <code>@sina.com</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>}</code> 
     </div> 
     <div> 
      <code>notification_email_from&nbsp;&nbsp;admin@</code> 
      <code>test</code> 
      <code>.com</code> 
     </div> 
     <div> 
      <code>smtp_server&nbsp;127.0.0.1</code> 
     </div> 
     <div> 
      <code>smtp_connect_timeout&nbsp;30</code> 
     </div> 
     <div> 
      <code>router_id&nbsp;MYSQL_HA&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#标识，双主相同</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>}</code> 
     </div> 
     <div> 
      <code>vrrp_instance&nbsp;VI_1&nbsp;{ </code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>state&nbsp;BACKUP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#两台都设置BACKUP</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>interface&nbsp;eth0</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>virtual_router_id&nbsp;51&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#主备相同</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>priority&nbsp;100&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#优先级，backup设置90</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>advert_int&nbsp;1</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>nopreempt&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#不主动抢占资源，只在master这台优先级高的设置，backup不设置</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>authentication&nbsp;{ </code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>auth_type&nbsp;PASS</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>auth_pass&nbsp;1111</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>}</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>virtual_ipaddress&nbsp;{ </code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>192.168.0.204</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>}</code> 
     </div> 
     <div> 
      <code>}</code> 
     </div> 
     <div> 
      <code>virtual_server&nbsp;192.168.0.204&nbsp;3306&nbsp;{ </code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>delay_loop&nbsp;2</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>#lb_algo&nbsp;rr&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#LVS算法，用不到，我们就关闭了</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>#lb_kind&nbsp;DR&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#LVS模式，如果不关闭，备用服务器不能通过VIP连接主MySQL</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>persistence_timeout&nbsp;50&nbsp;&nbsp;</code> 
      <code>#同一IP的连接60秒内被分配到同一台真实服务器</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>protocol&nbsp;TCP</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>real_server&nbsp;192.168.0.202&nbsp;3306&nbsp;{&nbsp;&nbsp;&nbsp;</code> 
      <code>#检测本地mysql，backup也要写检测本地mysql</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>weight&nbsp;3</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>notify_down&nbsp;</code> 
      <code>/usr/local/keepalived/mysql</code> 
      <code>.sh&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#当mysq服down时，执行此脚本，杀死keepalived实现切换</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>TCP_CHECK&nbsp;{ </code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>connect_timeout&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#连接超时</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>nb_get_retry&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code>#重试次数</code> 
     </div> 
     <div> 
      <code>&nbsp;</code> 
      <code>delay_before_retry&nbsp;3&nbsp;</code> 
      <code>#重试间隔时间</code> 
     </div> 
     <div> 
      <code>&nbsp;&nbsp;</code> 
      <code>}</code> 
     </div> 
     <div> 
      <code>}</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div>
      1 
    </div> 
    <div>
      2 
    </div> 
    <div>
      3 
    </div> 
    <div>
      4 
    </div> 
    <div>
      5 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;vi&nbsp;/usr/local/keepalived/mysql.sh</code> 
     </div> 
     <div> 
      <code>#!/bin/bash</code> 
     </div> 
     <div> 
      <code>pkill&nbsp;keepalived</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;chmod&nbsp;+x&nbsp;/usr/local/keepalived/mysql.sh</code> 
     </div> 
     <div> 
      <code>[root@master&nbsp;~]</code> 
      <code>#&nbsp;/etc/init.d/keepalived&nbsp;start</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

**\#backup服务器只修改priority为90、nopreempt不设置、real\_server设置本地IP。**

**\#授权两台Mysql服务器允许root远程登录，用于在其他服务器登陆测试！**

mysql> grant all on \*.\* to'root'@'192.168.0.%' identified by '123.com';

mysql> flush privileges;

3、测试高可用性

1、通过Mysql客户端通过VIP连接，看是否连接成功。

2、停止master这台mysql服务，是否能正常切换过去，可通过ip addr命令来查看VIP在哪台服务器上。

[![wKiom1MJYbCBORSGAAHChWpI93k009.jpg][]][wKiom1MJYbCBORSGAAHChWpI93k009.jpg]

3、可通过查看/var/log/messges日志，看出主备切换过程

4、master服务器故障恢复后，是否主动抢占资源，成为活动服务器。

本文出自 “[李振良的技术博客][Link 3]” 博客，请务必保留此出处[http://lizhenliang.blog.51cto.com/7876557/1362313][Link 2]

[mysql]: http://blog.51cto.com/tag-mysql.html
[Link 1]: http://blog.51cto.com/tag-高可用.html
[msql]: http://blog.51cto.com/tag-msql双机热备.html
[Link 2]: http://lizhenliang.blog.51cto.com/7876557/1362313
[wKiom1MJYSngwysqAADkFn9zcK8686.jpg]: /images/20220607/904e2ec280854feaab21bec62aed0cfe.png
[wKioL1MJYQ7QfZfXAAGQt0H1o1c742.jpg]: /images/20220607/2edd53fa384647118ed6a680a0b110a6.png
[wKiom1MJYUCC0nlQAAEk4ruZ3ys652.jpg]: /images/20220607/5ac6dcdcbdf84400928b08132c790737.png
[wKioL1MJYXyyChXUAADPZraUk3Y684.jpg]: /images/20220607/f82aa96dcbfc4117ba58fd5b360447f7.png
[wKiom1MJYbCBORSGAAHChWpI93k009.jpg]: /images/20220607/777ebed6eab04e1584ce1995012b1357.png
[Link 3]: http://lizhenliang.blog.51cto.com