Linux云计算架构-zabbix企业级监控部署及使用(3)-zabbix的具体使用(含监控apache、mysql)

一时失言乱红尘 2023-02-21 14:13 4阅读 0赞

### 文章目录 ###

*  Linux云计算架构-zabbix企业级监控部署及使用(3)-zabbix的具体使用
 *   *  1. 配置监控远程主机
     *  2. server端测试监控情况
     *  3. 添加远程监控主机
     *  4. 添加监控项
     *  5. 设置自动发现规则
     *  6. 设置触发器
     *  7. 自定义绘图颜色
     *  8. 监控apache服务
     *  9. 监控mysql

# Linux云计算架构-zabbix企业级监控部署及使用(3)-zabbix的具体使用 #

本文来说下，如何正确的使用zabbix。要是用zabbix，就先要部署好zabbix的web。如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70]  
默认账号密码分别为：`admin`/`zabbix`，登录zabbix web  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 1]

## 1. 配置监控远程主机 ##

**服务端ip**：`192.168.8.128`  
**客户端ip**：`192.168.8.127`

# 上传源码包
    [root@client ~]# ll /usr/local/src/zabbix-3.4.3.tar.gz 
    -rw-r--r--. 1 root root 17130788 6月  19 14:14 /usr/local/src/zabbix-3.4.3.tar.gz
    
    # 解决依赖
    [root@client ~]# yum install -y curl curl-devel net-snmp net-snmp-devel perl-DBI gcc gcc-c++
    
    # 创建zabbix用户，用来运行zabbix_agent
    [root@client ~]# useradd -M -s /sbin/nologin zabbix
    
    # 解压、编译前准备、编译安装
    [root@client ~]# cd /usr/local/src/
    [root@client src]# tar xzvf zabbix-3.4.3.tar.gz 
    [root@client src]# cd zabbix-3.4.3/
    [root@client zabbix-3.4.3]# ./configure --prefix=/usr/local/zabbix --enable-agent
    # 提示不用make，直接运行make install
    [root@client zabbix-3.4.3]# make install
    
    # 查看zabbix_agent服务是否存在
    [root@client ~]# cat /etc/services | grep zabbix
    zabbix-agent    10050/tcp               # Zabbix Agent
    zabbix-agent    10050/udp               # Zabbix Agent
    zabbix-trapper  10051/tcp               # Zabbix Trapper
    zabbix-trapper  10051/udp               # Zabbix Trapper
    
    # 启动agent，可以看到10050已经被zabbix_agentd占用
    [root@client ~]# /usr/local/zabbix/sbin/zabbix_agentd
    [root@client ~]# netstat -antup | grep 10050
    tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      16919/zabbix_agentd 
    [root@client ~]# cp /usr/local/src/zabbix-3.4.3/misc/init.d/fedora/core5/zabbix_agentd /etc/init.d/
    [root@client ~]# vim /etc/init.d/zabbix_agentd
    21 ZABBIX_BIN="/usr/local/zabbix/sbin/zabbix_agentd"
    [root@client ~]# ll /usr/local/zabbix/sbin/zabbix_agentd 
    -rwxr-xr-x. 1 root root 1440200 7月   1 19:58 /usr/local/zabbix/sbin/zabbix_agentd
    
    # 设置agent开机自启
    [root@client ~]# chkconfig --add zabbix_agentd
    [root@client ~]# chkconfig zabbix_agentd on
    
    # 修改agent配置文件
    [root@client ~]# ln -s /usr/local/zabbix/etc/zabbix_agentd.conf /etc/
    [root@client ~]# ll /etc/zabbix_agentd.conf 
    lrwxrwxrwx. 1 root root 40 7月   1 20:18 /etc/zabbix_agentd.conf -> /usr/local/zabbix/etc/zabbix_agentd.conf
    [root@client ~]# vim /etc/zabbix_agentd.conf 
     93 Server=192.168.8.128
    134 ServerActive=192.168.8.128
    145 Hostname=192.168.8.127
    280 UnsafeUserParameters=1
    
    # 重启agent服务
    [root@client ~]# /etc/init.d/zabbix_agentd restart
    Restarting zabbix_agentd (via systemctl):                  [  确定  ]
    [root@client ~]# netstat -antup | grep zabbix
    tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      16919/zabbix_agentd

## 2. server端测试监控情况 ##

# server端测试连接情况
    [root@server ~]# /usr/local/zabbix/bin/zabbix_get -s 192.168.8.127 -p 10050 -k system.uname
    zabbix_get [13412]: Get value error: cannot connect to [[192.168.8.127]:10050]: [113] No route to host
    # 连接不到远程主机的10050端口，可以用两种方法检测。
    ①ping 192.168.8.127 服务端ping客户端，看能否ping通
    ②firewall-cmd --list-all 客户端上查看10050端口是否允许访问，如果防火墙开启，且不允许访问10050，需放行10050端口
    [root@client ~]# firewall-cmd --permanent --zone=public --add-port=10050/tcp
    success
    [root@client ~]# firewall-cmd --reload 
    success
    [root@server ~]# /usr/local/zabbix/bin/zabbix_get -s 192.168.8.127 -p 10050 -k system.uname
    Linux client 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64
    
    # -k参数指要获取的信息，可以使用以下命令查看可以获取哪些信息
    [root@server ~]# /usr/local/zabbix/sbin/zabbix_agentd -p
    [root@server ~]# /usr/local/zabbix/bin/zabbix_get -s 192.168.8.127 -p 10050 -k system.hostname
    client
    [root@server ~]# /usr/local/zabbix/sbin/zabbix_agentd -p | grep uptime
    system.uptime                                 [u|49407]
    [root@server ~]# /usr/local/zabbix/bin/zabbix_get -s 192.168.8.130 -p10050 -k system.uptime
    49436

## 3. 添加远程监控主机 ##

创建主机：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 2]  
填写主机信息：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 3]  
填写模板信息：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 4]  
等待监控正常：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 5]  
可以看到，已经监控成功了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 6]  
可以看到，被监控的远程主机已经有数据显示了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 7]

## 4. 添加监控项 ##

在添加主机中，已经添加了一个模板，即`Template OS Linux (Template App Zabbix Agent)`，可以看看这个模板具体是监控了什么。一般来说是够用的，如有特殊的监控内容，可以添加监控项。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 8]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 9]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 10]

## 5. 设置自动发现规则 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 11]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 12]

## 6. 设置触发器 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 13]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 14]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 15]

## 7. 自定义绘图颜色 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 16]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 17]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 18]

## 8. 监控apache服务 ##

# 配置apache
    # yum安装apache
    [root@client ~]# yum install -y httpd
    
    # 修改httpd配置文件,在末尾添加本地服务器状态页面
    [root@client ~]# vim /etc/httpd/conf/httpd.conf 
    ExtendedStatus On
    <location /server-status>
     SetHandler server-status
     Order allow,deny 
     Allow from 127.0.0.1 192.168.8.0/24
    </location>
    
    95 ServerName localhost:80
    # 重启httpd服务，并放行80端口号
    [root@client ~]# systemctl restart httpd.service 
    [root@client ~]# netstat -antup | grep httpd
    tcp6       0      0 :::80                   :::*                    LISTEN      9624/httpd
    [root@client ~]# firewall-cmd --permanent --zone=public --add-port=80/tcp
    success
    [root@client ~]# firewall-cmd --reload 
    success

查看网址：`http://192.168.8.127/server-status`是否显示正常，该页面用于收集客户端上apache服务的信息。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 19]

# 下载apache的监控脚本，一份windows上，一份linux上
    [root@client ~]# wget https://github.com/lorf/zapache/archive/master.zip
    [root@client ~]# unzip master.zip 
    Archive:  master.zip
    24a764b2e224f9b8d0a6bc94dcadd7568b2fe650
       creating: zapache-master/
      inflating: zapache-master/README.md  
      inflating: zapache-master/httpd-server-status.conf.sample  
      inflating: zapache-master/userparameter_zapache.conf.sample  
      inflating: zapache-master/zapache  
      inflating: zapache-master/zapache-template-active.xml  
      inflating: zapache-master/zapache-template.xml  
    [root@client ~]# cd zapache-master/
    [root@client zapache-master]# ll
    总用量 124
    -rw-r--r--. 1 root root   676 4月  22 2016 httpd-server-status.conf.sample
    -rw-r--r--. 1 root root  1780 4月  22 2016 README.md
    -rw-r--r--. 1 root root   224 4月  22 2016 userparameter_zapache.conf.sample
    -rwxr-xr-x. 1 root root  6110 4月  22 2016 zapache
    -rw-r--r--. 1 root root 52258 4月  22 2016 zapache-template-active.xml
    -rw-r--r--. 1 root root 52024 4月  22 2016 zapache-template.xml
    [root@client zapache-master]# cp zapache /usr/local/bin/
    # 这里可能要修改下脚本中的STATUS_URL值，不然后面添加模板后，监控项可能会不支持。
    # STATUS_URL="http://192.168.8.127/server-status?auto"
    # 可以使用curl命令测试下
    [root@client zapache-master]# cp userparameter_zapache.conf.sample /usr/local/zabbix/etc/zabbix_agentd.conf.d/
    [root@client zapache-master]# cd /usr/local/zabbix/etc/zabbix_agentd.conf.d/
    [root@client zabbix_agentd.conf.d]# ll
    总用量 4
    -rw-r--r--. 1 root root 224 7月   2 21:54 userparameter_zapache.conf.sample
    [root@client zabbix_agentd.conf.d]# mv userparameter_zapache.conf.sample userparameter_zapache.conf
    [root@client zabbix_agentd.conf.d]# vim userparameter_zapache.conf 
    UserParameter=zapache[*],/usr/local/bin/zapache \$1
    
    # 查看用户是否有执行权限
    [root@client ~]# ll /usr/local/bin/zapache 
    -rwxr-xr-x. 1 root root 6110 7月   2 21:53 /usr/local/bin/zapache
    
    # 修改zabbix_agent的配置文件
    [root@client ~]# vim /usr/local/zabbix/etc/zabbix_agentd.conf
    264 Include=/usr/local/zabbix/etc/zabbix_agentd.conf.d/userparameter_zapache.conf
    
    # 重启zabbix_agent
    [root@client ~]# /etc/init.d/zabbix_agentd stop
    Stopping zabbix_agentd (via systemctl):                    [  确定  ]
    [root@client ~]# /etc/init.d/zabbix_agentd start
    Starting zabbix_agentd (via systemctl):                    [  确定  ]
    
    
    # 至此为止，客户端的配置已经结束。

导入模板，使用`master.zip`里的`zapache-template.xml`作为模板文件，点击“导入”  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 20]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 21]  
为监控主机添加模板：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 22]  
点击“监测中-最新数据”，查看最新数据。过一会或者刷新下apache状态页面就可以看到最新数据了。  
我这里使用wifi做的实验，IP变了，但不影响。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 23]

## 9. 监控mysql ##

zabbix默认提供了mysql的监控模板。这里使用zabbix服务器本身监控本身的mysql数据库。

# 对数据库用户zabbix，授予在任何对象上的usage权限
    # 由于我是监控的服务端上的mysql服务，在赋予权限的时候，密码应该设置为zabbix，于配置zabbix服务端时保持一致。
    # 如果监控的不是服务端上的mysql,赋予zabbix用户时密码随意。
    [root@server ~]# mysql -uroot -p123456
    mysql: [Warning] Using a password on the command line interface can be insecure.
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 68
    Server version: 5.7.19 Source distribution
    
    Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
    
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> grant usage on *.* to zabbix@'%' identified by 'zabbix';
    Query OK, 0 rows affected, 1 warning (0.03 sec)
    
    mysql> flush privileges;
    Query OK, 0 rows affected (0.01 sec)
    
    mysql> exit
    Bye
    
    # 复制一份zabbix中mysql用户参数配置文件
    [root@server ~]# ll /usr/local/zabbix/etc/zabbix_agentd.conf.d/
    总用量 0
    [root@server ~]# cp /usr/local/src/zabbix-3.4.3/conf/zabbix_agentd/userparameter_mysql.conf /usr/local/zabbix/etc/zabbix_agentd.conf.d/
    [root@server ~]# ll /usr/local/zabbix/etc/zabbix_agentd.conf.d/
    总用量 4
    -rw-r--r-- 1 root root 1531 7月   3 20:45 userparameter_mysql.conf
    
    # 编辑zabbix客户端配置文件，将mysql用户参数配置文件的路径配置进去
    [root@server ~]# vim /usr/local/zabbix/etc/zabbix_agentd.conf
    265 Include=/usr/local/zabbix/etc/zabbix_agentd.conf.d/*.conf

# 建立数据库与zabbix的连接信息
    # 隐藏文件可以用ls -a显示
    [root@server ~]# vim /usr/local/zabbix/etc/.my.cnf
    #Zabbix Agent
    [mysql]
    host=localhost
    user=zabbix
    password=zabbix
    socket=/var/lib/mysql/mysql.sock
    
    [mysqladmin]
    host=localhost
    user=zabbix
    password=zabbix
    socket=/var/lib/mysql/mysql.sock
    
    # 编辑zabbix中mysql用户参数配置文件，添加.my.cnf文件的路径
    # 添加mysql的路径
    # 主要看HOME所在行的目录是否正确。
    [root@server ~]# vim /usr/local/zabbix/etc/zabbix_agentd.conf.d/userparameter_mysql.conf 
    # For all the following commands HOME should be set to the directory that has .my.cnf file with password information.
    
    # Flexible parameter to grab global variables. On the frontend side, use keys like mysql.status[Com_insert].
    # Key syntax is mysql.status[variable].
    UserParameter=mysql.status[*],echo "show global status where Variable_name='$1';" | HOME=/usr/local/zabbix/etc/ /var/lib/mysql/bin/mysql -N | awk '{print $$2}'
    
    # Flexible parameter to determine database or table size. On the frontend side, use keys like mysql.size[zabbix,history,data].
    # Key syntax is mysql.size[<database>,<table>,<type>].
    # Database may be a database name or "all". Default is "all".
    # Table may be a table name or "all". Default is "all".
    # Type may be "data", "index", "free" or "both". Both is a sum of data and index. Default is "both".
    # Database is mandatory if a table is specified. Type may be specified always.
    # Returns value in bytes.
    # 'sum' on data_length or index_length alone needed when we are getting this information for whole database instead of a single table
    UserParameter=mysql.size[*],bash -c 'echo "select sum($(case "$3" in both|"") echo "data_length+index_length";; data|index) echo "$3_length";; free) echo "data_free";; esac)) from information_schema.tables$([[ "$1" = "all" || ! "$1" ]] || echo " where table_schema=\"$1\"")$([[ "$2" = "all" || ! "$2" ]] || echo "and table_name=\"$2\"");" | HOME=/usr/local/zabbix/etc/ /var/lib/mysql/bin/mysqladmin -N'
    
    UserParameter=mysql.ping,HOME=/usr/local/zabbix/etc/ /var/lib/mysql/bin/mysqladmin ping | grep -c alive
    UserParameter=mysql.version,/var/lib/mysql/bin/mysql -V
    
    
    # 重启zabbix_agentd服务
    [root@server ~]# /etc/init.d/zabbix_agentd restart 
    Restarting zabbix_agentd (via systemctl):                  [  确定  ]

为需要监控的主机添加模板“Template DB MySQL”  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 24]  
查看最新数据：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 25]  
可以看到，所有的监控信息都已经出来了。至于IP地址为什么和前面的不同，这个不影响，换了一个wifi而已，不影响。

如果在配置过程中没有得到想要的想过可以按照以下步骤排错：  
①查看zabbix\_server、zabbix\_agentd是否已经启动，有没有监听到10050和10051端口号。  
②查看zabbix服务端和客户端的日志文件，看看有没有报错。  
③查看模板的脚本文件，看脚本里的一些命令能不能正确获取的数据。apache的是`/usr/local/bin/zapache`，mysql的是`/usr/local/zabbix/etc/zabbix_agentd.conf.d/userparameter_mysql.conf`

如果以上方法还是报错，可以在评论区留言，一起讨论。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/2020070119362886.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20200701194114959.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020070121143698.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20200701212935989.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20200701213041627.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20200701213146453.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20200701213235295.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20200701213826478.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20200701214216618.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20200701214541523.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20200701214247415.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20200701222144412.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20200701233227757.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20200701233906896.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 14]: https://img-blog.csdnimg.cn/20200701233921920.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 15]: https://img-blog.csdnimg.cn/20200701233711911.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20200701233841975.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/20200701234007376.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 18]: https://img-blog.csdnimg.cn/20200701234035318.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 19]: https://img-blog.csdnimg.cn/20200702214540919.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 20]: https://img-blog.csdnimg.cn/20200702220449996.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 21]: https://img-blog.csdnimg.cn/2020070222071195.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 22]: https://img-blog.csdnimg.cn/20200702221049826.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 23]: https://img-blog.csdnimg.cn/20200704160046986.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 24]: https://img-blog.csdnimg.cn/20200703205732816.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70 25]: https://img-blog.csdnimg.cn/20200704231512619.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70