Linux云计算架构-部署Zabbix企业级监控系统5.0

偏执的太偏执、 2022-12-30 08:23 203阅读 0赞

### 文章目录 ###

*  Linux云计算架构-部署Zabbix企业级监控系统5.0
 *   *  1. Zabbix5.0简介
     *  2. 环境配置
     *  3. 安装zabbix server
     *  4. 配置zabbix web界面
     *  5. 设置中文界面和修复乱码
     *  6. 监控远程主机
     *  7. 安装grafana可视化界面
     *  8. 添加zabbix数据源
     *  9. Dashboard设计(待设计)
     *  附件1：zabbix.repo

# Linux云计算架构-部署Zabbix企业级监控系统5.0 #

## 1. Zabbix5.0简介 ##

Zabbix是一款开源免费的服务器监控管理软件，其功能强大、配置简单、可外接Grafana图形可视化，是企业运维监控软件的首选。

[zabbix 5.0][] 版本于 5 月 11 日正式发布，是最新的 LTS（长期支持）版本，5.0 带来很多功能和特性，详细见官方文档。

## 2. 环境配置 ##

# 系统版本
    [root@server ~]# cat /etc/redhat-release 
    CentOS Linux release 7.6.1810 (Core)

# 关闭并禁用防火墙，禁用selinux
    [root@server ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
    [root@server ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux 
    [root@server ~]# systemctl stop firewalld.service && systemctl disable firewalld.service 
    [root@server ~]# setenforce 0
    [root@server ~]# getenforce 
    Permissive
    
    # 配置zabbix的yum源
    [root@server ~]# wget https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm
    [root@server ~]# rpm -ivh zabbix-release-5.0-1.el7.noarch.rpm 
    [root@server ~]# cd /etc/yum.repos.d/
    [root@server yum.repos.d]# ll
    总用量 12
    -rw-r--r--. 1 root root 2523 6月  16 2018 Centos-7.repo
    -rw-r--r--. 1 root root  664 5月  11 2018 epel-7.repo
    -rw-r--r--. 1 root root  853 5月  11 2020 zabbix.repo
    [root@server ~]# sed -i 's#http://repo.zabbix.com#https://mirrors.aliyun.com/zabbix#' /etc/yum.repos.d/zabbix.repo
    [root@server ~]# yum clean all
    [root@server ~]# yum makecache

## 3. 安装zabbix server ##

[root@server ~]# yum install zabbix-server-mysql zabbix-agent -y
    [root@server ~]# yum install centos-release-scl -y
    [root@server ~]# yum install zabbix-web-mysql-scl zabbix-apache-conf-scl -y 【要启用zabbix.repo的某个源才可以安装】
    [root@server ~]# yum install mariadb-server -y
    
    # zabbix-server-mysql zabbix-agent zabbix服务端和客户端
    # centos-release-scl 用于安装高版本的php
    # zabbix-web-mysql-scl zabbix-apache-conf-scl zabbix的前端web界面
    # mariadb-server mariadb数据库
    
    
    # 配置mariadb数据库
    [root@server ~]# systemctl start mariadb
    [root@server ~]# systemctl enable mariadb
    [root@server ~]# mysql_secure_installation 
    
    NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
          SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!
    
    In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and
    you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] y ... Success! By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB
    installation should now be secure.
    
    Thanks for using MariaDB!
    
    
    # 创建zabbix数据库和zabbix用户
    [root@server ~]# mysql -uroot -p123456
    Welcome to the MariaDB monitor.  Commands end with ; or \g.
    Your MariaDB connection id is 10
    Server version: 5.5.68-MariaDB MariaDB Server
    
    Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    MariaDB [(none)]> create database zabbix character set utf8 collate utf8_bin;
    Query OK, 1 row affected (0.00 sec)
    
    MariaDB [(none)]> create user zabbix@localhost identified by '123456';
    
    MariaDB [(none)]> grant all privileges on zabbix.* to zabbix@localhost;
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
    
    MariaDB [(none)]> exit
    Bye
    
    
    # 导入zabbix的数据
    [root@server ~]# ll /usr/share/doc/zabbix-server-mysql-5.0.7/create.sql.gz 
    -rw-r--r--. 1 root root 1644549 12月 21 18:24 /usr/share/doc/zabbix-server-mysql-5.0.7/create.sql.gz
    [root@server ~]# zcat /usr/share/doc/zabbix-server-mysql-5.0.7/create.sql.gz | mysql -uzabbix -p zabbix
    Enter password: 123456

**修改配置文件：**

[root@server ~]# vim /etc/zabbix/zabbix_server.conf 
    124 DBPassword=123456
    [root@server ~]# vim /etc/opt/rh/rh-php72/php-fpm.d/zabbix.conf
     24 php_value[date.timezone] = Asia/Shanghai
    
    # 启动并加入到开机自启
    [root@server ~]# systemctl start zabbix-server zabbix-agent httpd rh-php72-php-fpm
    [root@server ~]# systemctl enable zabbix-server zabbix-agent httpd rh-php72-php-fpm

## 4. 配置zabbix web界面 ##

**输入网址**：`http://192.168.80.165/zabbix/setup.php`

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 1]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 2]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 3]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 4]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 5]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 6]

**初始账号密码**：`Admin`/`zabbix`

## 5. 设置中文界面和修复乱码 ##

**①设置中文界面**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 7]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 8]

**②修复中文字体乱码**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 9]

# 上传字体文件，然后替换配置文件中字体配置
    [root@server ~]# cd /usr/share/zabbix/assets/fonts/
    [root@server fonts]# ll
    总用量 0
    lrwxrwxrwx. 1 root root 33 12月 23 10:11 graphfont.ttf -> /etc/alternatives/zabbix-web-font
    [root@server fonts]# rz
    
    [root@server fonts]# ll
    总用量 11512
    lrwxrwxrwx. 1 root root       33 12月 23 10:11 graphfont.ttf -> /etc/alternatives/zabbix-web-font
    -rw-r--r--. 1 root root 11787328 3月   2 2019 simkai.ttf
    [root@server fonts]# vim /usr/share/zabbix/include/defines.inc.php 
      81 define('ZBX_GRAPH_FONT_NAME',           'simkai'); // font file name
     122 define('ZBX_FONT_NAME', 'simkai');
     [root@server fonts]# systemctl restart httpd

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 10]

## 6. 监控远程主机 ##

# 配置zabbix的yum源并安装zabbix-agent
    [root@slave1 ~]# wget https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/x86_64/zabbix-release-5.0-1.el7.noarch.rpm
    [root@slave1 ~]# rpm -ivh zabbix-release-5.0-1.el7.noarch.rpm 
    [root@slave1 ~]# cd /etc/yum.repos.d/
    [root@slave1 yum.repos.d]# ll
    总用量 12
    -rw-r--r--. 1 root root 2523 6月  16 2018 Centos-7.repo
    -rw-r--r--. 1 root root  664 5月  11 2018 epel-7.repo
    -rw-r--r--. 1 root root  853 5月  11 2020 zabbix.repo
    [root@slave1 ~]# sed -i 's#http://repo.zabbix.com#https://mirrors.aliyun.com/zabbix#' /etc/yum.repos.d/zabbix.repo
    [root@slave1 ~]# yum clean all
    [root@slave1 ~]# yum makecache
    [root@slave1 ~]# yum install zabbix-agent -y

# 修改zabbix-agent的配置文件
    sed -i 's/Server=127.0.0.1/Server=192.168.80.165/g' /etc/zabbix/zabbix_agentd.conf
    sed -i 's/ServerActive=127.0.0.1/ServerActive=192.168.80.165/g' /etc/zabbix/zabbix_agentd.conf
    sed -i 's/Hostname=Zabbix server/Hostname=192.168.80.166/g' /etc/zabbix/zabbix_agentd.conf
    sed -i 's/# UnsafeUserParameters=0/UnsafeUserParameters=1 /g' /etc/zabbix/zabbix_agentd.conf
    
    # 启动zabbix-agent服务
    [root@slave1 ~]# systemctl restart zabbix-agent.service 
    [root@slave1 ~]# systemctl enable zabbix-agent.service 
    
    # 可以看到zabbix-agent服务的默认端口号10050已经被监听
    [root@slave1 ~]# netstat -antup |grep 10050
    tcp        0      0 0.0.0.0:10050           0.0.0.0:*               LISTEN      20694/zabbix_agentd 
    tcp6       0      0 :::10050                :::*                    LISTEN      20694/zabbix_agentd 
    
    
    # 关闭防火墙或开放10050端口号
    [root@slave1 ~]# firewall-cmd --permanent --zone=public --add-port=10050/tcp
    success
    [root@slave1 ~]# firewall-cmd --reload
    success

web界面创建主机，并添加相应的监控模板：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 11]

写入主机信息：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 12]

选择监控模板：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 13]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 14]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 15]

查看监控结果，可用性显示绿色才算监控成功：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 16]

## 7. 安装grafana可视化界面 ##

**grafana版本**：`grafana-6.7.4-1`

[root@server ~]# cd /usr/local/src/
    [root@server src]# rz
    
    [root@server src]# ll
    总用量 62280
    -rw-r--r--. 1 root root 63773044 9月  28 21:27 grafana-6.7.4-1.x86_64.rpm
    [root@server src]# yum install grafana-6.7.4-1.x86_64.rpm -y
    [root@server src]# systemctl start grafana-server.service && systemctl enable grafana-server.service 
    
    # 默认占用3000号端口
    [root@server src]# netstat -antup |grep 3000
    tcp6       0      0 :::3000                 :::*                    LISTEN      87602/grafana-serve

首次登录默认账号密码：`admin`/`admin`  
会要求修改登录密码，这里建议修改成和zabbix系统一致，方便记忆。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 17]

## 8. 添加zabbix数据源 ##

**这里有很多数据源可以选择，但就是没有zabbix的。**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 18]

**手动下载zabbix的plugin，导入进去：**

[root@server ~]# ll grafana-zabbix-plugins-v3.12.4-1.zip 
    -rw-r--r--. 1 root root 2747693 9月  30 15:49 grafana-zabbix-plugins-v3.12.4-1.zip
    [root@server ~]# unzip grafana-zabbix-plugins-v3.12.4-1.zip -d /var/lib/grafana/plugins/
    [root@server ~]# systemctl restart grafana-server.service

**启用zabbix数据源的plugins：**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 19]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 20]

**重新添加zabbix数据源，可以看到有zabbix了。**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 21]

**设置zabbix的api接口信息：**

[root@server ~]# ll /usr/share/zabbix/api_jsonrpc.php 
    -rw-r--r--. 1 root root 2326 12月 21 17:38 /usr/share/zabbix/api_jsonrpc.php

**接口地址**：`http://192.168.80.165/zabbix/api_jsonrpc.php`

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 22]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 23]

保存并测试显示没问题，就说明已经将zabbix数据源正确添加了。

zabbix默认带有三个面板，可以导入看看效果，不满意可以自定义dashboard。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 24]

## 9. Dashboard设计(待设计) ##

**预期效果：**

*  **总览面板**：可以看到所有服务器的监控情况。
 *  **分览面板**：可以看到某台服务器的监控情况。
 *  **监控指标**：硬件、cpu、内存、磁盘、网络、应用(nginx、tomcat)、数据库
 *  **阈值预警**：超出阈值会有颜色变化
 *  使用模板和变量设计Dashboard，可以直接导入Dashboard进行复用。

## 附件1：zabbix.repo ##

[root@server yum.repos.d]# cat zabbix.repo 
    [zabbix]
    name=Zabbix Official Repository - $basearch
    baseurl=https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/$basearch/
    enabled=1
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
    
    [zabbix-frontend]
    name=Zabbix Official Repository frontend - $basearch
    baseurl=https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/$basearch/frontend
    enabled=1
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
    
    [zabbix-debuginfo]
    name=Zabbix Official Repository debuginfo - $basearch
    baseurl=https://mirrors.aliyun.com/zabbix/zabbix/5.0/rhel/7/$basearch/debuginfo/
    enabled=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX-A14FE591
    gpgcheck=1
    
    [zabbix-non-supported]
    name=Zabbix Official Repository non-supported - $basearch
    baseurl=https://mirrors.aliyun.com/zabbix/non-supported/rhel/7/$basearch/
    enabled=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-ZABBIX
    gpgcheck=1

[zabbix 5.0]: https://www.zabbix.com/documentation/current/manual/introduction/whatsnew500
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/e58bb1b12e2f42f192823e4fec48100c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/12924188bad04b14b76fd753c244afb4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/2020122418014560.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 3]: https://img-blog.csdnimg.cn/20201224180153946.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 4]: https://img-blog.csdnimg.cn/20201224180207256.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 5]: https://img-blog.csdnimg.cn/20201224180216103.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 6]: https://img-blog.csdnimg.cn/20201224180228250.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 7]: https://img-blog.csdnimg.cn/20201224180240657.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 8]: https://img-blog.csdnimg.cn/20201224180255266.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 9]: https://img-blog.csdnimg.cn/20201224180306807.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 10]: https://img-blog.csdnimg.cn/20201224180323167.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 11]: https://img-blog.csdnimg.cn/20201224180340562.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 12]: https://img-blog.csdnimg.cn/2020122418035828.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 13]: https://img-blog.csdnimg.cn/20201224180410767.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 14]: https://img-blog.csdnimg.cn/20201224180431453.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 15]: https://img-blog.csdnimg.cn/20201224180443266.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 16]: https://img-blog.csdnimg.cn/20201224180454616.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 17]: https://img-blog.csdnimg.cn/20201224180524630.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 18]: https://img-blog.csdnimg.cn/20201224180553675.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 19]: https://img-blog.csdnimg.cn/202012241806072.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 20]: https://img-blog.csdnimg.cn/2020122418061956.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 21]: https://img-blog.csdnimg.cn/20201224180628778.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 22]: https://img-blog.csdnimg.cn/20201224180641860.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 23]: https://img-blog.csdnimg.cn/20201224180651504.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 24]: https://img-blog.csdnimg.cn/20201224180659855.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ==,size_16,color_FFFFFF,t_70#pic_center