Linux云计算架构-docker容器命名和资源配额控制（2）

布满荆棘的人生 2022-12-13 04:15 89阅读 0赞

### 文章目录 ###

*  Linux云计算架构-docker容器命名和资源配额控制（2）
 *   *  1. docker容器命名和重命名
     *  2. 创建docker容器时指定主机名
     *  3. 设置docker容器开机自动启动
     *  4. cgroup资源配置介绍
     *  5. 容器cpu资源配置
     *  6. 容器内存资源配置
     *  7. docker数据映射
     *  8. 容器IO资源配置
     *  9. run exec update的常用语法

# Linux云计算架构-docker容器命名和资源配额控制（2） #

## 1. docker容器命名和重命名 ##

# 容器命名
    [root@server ~]# docker run -itd --name docker1 centos:latest /bin/bash
    ff4a82982160eaf4f652333088c02e3958d5e641b5e361e166c03d49f2d737a4

![在这里插入图片描述][20201009203842650.png_pic_center]

# 容器重命名
    [root@server ~]# docker rename docker1 docker2

![在这里插入图片描述][20201009204023735.png_pic_center]

## 2. 创建docker容器时指定主机名 ##

# 设置容器的主机名
    [root@server ~]# docker run -itd --name docker1 -h master_server centos:latest /bin/bash
    baf20246fd2f654a7611599cc116260b5a8094f0a1aa61641bd526dba6a38351

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center]

## 3. 设置docker容器开机自动启动 ##

<table> 
 <thead> 
  <tr> 
   <th>重启策略</th> 
   <th>作用</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>no</td> 
   <td>容器退出时不重启容器</td> 
  </tr> 
  <tr> 
   <td>on-failure</td> 
   <td>容器非正常退出（退出状态非0）时重启容器</td> 
  </tr> 
  <tr> 
   <td>on-failure:3</td> 
   <td>容器非正常退出时重启容器，最多重启3次</td> 
  </tr> 
  <tr> 
   <td>always</td> 
   <td>容器退出时总是重启容器，重启策略一般设置为always</td> 
  </tr> 
  <tr> 
   <td>unless-stopped</td> 
   <td>容器退出时总是重启非停止的容器</td> 
  </tr> 
 </tbody> 
</table>

# 创建时设置--restart
    [root@server ~]# docker run --restart=always -itd --name=docker1 centos:latest /bin/bash
    584a379e3c6261ec082a92cacf0c1e85d1c4f01c65d374a7e1f03a83716fb922

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 1]  
未设置`--restart`参数时，容器不会随着docker服务的启动而启动。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 2]

# 启动后设置--restart
    [root@server ~]# docker update --restart=always docker2
    docker2
    [root@server ~]# systemctl restart docker

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 3]

## 4. cgroup资源配置介绍 ##

**①Docker通过cgroup来控制容器使用的资源，如CPU、内存、磁盘等。**  
**②`cgroup`，即Control groups，是linux内核提供的一种可以限制、记录、隔离进程组所使用的物理资源（CPU、memory、磁盘）的机制。被LXC、docker等项目用于实现进程资源的控制。cgroup提供将进程分组化管理的功能和接口的基础结构，资源管理都是通过这个功能来实现的。**  
**③进行硬件配额，可以当在docker服务器上运行多个容器时，防止某容器把所有的硬件资源都占用了。**

## 5. 容器cpu资源配置 ##

**①指定docker容器可以在单个CPU上使用的cpu份额（默认cpu份额是1024）【`--cpu-shares`】**

# 指定单个cpu的使用份额为512
    [root@server ~]# docker run -itd --cpu-shares 512 centos:latest /bin/bash
    97ae61edc49f337c1e8b9c90f86e4ef43c2c5343bc181670f18bf1f4af8ad3f4
    [root@server ~]# docker ps |grep 97ae61edc49
    97ae61edc49f        centos:latest       "/bin/bash"         19 seconds ago      Up 18 seconds                           upbeat_bartik
    [root@server ~]# docker exec -it 97ae61edc49f /bin/bash
    # 查看单个CPU的份额
    [root@97ae61edc49f /]# cat /sys/fs/cgroup/cpu/cpu.shares
    512
    # 查看该容器能使用的CPU核心数
    [root@97ae61edc49f /]# cat /sys/fs/cgroup/cpuset/cpuset.cpus 
    0-7
    # 跑满CPU测试CPU使用情况
    [root@97ae61edc49f /]# yum install epel-release -y
    [root@97ae61edc49f /]# yum install stress -y
    [root@97ae61edc49f /]# stress -c 2 -i 2 --verbose --timeout 1m
    ========================================
    stress命令各参数介绍：
    -？ 显示帮助信息
    -v  显示版本号
    -q  不显示运行信息
    -n  显示已完成的指令情况
    -t  --timeout N 指定运行N秒后停止
        --backoff N 等待N微妙后开始运行
    -c  产生n个进程，每个进程不停的计算随机数的平方根，测试cpu。
    -i  产生n个进程，每个进程反复调用sync()，用于将内存上的内容写到磁盘上，用于测试磁盘。
    -m  产生n个进程，每个进程不断调用内存分配malloc()和内存释放free()函数，测试内存。
    --verbose  显示stress程序运行过程中的详细信息
    
    常用语法：
    stress -c 2 -i 2 --verbose --timeout 1m
    ========================================

使用`top`命令，然后按`1`查看各CPU核心使用情况，**由于只启用了一个容器，不存在多个容器抢占同一个CPU的情况，故CPU份额设置看不出效果。** 但可以看到有两个核心的CPU已经跑满了。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 4]

**②多核心CPU控制【`--cpuset-cpus`】**  
`taskset`命令设定cpu亲和力，可以将一个或多个进程绑定到一个或多个处理器上运行。当cpu数量较多时，将进程绑定到某些cpu上运行，可以减少cpu上下文切换带来的开销，节约时间。

# 将指定CPU核心和进程PID绑定
    -c  --cpu-list 以列表格式显示和指定cpu
    -p  --pid   指定进程pid
    taskset -cp 1,2 PID
    
    # sshd进程只运行在1号和2号cpu上
    # 可以看到允许使用的核心由0-7变为1,2
    [root@server ~]# ps aux | grep sshd
    root       6629  0.0  0.2 112756  4316 ?        Ss   20:34   0:00 /usr/sbin/sshd -D
    root      19038  0.0  0.0 112728   988 pts/0    S+   20:40   0:00 grep --color=auto sshd
    [root@server ~]# taskset -cp 1,2 6629
    pid 6629's current affinity list: 0-7 pid 6629's new affinity list: 1,2
    
    # 查看进程在哪个cpu上运行
    [root@server ~]# taskset -cp 6629
    pid 6629‘s current affinity list: 1,2
    
    # 创建docker容器时，仅允许容器使用0，1，2三个核心
    [root@server ~]# docker run -it --name cpu1 --cpuset-cpus 0-2 centos:latest /bin/bash
    # 查看该容器能使用的cpu核心
    [root@b58e346972cf /]# cat /sys/fs/cgroup/cpuset/cpuset.cpus
    0-2
    
    # 查看PID为1的进程在哪些cpu上运行
    [root@b58e346972cf /]# taskset -cp 1
    pid 1's current affinity list: 0-2
    # # 查看PID为1是哪个进程，即第一个打开的进程。这里是/bin/bash
    [root@b58e346972cf /]# ps aux
    USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root          1  0.1  0.1  12108  2240 pts/0    Ss   12:45   0:00 /bin/bash
    root         20  0.0  0.0  44584  1792 pts/0    R+   12:48   0:00 ps aux

**③CPU份额和核心混合控制**

# 创建第一个容器，单个cpu份额为512，仅允许使用0号核心。
    [root@server ~]# docker run -itd --name docker10 --cpuset-cpus 0 --cpu-shares 512 centos:latest /bin/bash
    5a0581f6c20ec9defd4fba2d79d7541db7f71282aa6d324ef277fbf88f4d4a45
    
    # 创建第二个容器，单个cpu份额为1024，即默认。仅允许使用0号核心。
    [root@server ~]# docker run -itd --name docker20 --cpuset-cpus 0 --cpu-shares 1024 centos:latest /bin/bash 
    41c9a89a4c430f1227798e13477cb2ab64e724ca3f7e550ca2fd30c42bd864dd
    
    # 从上面两个容器的配置可以看到，docker10和docker20都只能使用1个cpu，故肯定会出现抢占cpu的时候。并且单个cpu份额docker20是docker10的两倍。
    # 在两个容器上跑满cpu，最后使用top+1命令查看各cpu使用情况.
    # docker1跑满仅能使用33.3%的cpu，而docker2跑满应该是可以使用66.7%的cpu
    # 在两个容器中运行如下命令：
    [root@5a0581f6c20e /]# stress -c 1 -t 10m
    [root@41c9a89a4c43 /]# stress -c 1 -t 10m

通过观察cpu的使用情况，可以看到cpu0已经跑满了，而且两个进程的cpu使用比和配置的份额比保持一致，都是2:1。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 5]

**④CPU周期和时间片控制【`--cpu-period` `--cpu-quota`】**  
**控制cpu被容器占用的时间**

# 指定一个cpu运行周期，在这个周期内cpu使用是固定的。超过这个周期，cpu会做重新分配。
    # 单位微秒，最小值1000微妙，最大值1000000微秒，默认值100000微秒。
    --cpu-period
    # 指定在这个周期内允许使用多少时间片，默认值-1，即不做控制。 
    --cpu-quota
    
    # 设置docker实例每1秒只能使用单个cpu的0.2秒的时间。
    docker run -itd --cpu-period 1000000 --cpu-quota 200000 centos:latest /bin/bash
    
    # 查看--cpu-period值
    cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
    # 查看--cpu-quota值
    cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us

**⑤docker容器运行结束后自动释放资源**

# 创建一个容器abong,在5s后自动删除
    docker run -itd --rm --name abong centos:latest sleep 5

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 6]

## 6. 容器内存资源配置 ##

# 允许容器使用的内存上限是128m
    [root@server ~]# docker run -itd -m 128m centos:latest /bin/bash
    d35ac68866569fe301f5628bcea51245d0a04fe9efca66c9505040758fb386cf
    
    # 查看容器允许使用多大的内存
    [root@d35ac6886656 /]# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
    134217728
    
    # 限制docker容器为2核心，内存为128m
    [root@server ~]# docker run -itd --cpuset-cpus 0,1 -m 128m centos:latest /bin/bash
    a67f129c452527f7c80096e104306b1d0ea44af7b1be1c62e7ad396fbcd06373
    [root@server ~]# docker exec -it a67f129c4525 /bin/bash
    [root@a67f129c4525 /]# cat /sys/fs/cgroup/cpuset/cpuset.cpus
    0-1
    [root@a67f129c4525 /]# cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
    134217728

## 7. docker数据映射 ##

**docker主要用于计算，不擅长存储，故可以将存储目录映射到物理机上的目录上。**  
**数据映射格式**：`docker run -itd --name web1 -v 物理机目录:容器中数据存储目录`

[root@server ~]# docker run -itd --name web1 -v /var/www/html:/var/www/html centos:httpd /bin/init
    2adedbe8723ba051da197da0ebff285aa1b39381505d1f51e6da06f2b31f3e9b
    [root@server ~]# docker exec -it 2adedbe8723 /bin/bash
    [root@2adedbe8723b /]# echo "docker image" >> /var/www/html/index.html # 在容器中创建index.html文件
    
    # 在物理机上对应目录也可以查看到index.html文件
    [root@server ~]# ll /var/www/html/
    总用量 4
    -rw-r--r-- 1 root root 13 10月 10 21:48 index.html
    [root@server ~]# cat /var/www/html/index.html 
    docker image

## 8. 容器IO资源配置 ##

# 查看读写资源配置参数
    [root@server ~]# docker run --help | grep device
          --blkio-weight-device list       Block IO weight (relative device weight) (default [])
          --device list                    Add a host device to the container
          --device-cgroup-rule list        Add a rule to the cgroup allowed devices list
          --device-read-bps list           Limit read rate (bytes per second) from a device (default [])
          --device-read-iops list          Limit read rate (IO per second) from a device (default [])
          --device-write-bps list          Limit write rate (bytes per second) to a device (default [])
          --device-write-iops list         Limit write rate (IO per second) to a device (default [])
          --gpus gpu-request               GPU devices to add to the container ('all' to pass all GPUs)
    [root@server ~]# docker run --help | grep write
          --device-write-bps list          Limit write rate (bytes per second) to a device (default [])
          --device-write-iops list         Limit write rate (IO per second) to a device (default [])
          --entrypoint string              Overwrite the default ENTRYPOINT of the image
    [root@server ~]# docker run --help | grep read
          --device-read-bps list           Limit read rate (bytes per second) from a device (default [])
          --device-read-iops list          Limit read rate (IO per second) from a device (default [])
          --read-only                      Mount the container's root filesystem as read only

**例子：限制容器实例对硬盘的最高读写速度为1MB/S**

[root@server ~]# docker run -it -v /var/www/html:/var/www/html --device /dev/sda:/dev/sda --device-write-bps /dev/sda:1mb centos:httpd /bin/bash
    [root@6d0b9d746d0e /]# time dd if=/dev/zero of=/var/www/html/test.out bs=1M count=10 oflag=direct,nonblock
    10+0 records in
    10+0 records out
    10485760 bytes (10 MB, 10 MiB) copied, 10.0048 s, 1.0 MB/s
    
    real	0m10.010s
    user	0m0.006s
    sys	0m0.010s
    ===============================================
    --device  # 添加一块硬盘给该容器使用
    --device-write-bps /dev/sda:1mb    # /dev/sda设备的写速度为1mb
    time # 用于计时
    direct  # 读写数据采用直接IO方式，即直接从内存写入硬盘中，不走缓存。
    nonblock  # 读写数据采用非阻塞IO方式，优先写dd命令的数据

## 9. run exec update的常用语法 ##

①`docker run`

[root@server ~]# docker run --help
    
    Usage:	docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
    
    Run a command in a new container
    
    Options:
          --add-host list                  Add a custom host-to-IP mapping (host:ip)
      -a, --attach list                    Attach to STDIN, STDOUT or STDERR
          --blkio-weight uint16            Block IO (relative weight), between 10 and 1000, or 0 to
                                           disable (default 0)
          --blkio-weight-device list       Block IO weight (relative device weight) (default [])
          --cap-add list                   Add Linux capabilities
          --cap-drop list                  Drop Linux capabilities
          --cgroup-parent string           Optional parent cgroup for the container
          --cidfile string                 Write the container ID to the file
          --cpu-period int                 Limit CPU CFS (Completely Fair Scheduler) period
          --cpu-quota int                  Limit CPU CFS (Completely Fair Scheduler) quota
          --cpu-rt-period int              Limit CPU real-time period in microseconds
          --cpu-rt-runtime int             Limit CPU real-time runtime in microseconds
      -c, --cpu-shares int                 CPU shares (relative weight)
          --cpus decimal                   Number of CPUs
          --cpuset-cpus string             CPUs in which to allow execution (0-3, 0,1)
          --cpuset-mems string             MEMs in which to allow execution (0-3, 0,1)
      -d, --detach                         Run container in background and print container ID
          --detach-keys string             Override the key sequence for detaching a container
          --device list                    Add a host device to the container
          --device-cgroup-rule list        Add a rule to the cgroup allowed devices list
          --device-read-bps list           Limit read rate (bytes per second) from a device (default [])
          --device-read-iops list          Limit read rate (IO per second) from a device (default [])
          --device-write-bps list          Limit write rate (bytes per second) to a device (default [])
          --device-write-iops list         Limit write rate (IO per second) to a device (default [])
          --disable-content-trust          Skip image verification (default true)
          --dns list                       Set custom DNS servers
          --dns-option list                Set DNS options
          --dns-search list                Set custom DNS search domains
          --domainname string              Container NIS domain name
          --entrypoint string              Overwrite the default ENTRYPOINT of the image
      -e, --env list                       Set environment variables
          --env-file list                  Read in a file of environment variables
          --expose list                    Expose a port or a range of ports
          --gpus gpu-request               GPU devices to add to the container ('all' to pass all GPUs)
          --group-add list                 Add additional groups to join
          --health-cmd string              Command to run to check health
          --health-interval duration       Time between running the check (ms|s|m|h) (default 0s)
          --health-retries int             Consecutive failures needed to report unhealthy
          --health-start-period duration   Start period for the container to initialize before
                                           starting health-retries countdown (ms|s|m|h) (default 0s)
          --health-timeout duration        Maximum time to allow one check to run (ms|s|m|h) (default 0s)
          --help                           Print usage
      -h, --hostname string                Container host name
          --init                           Run an init inside the container that forwards signals and
                                           reaps processes
      -i, --interactive                    Keep STDIN open even if not attached
          --ip string                      IPv4 address (e.g., 172.30.100.104)
          --ip6 string                     IPv6 address (e.g., 2001:db8::33)
          --ipc string                     IPC mode to use
          --isolation string               Container isolation technology
          --kernel-memory bytes            Kernel memory limit
      -l, --label list                     Set meta data on a container
          --label-file list                Read in a line delimited file of labels
          --link list                      Add link to another container
          --link-local-ip list             Container IPv4/IPv6 link-local addresses
          --log-driver string              Logging driver for the container
          --log-opt list                   Log driver options
          --mac-address string             Container MAC address (e.g., 92:d0:c6:0a:29:33)
      -m, --memory bytes                   Memory limit
          --memory-reservation bytes       Memory soft limit
          --memory-swap bytes              Swap limit equal to memory plus swap: '-1' to enable
                                           unlimited swap
          --memory-swappiness int          Tune container memory swappiness (0 to 100) (default -1)
          --mount mount                    Attach a filesystem mount to the container
          --name string                    Assign a name to the container
          --network network                Connect a container to a network
          --network-alias list             Add network-scoped alias for the container
          --no-healthcheck                 Disable any container-specified HEALTHCHECK
          --oom-kill-disable               Disable OOM Killer
          --oom-score-adj int              Tune host’s OOM preferences (-1000 to 1000)
          --pid string                     PID namespace to use
          --pids-limit int                 Tune container pids limit (set -1 for unlimited)
          --platform string                Set platform if server is multi-platform capable
          --privileged                     Give extended privileges to this container
      -p, --publish list                   Publish a container's port(s) to the host -P, --publish-all Publish all exposed ports to random ports --read-only Mount the container's root filesystem as read only
          --restart string                 Restart policy to apply when a container exits (default "no")
          --rm                             Automatically remove the container when it exits
          --runtime string                 Runtime to use for this container
          --security-opt list              Security Options
          --shm-size bytes                 Size of /dev/shm
          --sig-proxy                      Proxy received signals to the process (default true)
          --stop-signal string             Signal to stop a container (default "SIGTERM")
          --stop-timeout int               Timeout (in seconds) to stop a container
          --storage-opt list               Storage driver options for the container
          --sysctl map                     Sysctl options (default map[])
          --tmpfs list                     Mount a tmpfs directory
      -t, --tty                            Allocate a pseudo-TTY
          --ulimit ulimit                  Ulimit options (default [])
      -u, --user string                    Username or UID (format: <name|uid>[:<group|gid>])
          --userns string                  User namespace to use
          --uts string                     UTS namespace to use
      -v, --volume list                    Bind mount a volume
          --volume-driver string           Optional volume driver for the container
          --volumes-from list              Mount volumes from the specified container(s)
      -w, --workdir string                 Working directory inside the container

②`docker exec`

[root@server ~]# docker exec --help
    
    Usage:	docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
    
    Run a command in a running container
    
    Options:
      -d, --detach               Detached mode: run command in the background
          --detach-keys string   Override the key sequence for detaching a container
      -e, --env list             Set environment variables
      -i, --interactive          Keep STDIN open even if not attached
          --privileged           Give extended privileges to the command
      -t, --tty                  Allocate a pseudo-TTY
      -u, --user string          Username or UID (format: <name|uid>[:<group|gid>])
      -w, --workdir string       Working directory inside the container

③`docker update`

[root@server ~]# docker update --help 
    
    Usage:	docker update [OPTIONS] CONTAINER [CONTAINER...]
    
    Update configuration of one or more containers
    
    Options:
          --blkio-weight uint16        Block IO (relative weight), between 10 and 1000, or 0 to
                                       disable (default 0)
          --cpu-period int             Limit CPU CFS (Completely Fair Scheduler) period
          --cpu-quota int              Limit CPU CFS (Completely Fair Scheduler) quota
          --cpu-rt-period int          Limit the CPU real-time period in microseconds
          --cpu-rt-runtime int         Limit the CPU real-time runtime in microseconds
      -c, --cpu-shares int             CPU shares (relative weight)
          --cpus decimal               Number of CPUs
          --cpuset-cpus string         CPUs in which to allow execution (0-3, 0,1)
          --cpuset-mems string         MEMs in which to allow execution (0-3, 0,1)
          --kernel-memory bytes        Kernel memory limit
      -m, --memory bytes               Memory limit
          --memory-reservation bytes   Memory soft limit
          --memory-swap bytes          Swap limit equal to memory plus swap: '-1' to enable unlimited swap
          --pids-limit int             Tune container pids limit (set -1 for unlimited)
          --restart string             Restart policy to apply when a container exits

**通过以上的学习，我们可以快速创建一个拥有指定`主机名`、`动态IP地址`、`CPU核数`、`内存大小` 、`对某设备限制读写速度`的容器实例。**

[20201009203842650.png_pic_center]: /images/20221123/3f9f7c666c614af48afecd007a3aa38f.png
[20201009204023735.png_pic_center]: /images/20221123/c89cbdc83b2c4879a5c7b30d45e43c0c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/6276964ecf3144f8990c004ba491ae5c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221123/045205f09ea14553b41f75c4bd59cdc6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221123/077b346ed7964fa4885c29eb4ec0c49c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221123/df3286317b324d7dbe98a38ecb8d45c3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20221123/2baf962931b74a80893d1b596659e5d6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 5]: /images/20221123/b188e40a006c4c84809abc3288733e3a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl8zNjUyMjA5OQ_size_16_color_FFFFFF_t_70_pic_center 6]: /images/20221123/ad9141cd8da64e1eb9af6a7031257817.png