神经网络学习小记录58——Keras GhostNet模型的复现详解

Myth丶恋晨 2022-12-23 06:52 252阅读 0赞

### 神经网络学习小记录58——Keras GhostNet模型的复现详解 ###

*  学习前言
 *  什么是GhostNet模型
 *  源码下载
 *  GhostNet模型的实现思路
 *   *  1、Ghost Module
     *  2、Ghost Bottlenecks
     *  3、Ghostnet的构建
 *  GhostNet的代码构建
 *   *  1、模型代码的构建
     *  2、Yolov4上的应用

# 学习前言 #

GhostNet是华为诺亚方舟实验室提出来的一个非常有趣的网络，我们一起来学习一下。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center]

# 什么是GhostNet模型 #

**2020年，华为新出了一个轻量级网络，命名为GhostNet。**

在优秀CNN模型中，特征图存在冗余是非常重要的。如图所示，这个是对ResNet-50第一个残差块特征图进行可视化的结果，当我们给一个神经网络输入一张图片时，我们可以获得特别多的特征图。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 1]  
**利用小扳手连接起来的两幅特征图，它们的相似性就特别高**，这个就是神经网络中存在的特征图冗杂的情况。

**作者将相似的特征图认为是彼此的Ghost，所以这个网络就叫做GhostNet（误）。**

在GhostNet这篇论文里面，作者认为可以使用一些**计算量更低（Cheap Operations）的操作去生成这些冗余的特征图**，这样就可以在保证良好检测效果的情况下，减少模型的参数量与提高模型的执行速度。

# 源码下载 #

https://github.com/bubbliiiing/mobilenet-yolov4-lite-keras

# GhostNet模型的实现思路 #

## 1、Ghost Module ##

通过上述的介绍，我们了解到了，**GhostNet的核心思想就是**使用一些**计算量更低（Cheap Operations）的操作去生成这些冗余的特征图**。

在论文中，作者设计了一个名为Ghost Module的模块，**他的功能是代替普通卷积**。

Ghost Module将普通卷积分为两部分，首先进行一个**普通的1x1卷积**，这是一个少量卷积，比如**正常使用32通道的卷积，这里就用16通道的卷积**，这个1x1卷积的作用类似于特征整合，**生成输入特征层的特征浓缩**。

然后我们再进行**深度可分离卷积**，这个**深度可分离卷积是逐层卷积**，它也就是我们上面提到的**Cheap Operations**。它利用上一步获得的**特征浓缩**生成**Ghost特征图**。

因此，如果我们从整体上去看这个Ghost Module，它其实就是两步简单思想的汇总：  
1、利用**1x1卷积获得输入特征的必要特征浓缩**。  
2、利用**深度可分离卷积获得特征浓缩的相似特征图**（Ghost）。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 2]  
Ghost Module的实现代码如下：

def _ghost_module(inputs, exp, kernel, dw_kernel, ratio, strides=1,
                        padding='same',use_bias=False, relu=True):
        output_channels = math.ceil(exp * 1.0 / ratio)
    
        x = Conv2D(output_channels, kernel, strides=strides, padding=padding, use_bias=use_bias)(inputs)
        x = BatchNormalization()(x)
        if relu:
            x = Activation('relu')(x)
        dw = DepthwiseConv2D(dw_kernel, strides, padding=padding, depth_multiplier=ratio-1, use_bias=use_bias)(x)
        dw = BatchNormalization()(dw)
        if relu:
            dw = Activation('relu')(dw)
    
        x = Concatenate(axis=-1)([x,dw])
        x = Lambda(slices, arguments={
        'n':exp})(x)
    
        return x

## 2、Ghost Bottlenecks ##

Ghost Bottlenecks是由Ghost Module组成的瓶颈结构，就像这样。

其实本质上就是用Ghost Module，来代替瓶颈结构里面的普通卷积。

**Ghost Bottlenecks可以分为两个部分，分别是主干部分和残差边部分，包含Ghost Module的，我们称它为主干部分。**

Ghost Bottlenecks有两个种类，如下图所示，**当我们需要对特征层的宽高进行压缩的时候，我们会设置这个Ghost Bottlenecks的Stride=2，即步长为2**。此时我们会Bottlenecks里面多添加一些卷积层，在主干部分里，我们会在两个Ghost Module中**添加一个步长为2x2的深度可分离卷积进行特征层的宽高压缩**。在残差边部分，我们也会添加上一个**步长为2x2的深度可分离卷积**和1x1的普通卷积。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 3]  
Ghost Bottlenecks的实现代码如下：

def _ghost_bottleneck(inputs, output_channel, hidden_channel, kernel, ghost_kernel, strides, ratio, squeeze):
    
        input_shape = K.int_shape(inputs)       # 获取输入张量的尺寸
    
        x = _ghost_module(inputs, hidden_channel, [1,1], ghost_kernel, ratio)
    
        if strides > 1:
            x = DepthwiseConv2D(kernel, strides, padding='same', depth_multiplier=1, use_bias=False)(x)
            x = BatchNormalization()(x)
    
        if squeeze:
            x = _squeeze(x, hidden_channel, 4)
    
        x = _ghost_module(x, output_channel, [1,1], ghost_kernel, ratio, relu=False)
    
        if strides == 1 and input_shape[-1] == output_channel:
            res = inputs
        else:
            res = DepthwiseConv2D(kernel, strides=strides, padding='same', depth_multiplier=1, use_bias=False)(inputs)
            res = BatchNormalization()(res)
            res = Conv2D(output_channel, (1, 1), padding='same', strides=(1, 1), use_bias=False)(res)
            res = BatchNormalization()(res)
        x = Add()([res, x])
    
        return x

## 3、Ghostnet的构建 ##

整个Ghostnet的构建方式如列表所示：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 4]  
可以看到，整个Ghostnet都是由Ghost Bottlenecks进行组成的。

当一张图片输入到Ghostnet当中时，**我们首先进行一个16通道的普通1x1卷积块（卷积+标准化+激活函数）**。

之后我们就开始Ghost Bottlenecks的堆叠了，**利用Ghost Bottlenecks，我们最终获得了一个7x7x160的特征层（当输入是224x224x3的时候）**。

然后我们**会利用一个1x1的卷积块进行通道数的调整，此时我们可以获得一个7x7x960的特征层。**

之后我们**进行一次全局平均池化，然后再利用一个1x1的卷积块进行通道数的调整**，获得一个1x1x1280的特征层。

**然后平铺后进行全连接就可以进行分类了。**

# GhostNet的代码构建 #

## 1、模型代码的构建 ##

GhostNet的实现代码如下，该代码是Ghostnet在YoloV4上的应用，可以参考一下：

import math
    import warnings
    
    import numpy as np
    import tensorflow as tf
    from keras import backend as K
    from keras.applications.imagenet_utils import decode_predictions
    from keras.initializers import random_normal
    from keras.layers import (Activation, Add, BatchNormalization, Concatenate,
                              Conv2D, DepthwiseConv2D, GlobalAveragePooling2D, 
                              Lambda, Multiply, Reshape)
    
    
    def slices(dw, n):
        return dw[:,:,:,:n]
    
    def _make_divisible(v, divisor, min_value=None):
        if min_value is None:
            min_value = divisor
        new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)
        if new_v < 0.9 * v:
            new_v += divisor
        return new_v
        
    def _squeeze(inputs, hidden_channel, ratio, block_id, sub_block_id):
        x = GlobalAveragePooling2D()(inputs)
        x = Reshape((1,1,-1))(x)
    
        x = Conv2D(_make_divisible(hidden_channel/ratio, 4), (1,1), strides=(1,1), padding='same', kernel_initializer=random_normal(stddev=0.02),
                name="blocks."+str(block_id)+"."+str(sub_block_id)+".se.conv_reduce")(x)
        x = Activation('relu')(x)
    
        x = Conv2D(hidden_channel, (1,1),strides=(1,1), padding='same', kernel_initializer=random_normal(stddev=0.02),
                name="blocks."+str(block_id)+"."+str(sub_block_id)+".se.conv_expand")(x)
        x = Activation('hard_sigmoid')(x)
    
        x = Multiply()([inputs, x])    # inputs和x逐元素相乘
        return x
    
    def _ghost_module(inputs, exp, ratio, block_id, sub_block_id, part, kernel_size=1, dw_size=3, stride=1, relu=True):
        output_channels = math.ceil(exp * 1.0 / ratio)
    
        x = Conv2D(output_channels, kernel_size, strides=stride, padding="same", use_bias=False, kernel_initializer=random_normal(stddev=0.02),
                name="blocks."+str(block_id)+"."+str(sub_block_id)+".ghost"+str(part)+".primary_conv.0")(inputs)
        x = BatchNormalization(name="blocks."+str(block_id)+"."+str(sub_block_id)+".ghost"+str(part)+".primary_conv.1")(x)
        if relu:
            x = Activation('relu')(x)
    
        dw = DepthwiseConv2D(dw_size, 1, padding="same", depth_multiplier=ratio-1, use_bias=False, depthwise_initializer=random_normal(stddev=0.02), 
                name="blocks."+str(block_id)+"."+str(sub_block_id)+".ghost"+str(part)+".cheap_operation.0")(x)
        dw = BatchNormalization(name="blocks."+str(block_id)+"."+str(sub_block_id)+".ghost"+str(part)+".cheap_operation.1")(dw)
        if relu:
            dw = Activation('relu')(dw)
    
        x = Concatenate(axis=-1)([x,dw])
        x = Lambda(slices, arguments={
        'n':exp})(x)
        return x
    
    def _ghost_bottleneck(inputs, output_channel, hidden_channel, kernel, strides, ratio, squeeze, block_id, sub_block_id):
        input_shape = K.int_shape(inputs)
    
        x = _ghost_module(inputs, hidden_channel, ratio, block_id, sub_block_id, 1)
    
        if strides > 1:
            x = DepthwiseConv2D(kernel, strides, padding='same', depth_multiplier=1, use_bias=False, depthwise_initializer=random_normal(stddev=0.02),
                    name="blocks."+str(block_id)+"."+str(sub_block_id)+".conv_dw")(x)
            x = BatchNormalization(name="blocks."+str(block_id)+"."+str(sub_block_id)+".bn_dw")(x)
    
        if squeeze:
            x = _squeeze(x, hidden_channel, 4, block_id, sub_block_id)
    
        x = _ghost_module(x, output_channel, ratio, block_id, sub_block_id, 2, relu=False)
    
        if strides == 1 and input_shape[-1] == output_channel:
            res = inputs
        else:
            res = DepthwiseConv2D(kernel, strides=strides, padding='same', depth_multiplier=1, use_bias=False, depthwise_initializer=random_normal(stddev=0.02),
                    name="blocks."+str(block_id)+"."+str(sub_block_id)+".shortcut.0")(inputs)
            res = BatchNormalization(name="blocks."+str(block_id)+"."+str(sub_block_id)+".shortcut.1")(res)
            res = Conv2D(output_channel, (1, 1), padding='same', strides=(1, 1), use_bias=False, kernel_initializer=random_normal(stddev=0.02),
                    name="blocks."+str(block_id)+"."+str(sub_block_id)+".shortcut.2")(res)
            res = BatchNormalization(name="blocks."+str(block_id)+"."+str(sub_block_id)+".shortcut.3")(res)
        x = Add()([res, x])
    
        return x
    
    def Ghostnet(inputs):
        x = Conv2D(16, (3, 3), padding="same", strides=(2, 2), use_bias=False, kernel_initializer=random_normal(stddev=0.02), name="conv_stem")(inputs)
        x = BatchNormalization(name="bn1")(x)
        x = Activation('relu')(x)
    
        x = _ghost_bottleneck(x, 16, 16, (3, 3), strides=1, ratio=2, squeeze=False, block_id=0, sub_block_id=0)
        
        x = _ghost_bottleneck(x, 24, 48, (3, 3), strides=2, ratio=2, squeeze=False, block_id=1, sub_block_id=0)
        x = _ghost_bottleneck(x, 24, 72, (3, 3), strides=1, ratio=2, squeeze=False, block_id=2, sub_block_id=0)
    
        x = _ghost_bottleneck(x, 40, 72, (5, 5), strides=2, ratio=2, squeeze=True, block_id=3, sub_block_id=0)
        x = _ghost_bottleneck(x, 40, 120, (5, 5), strides=1, ratio=2, squeeze=True, block_id=4, sub_block_id=0)
        feat1 = x
    
        x = _ghost_bottleneck(x, 80, 240, (3, 3), strides=2, ratio=2, squeeze=False, block_id=5, sub_block_id=0)
        x = _ghost_bottleneck(x, 80, 200, (3, 3), strides=1, ratio=2, squeeze=False, block_id=6, sub_block_id=0)
        x = _ghost_bottleneck(x, 80, 184, (3, 3), strides=1, ratio=2, squeeze=False, block_id=6, sub_block_id=1)
        x = _ghost_bottleneck(x, 80, 184, (3, 3), strides=1, ratio=2, squeeze=False, block_id=6, sub_block_id=2)
        x = _ghost_bottleneck(x, 112, 480, (3, 3), strides=1, ratio=2, squeeze=True, block_id=6, sub_block_id=3)
        x = _ghost_bottleneck(x, 112, 672, (3, 3), strides=1, ratio=2, squeeze=True, block_id=6, sub_block_id=4)
        feat2 = x
    
        x = _ghost_bottleneck(x, 160, 672, (5, 5), strides=2, ratio=2, squeeze=True, block_id=7, sub_block_id=0)
        x = _ghost_bottleneck(x, 160, 960, (5, 5), strides=1, ratio=2, squeeze=False, block_id=8, sub_block_id=0)
        x = _ghost_bottleneck(x, 160, 960, (5, 5), strides=1, ratio=2, squeeze=True, block_id=8, sub_block_id=1)
        x = _ghost_bottleneck(x, 160, 960, (5, 5), strides=1, ratio=2, squeeze=False, block_id=8, sub_block_id=2)
        x = _ghost_bottleneck(x, 160, 960, (5, 5), strides=1, ratio=2, squeeze=True, block_id=8, sub_block_id=3)
        feat3 = x
    
        return feat1,feat2,feat3

## 2、Yolov4上的应用 ##

作为一个轻量级网络，我把Ghostnet和Mobilenet放在一起，作为Yolov4的主干网络进行特征提取。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_cneter]  
对于yolov4来讲，我们需要利用主干特征提取网络获得的**三个有效特征进行加强特征金字塔的构建**。

我们通过上述代码可以取出三个有效特征层，我们可以利用这三个有效特征层替换原来yolov4主干网络CSPdarknet53的有效特征层。

**为了进一步减少参数量，我们可以使用深度可分离卷积代替yoloV3中用到的普通卷积。**

最终Ghostnet-Yolov4的构建代码如下：

from functools import wraps
    
    from keras import backend as K
    from keras.initializers import random_normal
    from keras.layers import (Activation, BatchNormalization, Concatenate, Conv2D,
                              DepthwiseConv2D, Input, Lambda, MaxPooling2D,
                              UpSampling2D)
    from keras.layers.normalization import BatchNormalization
    from keras.models import Model
    from keras.regularizers import l2
    from utils.utils import compose
    
    from nets.ghostnet import Ghostnet
    from nets.mobilenet_v1 import MobileNetV1
    from nets.mobilenet_v2 import MobileNetV2
    from nets.mobilenet_v3 import MobileNetV3
    from nets.yolo_training import yolo_loss
    
    
    def relu6(x):
        return K.relu(x, max_value=6)
        
    #------------------------------------------------------#
    #   单次卷积DarknetConv2D
    #   如果步长为2则自己设定padding方式。
    #------------------------------------------------------#
    @wraps(Conv2D)
    def DarknetConv2D(*args, **kwargs):
        darknet_conv_kwargs = {
        'kernel_initializer' : random_normal(stddev=0.02), 'kernel_regularizer': l2(5e-4)}
        darknet_conv_kwargs['padding'] = 'valid' if kwargs.get('strides')==(2,2) else 'same'
        darknet_conv_kwargs.update(kwargs)
        return Conv2D(*args, **darknet_conv_kwargs)
    
    #---------------------------------------------------#
    #   卷积块 -> 卷积 + 标准化 + 激活函数
    #   DarknetConv2D + BatchNormalization + Relu6
    #---------------------------------------------------#
    def DarknetConv2D_BN_Leaky(*args, **kwargs):
        no_bias_kwargs = {
        'use_bias': False}
        no_bias_kwargs.update(kwargs)
        return compose( 
            DarknetConv2D(*args, **no_bias_kwargs),
            BatchNormalization(),
            Activation(relu6))
    
    #---------------------------------------------------#
    #   深度可分离卷积块
    #   DepthwiseConv2D + BatchNormalization + Relu6
    #---------------------------------------------------#
    def _depthwise_conv_block(inputs, pointwise_conv_filters, alpha = 1,
                              depth_multiplier=1, strides=(1, 1)):
    
        pointwise_conv_filters = int(pointwise_conv_filters * alpha)
        
        x = DepthwiseConv2D((3, 3), depthwise_initializer=random_normal(stddev=0.02),
                            padding='same',
                            depth_multiplier=depth_multiplier,
                            strides=strides,
                            use_bias=False)(inputs)
    
        x = BatchNormalization()(x)
        x = Activation(relu6)(x)
    
        x = DarknetConv2D(pointwise_conv_filters, (1, 1), 
                        padding='same',
                        use_bias=False,
                        strides=(1, 1))(x)
        x = BatchNormalization()(x)
        return Activation(relu6)(x)
        
    #---------------------------------------------------#
    #   进行五次卷积
    #---------------------------------------------------#
    def make_five_convs(x, num_filters):
        # 五次卷积
        x = DarknetConv2D_BN_Leaky(num_filters, (1,1))(x)
        x = _depthwise_conv_block(x, num_filters*2,alpha=1)
        x = DarknetConv2D_BN_Leaky(num_filters, (1,1))(x)
        x = _depthwise_conv_block(x, num_filters*2,alpha=1)
        x = DarknetConv2D_BN_Leaky(num_filters, (1,1))(x)
        return x
    
    #---------------------------------------------------#
    #   Panet网络的构建，并且获得预测结果
    #---------------------------------------------------#
    def yolo_body(input_shape, anchors_mask, num_classes, backbone="mobilenetv1", alpha=1):
        inputs = Input(input_shape)
        #---------------------------------------------------#   
        #   生成mobilnet的主干模型，获得三个有效特征层。
        #---------------------------------------------------#
        if backbone=="mobilenetv1":
            #---------------------------------------------------#   
            #   52,52,256；26,26,512；13,13,1024
            #---------------------------------------------------#
            feat1,feat2,feat3 = MobileNetV1(inputs, alpha=alpha)
        elif backbone=="mobilenetv2":
            #---------------------------------------------------#   
            #   52,52,32；26,26,92；13,13,320
            #---------------------------------------------------#
            feat1,feat2,feat3 = MobileNetV2(inputs, alpha=alpha)
        elif backbone=="mobilenetv3":
            #---------------------------------------------------#   
            #   52,52,40；26,26,112；13,13,160
            #---------------------------------------------------#
            feat1,feat2,feat3 = MobileNetV3(inputs, alpha=alpha)
        elif backbone=="ghostnet":
            #---------------------------------------------------#   
            #   52,52,40；26,26,112；13,13,160
            #---------------------------------------------------#
            feat1,feat2,feat3 = Ghostnet(inputs)
        else:
            raise ValueError('Unsupported backbone - `{}`, Use mobilenetv1, mobilenetv2, mobilenetv3, ghostnet.'.format(backbone))
        
        P5 = DarknetConv2D_BN_Leaky(int(512* alpha), (1,1))(feat3)
        P5 = _depthwise_conv_block(P5, int(1024* alpha))
        P5 = DarknetConv2D_BN_Leaky(int(512* alpha), (1,1))(P5)
        maxpool1 = MaxPooling2D(pool_size=(13,13), strides=(1,1), padding='same')(P5)
        maxpool2 = MaxPooling2D(pool_size=(9,9), strides=(1,1), padding='same')(P5)
        maxpool3 = MaxPooling2D(pool_size=(5,5), strides=(1,1), padding='same')(P5)
        P5 = Concatenate()([maxpool1, maxpool2, maxpool3, P5])
        P5 = DarknetConv2D_BN_Leaky(int(512* alpha), (1,1))(P5)
        P5 = _depthwise_conv_block(P5, int(1024* alpha))
        P5 = DarknetConv2D_BN_Leaky(int(512* alpha), (1,1))(P5)
    
        P5_upsample = compose(DarknetConv2D_BN_Leaky(int(256* alpha), (1,1)), UpSampling2D(2))(P5)
        
        P4 = DarknetConv2D_BN_Leaky(int(256* alpha), (1,1))(feat2)
        P4 = Concatenate()([P4, P5_upsample])
        P4 = make_five_convs(P4,int(256* alpha))
    
        P4_upsample = compose(DarknetConv2D_BN_Leaky(int(128* alpha), (1,1)), UpSampling2D(2))(P4)
        
        P3 = DarknetConv2D_BN_Leaky(int(128* alpha), (1,1))(feat1)
        P3 = Concatenate()([P3, P4_upsample])
        P3 = make_five_convs(P3,int(128* alpha))
    
        #---------------------------------------------------#
        #   第三个特征层
        #   y3=(batch_size,52,52,3,85)
        #---------------------------------------------------#
        P3_output = _depthwise_conv_block(P3, int(256* alpha))
        P3_output = DarknetConv2D(len(anchors_mask[0])*(num_classes+5), (1,1))(P3_output)
    
        P3_downsample = _depthwise_conv_block(P3, int(256* alpha), strides=(2,2))
        P4 = Concatenate()([P3_downsample, P4])
        P4 = make_five_convs(P4,int(256* alpha))
        
        #---------------------------------------------------#
        #   第二个特征层
        #   y2=(batch_size,26,26,3,85)
        #---------------------------------------------------#
        P4_output = _depthwise_conv_block(P4, int(512* alpha))
        P4_output = DarknetConv2D(len(anchors_mask[1])*(num_classes+5), (1,1))(P4_output)
    
        P4_downsample = _depthwise_conv_block(P4, int(512* alpha), strides=(2,2))
        P5 = Concatenate()([P4_downsample, P5])
        P5 = make_five_convs(P5,int(512* alpha))
        
        #---------------------------------------------------#
        #   第一个特征层
        #   y1=(batch_size,13,13,3,85)
        #---------------------------------------------------#
        P5_output = _depthwise_conv_block(P5, int(1024* alpha))
        P5_output = DarknetConv2D(len(anchors_mask[2])*(num_classes+5), (1,1))(P5_output)
    
        return Model(inputs, [P5_output, P4_output, P3_output])

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/ba9bf02e9aeb466aaa76ab5cbbd7c75e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221120/39a8ae305ae24a439de68043b00360fe.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221120/c5a96ccd17b24d5bb047267cd8f554d0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221120/37a493aae7434d449f71520e10cc555d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_center 4]: /images/20221120/9affecd4616c4d899985252a9e99735a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc5MTk2NA_size_16_color_FFFFFF_t_70_pic_cneter]: /images/20221120/fffb6f0c990948f3891b917e8027a48d.png