k8s--kubeadm 搭建k8s

我会带着你远行 2022-08-30 04:10 315阅读 0赞

## 一、Kubernetes概述 ##

### 1.1 Kubernetes是什么 ###

Kubernetes是Google在2014年开源的一个容器集群管理系统，Kubernetes简称K8S。

K8S用于容器化应用程序的部署，扩展和管理。

K8S提供了容器编排，资源调度，弹性伸缩，部署管理，服务发现等一系列功能。

Kubernetes目标是让部署容器化应用简单高效。

### 1.2 Kubernetes特性 ###

自我修复

在节点故障时重新启动失败的容器，替换和重新部署，保证预期的副本数量；杀死健康检查失败的容器，并且在未准备好之前不会处理客户端请求，确保线上服务不中断。

弹性伸缩

使用命令、UI或者基于CPU使用情况自动快速扩容和缩容应用程序实例，保证应用业务高峰并发时的高可用性；业务低峰时回收资源，以最小成本运行服务。

自动部署和回滚

K8S采用滚动更新策略更新应用，一次更新一个Pod，而不是同时删除所有Pod，如果更新过程中出现问题，将回滚更改，确保升级不受影响业务。

服务发现和负载均衡

K8S为多个容器提供一个统一访问入口（内部IP地址和一个DNS名称），并且负载均衡关联的所有容器，使得用户无需考虑容器IP问题。

机密和配置管理

管理机密数据和应用程序配置，而不需要把敏感数据暴露在镜像里，提高敏感数据安全性。并可以将一些常用的配置存储在K8S中，方便应用程序使用。

存储编排

挂载外部存储系统，无论是来自本地存储，公有云（如AWS），还是网络存储（如NFS、GlusterFS、Ceph）都作为集群资源的一部分使用，极大提高存储使用灵活性。

批处理

提供一次性任务，定时任务；满足批量数据处理和分析的场景。

### 1.3 Kubernetes集群架构与组件 ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70]

### 1.4 Kubernetes集群组件介绍 ###

#### 1.4.1 Master组件 ####

kube-apiserver

Kubernetes API，

集群的统一入口  
，各组件协调者，以RESTful API提供接口服务，所有对象资源的增删改查和监听操作都交给APIServer处理后再提交给Etcd存储。

kube-controller-manager

处理集群中常规后台任务，一个资源对应一个控制器，而ControllerManager就是负责管理这些控制器的。

kube-scheduler

根据调度算法为新创建的Pod选择一个Node节点，可以任意部署,可以部署在同一个节点上,也可以部署在不同的节点上。

etcd

分布式键值存储系统。用于保存集群状态数据，比如Pod、Service等对象信息。

1.4.2 Node组件  
kubelet

kubelet是Master在Node节点上的Agent，管理本机运行容器的生命周期，比如创建容器、Pod挂载数据卷、下载secret、获取容器和节点状态等工作。kubelet将每个Pod转换成一组容器。

kube-proxy

在Node节点上实现Pod网络代理，维护网络规则和四层负载均衡工作。

docker或rocket

容器引擎，运行容器。

#### 1.5 Kubernetes 核心概念 ####

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70 1]

##### Pod #####

最小部署单元

一组容器的集合

一个Pod中的容器共享网络命名空间

Pod是短暂的

##### Controllers #####

*  ReplicaSet ：确保预期的Pod副本数量
 *  Deployment ：无状态应用部署
 *  StatefulSet ：有状态应用部署
 *  DaemonSet ：确保所有Node运行同一个Pod
 *  Job ：一次性任务
 *  Cronjob ：定时任务

##### 更高级层次对象，部署和管理Pod #####

*  Service  
    防止Pod失联  
    定义一组Pod的访问策略  
    Label ：标签，附加到某个资源上，用于关联对象、查询和筛选
 *  Namespaces：命名空间，将对象逻辑上隔离
 *  Annotations ：注释

## 二、kubeadm 快速部署K8S集群 ##

### 2.1 kubernetes 官方提供的三种部署方式 ###

minikube

Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，仅用于尝试Kubernetes或日常开发的用户使用。部署地址：https://kubernetes.io/docs/setup/minikube/

kubeadm

Kubeadm也是一个工具，提供kubeadm init和kubeadm join，用于快速部署Kubernetes集群。部署地址：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/

二进制包

推荐，从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。下载地址：https://github.com/kubernetes/kubernetes/releases

### 2.2 安装kubeadm环境准备 ###

以下操作,在三台节点都执行

#### 2.2.1 环境需求 ####

环境：centos 7.4 +

硬件需求：CPU>=2c ,内存>=2G

#### 2.2.2 环境角色 ####

IP 角色 安装软件

#### 2.2.3 环境初始化 ####

PS : 以下所有操作，在三台节点全部执行

1、关闭防火墙及selinux

systemctl stop firewalld && systemctl disable firewalld
    sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config  && setenforce 0

2、关闭 swap 分区

swapoff -a # 临时

sed -i '/ swap / s/^$.*$$/#\1/g' /etc/fstab #永久

设置/etc/hosts

cat >> /etc/hosts << EOF
     192.168.48.61 master
     192.168.48.62 node01
     192.168.48.63 node02
     EOF

内核调整,将桥接的IPv4流量传递到iptables的链

cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF

sysctl --system

设置系统时区并同步时间服务器

yum install -y ntpdate
    ntpdate time.windows.com

#### 2.2.4 docker 安装 ####

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
    yum -y install docker-ce-18.06.1.ce-3.el7
    systemctl enable docker && systemctl start docker
    docker --version
    Docker version 18.06.1-ce, build e68fc7a

docker 初始化

cat > /etc/docker/daemon.json << EOF { "data-root":"/data/docker", "insecure-registries":["registry.access.redhat.com","quay.io"], "exec-opts":["native.cgroupdriver=systemd"], "live-restore":true, "bip":"172.0.100.1/24", "registry-mirrors":["https://q2gr04ke.mirror.aliyuncs.com"] } EOF

“exec-opts”:\[“native.cgroupdriver=systemd”\] 这个配置项对于docker很重要

#### 2.2.5 添加kubernetes YUM软件源 ####

cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF

#### 2.2.6 安装kubeadm,kubelet和kubectl ####

yum install -y kubelet-1.15.0 kubeadm-1.15.0 kubectl-1.15.0
     systemctl enable kubelet

### 2.3 部署Kubernetes Master ###

kubeadm init \
    --apiserver-advertise-address=192.168.48.61 \
    --image-repository registry.aliyuncs.com/google_containers \
    --kubernetes-version v1.15.0 \
    --service-cidr=10.1.0.0/16 \
    --pod-network-cidr=10.244.0.0/16

如果对init的结果不满意，可使用kubeadm reset 进行重置，并且停掉相关的进程，然后再次使用上面的命令。

[init] Using Kubernetes version: v1.15.0
    [preflight] Running pre-flight checks
    [preflight] Pulling images required for setting up a Kubernetes cluster
    [preflight] This might take a minute or two, depending on the speed of your internet connection
    [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
    [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
    [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
    [kubelet-start] Activating the kubelet service
    [certs] Using certificateDir folder "/etc/kubernetes/pki"
    [certs] Generating "ca" certificate and key
    [certs] Generating "apiserver" certificate and key
    [certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.48.61]
    [certs] Generating "apiserver-kubelet-client" certificate and key
    [certs] Generating "front-proxy-ca" certificate and key
    [certs] Generating "front-proxy-client" certificate and key
    [certs] Generating "etcd/ca" certificate and key
    [certs] Generating "etcd/peer" certificate and key
    [certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.48.61 127.0.0.1 ::1]
    [certs] Generating "etcd/healthcheck-client" certificate and key
    [certs] Generating "apiserver-etcd-client" certificate and key
    [certs] Generating "etcd/server" certificate and key
    [certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.48.61 127.0.0.1 ::1]
    [certs] Generating "sa" key and public key
    [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
    [kubeconfig] Writing "admin.conf" kubeconfig file
    [kubeconfig] Writing "kubelet.conf" kubeconfig file
    [kubeconfig] Writing "controller-manager.conf" kubeconfig file
    [kubeconfig] Writing "scheduler.conf" kubeconfig file
    [control-plane] Using manifest folder "/etc/kubernetes/manifests"
    [control-plane] Creating static Pod manifest for "kube-apiserver"
    [control-plane] Creating static Pod manifest for "kube-controller-manager"
    [control-plane] Creating static Pod manifest for "kube-scheduler"
    [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
    [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
    [apiclient] All control plane components are healthy after 20.003894 seconds
    [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
    [kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
    [upload-certs] Skipping phase. Please see --upload-certs
    [mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
    [mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
    [bootstrap-token] Using token: iv8whr.mz4mwki34oeyb985
    [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
    [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
    [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
    [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
    [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
    [addons] Applied essential addon: CoreDNS
    [addons] Applied essential addon: kube-proxy
    
    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 192.168.48.61:6443 --token iv8whr.mz4mwki34oeyb985 \
        --discovery-token-ca-cert-hash sha256:7e54dfe36303dc5beb3bbcaf2aafe0edb24b8f84f7632cb647c9c0817edbd2a9

根据提示进行如下操作

mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

### 2.4节点加入集群 ###

默认token的有效期为24小时，当过期之后，该token就不可用了，如果后续有nodes节点加入，解决方法如下：  
重新生成新的token

kubeadm token create

kubeadm token list

获取ca证书sha256编码hash值

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //

节点加入集群

kubeadm join --token aa78f6.8b4cafc8ed26c34f --discovery-token-ca-cert-hash sha256:0fd95a9bc67a7bf0ef42da968a0d55d92e52898ec37c971bd77ee501d845b538 192.168.73.138:6443 --skip-preflight-chec

kubeadm join --token 9xdks8.0mn3jot1pzk4c6iw --discovery-token-ca-cert-hash sha256:7e54dfe36303dc5beb3bbcaf2aafe0edb24b8f84f7632cb647c9c0817edbd2a9 192.168.48.61:6443 --skip-preflight-chec

### 2.5 安装flannel ###

只需要在Master 节点执行

wget https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

kubectl apply -f kube-flannel.yml

### 2.6 测试k8s 集群 ###

[root@k8s-master ~]# kubectl create deployment nginx --image=nginx
    deployment.apps/nginx created
    
    [root@k8s-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort
    service/nginx exposed
    
    [root@k8s-master ~]# kubectl get pods,svc
    NAME                         READY   STATUS    RESTARTS   AGE
    pod/nginx-554b9c67f9-wf5lm   1/1     Running   0          24s
    
    NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
    service/kubernetes   ClusterIP   10.1.0.1       <none>        443/TCP        39m
    service/nginx        NodePort    10.1.224.251   <none>        80:31745/TCP   9

### 2.7 安装dashboard ###

[root@k8s-master ~]# wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
    
    [root@k8s-master ~]# vim kubernetes-dashboard.yaml
    修改内容：
    109     spec:
    110       containers:
    111       - name: kubernetes-dashboard
    112         image: lizhenliang/kubernetes-dashboard-amd64:v1.10.1   # 修改此行
    
    ......
    
    157 spec:
    158   type: NodePort     # 增加此行
    159   ports:
    160     - port: 443
    161       targetPort: 8443
    162       nodePort: 30001   # 增加此行
    163   selector:
    164     k8s-app: kubernetes-dashboard
    
    [root@k8s-master ~]# kubectl apply -f kubernetes-dashboard.yaml

创建service account并绑定默认cluster-admin管理员集群角色：

[root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
    serviceaccount/dashboard-admin created
    [root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin
    --serviceaccount=kube-system:dashboard-admin
    [root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
    Name:         dashboard-admin-token-d9jh2
    Namespace:    kube-system
    Labels:       <none>
    Annotations:  kubernetes.io/service-account.name: dashboard-admin
                  kubernetes.io/service-account.uid: 4aa1906e-17aa-4880-b848-8b3959483323
    
    Type:  kubernetes.io/service-account-token
    
    Data
    ====
    ca.crt:     1025 bytes
    namespace:  11 bytes
    token:      eyJhbGciOiJ...(省略如下)...AJdQ
    
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tZDlqaDIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGFhMTkwNmUtMTdhYS00ODgwLWI4NDgtOGIzOTU5NDgzMzIzIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.OkF6h7tVQqmNJniCHJhY02G6u6dRg0V8PTiF8xvMuJJUphLyWlWctgmplM4kjKVZo0fZkAthL7WAV5p_AwAuj4LMfo1X5IpxUomp4YZyhqgsBM0A2ksWoKoLDjbizFwOty8TylWlsX1xcJXZjmP9OvNgjjSq5J90N5PnxYIIgwAMP3fawTP7kUXxz5WhJo-ogCijJCFyYBHoqHrgAbk9pusI8DpGTNIZxBMxkwPPwFwzNCOfKhD0c8HjhNeliKsOYLryZObRdmTQXmxsDfxynTKsRxv_EPQb99yW9GXJPQL0OwpYb4b164CFv857ENitvvKEOU6y55P9hFkuQuAJdQ

当匿名账户无法查看看k8s集群中的某些资源时，需要执行以下命令：

kubectl create clusterrolebinding test:anonymous --clusterrole=cluster-admin --user=system:anonymous

在火狐浏览器访问(google受信任问题不能访问)地址： https://NodeIP:30001  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70 2]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70 3]  
查看dashboard 时需要的token

kubectl -n kube-system describe $(kubectl -n kube-system get secret -n kube-system -o name | grep namespace) | grep token

解决只能火狐访问的问题

[root@k8s-master ~]# cd /etc/kubernetes/pki/
    [root@k8s-master pki]# mkdir ui
    [root@k8s-master pki]# cp apiserver.crt ui/
    [root@k8s-master pki]# cp apiserver.key ui/
    [root@k8s-master pki]# cd ui/
    [root@k8s-master ui]# mv apiserver.crt dashboard.pem
    [root@k8s-master ui]# mv apiserver.key dashboard-key.pem
    [root@k8s-master ui]# kubectl delete secret kubernetes-dashboard-certs -n kube-system
    [root@k8s-master ui]# kubectl create secret generic kubernetes-dashboard-certs --from-file=./ -n kube-system

[root@k8s-master]# vim kubernetes-dashboard.yaml #回到这个yaml的路径下修改
    修改 dashboard-controller.yaml 文件，在args下面增加证书两行
              - --tls-key-file=dashboard-key.pem
              - --tls-cert-file=dashboard.pem
    [root@k8s-master ~]kubectl apply -f kubernetes-dashboard.yaml
    [root@k8s-master ~]# kubectl create serviceaccount dashboard-admin -n kube-system
    serviceaccount/dashboard-admin created
    [root@k8s-master ~]# kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin
    --serviceaccount=kube-system:dashboard-admin
    [root@k8s-master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
    Name:         dashboard-admin-token-zbn9f
    Namespace:    kube-system
    Labels:       <none>
    Annotations:  kubernetes.io/service-account.name: dashboard-admin
                  kubernetes.io/service-account.uid: 40259d83-3b4f-4acc-a4fb-43018de7fc19
    
    Type:  kubernetes.io/service-account-token
    
    Data
    ====
    namespace:  11 bytes
    token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4temJuOWYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDAyNTlkODMtM2I0Zi00YWNjLWE0ZmItNDMwMThkZTdmYzE5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.E0hGAkeQxd6K-YpPgJmNTv7Sn_P_nzhgCnYXGc9AeXd9k9qAcO97vBeOV-pH518YbjrOAx_D6CKIyP07aCi_3NoPlbbyHtcpRKFl-lWDPdg8wpcIefcpbtS6uCOrpaJdCJjWFcAEHdvcfmiFpdVVT7tUZ2-eHpRTUQ5MDPF-c2IOa9_FC9V3bf6XW6MSCZ_7-fOF4MnfYRa8ucltEIhIhCAeDyxlopSaA5oEbopjaNiVeJUGrKBll8Edatc7-wauUIJXAN-dZRD0xTULPNJ1BsBthGQLyFe8OpL5n_oiHM40tISJYU_uQRlMP83SfkOpbiOpzuDT59BBJB57OQtl3w
    ca.crt:     1025 bytes

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70]: /images/20220829/44674cb05709400fb0aa832638ac151c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70 1]: /images/20220829/a10e8269c65b4512b8781d12a4387f8f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70 2]: /images/20220829/4d13de8ed6c44f9989afe8b7d4e47b55.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzI2NDk2ODc3_size_16_color_FFFFFF_t_70 3]: /images/20220829/a259604c8b5b45d6b7487b3d5c0f065d.png