Jenkins+harbor+gitlab+k8s 部署maven项目

左手的ㄟ右手 2024-04-17 19:52 73阅读 0赞

# 一、概述 #

maven项目部署流程图如下：

![1341090-20190830144305422-1548071491.png][]

## 环境介绍 ##

<table> 
 <tbody> 
  <tr> 
   <td>操作系统</td> 
   <td>ip</td> 
   <td>角色</td> 
   <td>版本</td> 
  </tr> 
  <tr> 
   <td>ubuntu-16.04.4-server-amd64</td> 
   <td>192.168.10.122</td> 
   <td>Jenkins+harbor</td> 
   <td>Jenkins 2.176.2，harbor 1.8.1</td> 
  </tr> 
  <tr> 
   <td>ubuntu-16.04.4-server-amd64</td> 
   <td>192.168.10.134</td> 
   <td>gitlab</td> 
   <td>gitlab-ce_12.1.4</td> 
  </tr> 
  <tr> 
   <td>ubuntu-16.04.4-server-amd64</td> 
   <td>192.168.10.130</td> 
   <td>k8s master</td> 
   <td>kubernetes 1.15.2</td> 
  </tr> 
  <tr> 
   <td>ubuntu-16.04.4-server-amd64</td> 
   <td>192.168.10.131</td> 
   <td>k8s node</td> 
   <td>kubernetes 1.15.2</td> 
  </tr> 
 </tbody> 
</table>

由于资源紧张，Jenkins+harbor合并为一台了。实际上，应该是要单独部署的。

k8s安装，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11317844.html][https_www.cnblogs.com_xiao987334176_p_11317844.html]

gitlab安装，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11329018.html][https_www.cnblogs.com_xiao987334176_p_11329018.html]

harbor安装，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11326467.html][https_www.cnblogs.com_xiao987334176_p_11326467.html]

Jenkins安装，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11323795.html][https_www.cnblogs.com_xiao987334176_p_11323795.html]

**注意：Jenkins安装之后，还需要做以下配置。否则无法实现本文章的功能！！！**

Jenkins基于https的k8s配置，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11338827.html][https_www.cnblogs.com_xiao987334176_p_11338827.html]

GitLab+Jenkins持续集成，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11425560.html][https_www.cnblogs.com_xiao987334176_p_11425560.html]

Jenkins配置maven，请参考链接：

[https://www.cnblogs.com/xiao987334176/p/11433636.html][https_www.cnblogs.com_xiao987334176_p_11433636.html]

# 二、gitlab创建maven项目 #

首先从github上面，找一个基于maven的项目，地址如下：

[https://github.com/solochen84/SpringBootDemo][https_github.com_solochen84_SpringBootDemo]

使用git客户端，下载此项目代码

## 创建项目 ##

登录gitlab，创建一个项目springbootdemo

![1341090-20190830150610885-307913411.png][]

点击 设置--> 成员

![1341090-20190830150722172-689417445.png][]

添加2个成员，一个是jenkins用户，用来拉取代码的。一个是我，用来提交代码的。

**注意：我的账号，权限要高一些，可以直接提交到master分支。**

![1341090-20190830151101456-2050835692.png][]

将github下载的代码，提交到刚刚创建的项目中。效果如下：

![1341090-20190830151534477-154384474.png][]

# 三、harbor 创建用户和项目 #

## 创建jenkins用户 ##

登录到harbor后台，点击创建用户

![1341090-20190830151842364-829260934.png][]

注意，每一项都填写一项。尤其是密码，必须符合密码复杂性要求。

![1341090-20190830151950418-386778551.png][]

## 创建项目 ##

点击新建项目

![1341090-20190830152112666-265413376.png][]

输入名字，java。这就是私有项目。

![1341090-20190830152146386-1047807459.png][]

点击java

![1341090-20190830152321789-304974899.png][]

点击成员-->用户

![1341090-20190830152353667-1507706210.png][]

输入jenkins，会有提示的。角色选择 开发人员。

![1341090-20190830152432727-1932372690.png][]

# 四、jenkins修改默认用户 #

使用jenkins执行shell命令时，可能会出现权限不足的情况。

## 查看jenkins默认用户 ##

cat /etc/default/jenkins

内容如下：

# pulled in from the init script; makes things easier.
    NAME=jenkins
    
    # arguments to pass to java
    
    # Allow graphs etc. to work even when an X server is present
    JAVA_ARGS="-Djava.awt.headless=true"
    
    #JAVA_ARGS="-Xmx256m"
    
    # make jenkins listen on IPv4 address
    #JAVA_ARGS="-Djava.net.preferIPv4Stack=true"
    
    PIDFILE=/var/run/$NAME/$NAME.pid
    
    # user and group to be invoked as (default to jenkins)
    JENKINS_USER=$NAME
    JENKINS_GROUP=$NAME
    ...

可以发现JENKINS\_USER和JENKINS\_GROUP变量的值是jenkins

## 修改默认用户为root ##

将下面2个变量，修改为root

JENKINS_USER=root
    JENKINS_GROUP=root

## 重启jenkins服务 ##

service jenkins restart

# 五、jenkins和k8s master做ssh免密 #

为什么要和k8s master做ssh免密呢？因为jenkins需要登录到k8s master，做一些pod操作。

## 生成秘钥 ##

登录到jenkins服务器，生成秘钥

ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa

## copy秘钥 ##

ssh-copy-id 192.168.10.130

测试root免密

ssh 192.168.10.130

如果没有提示输入密码，说明成功了！

# 六、jenkins登录harbor #

修改docker配置文件，添加harbor地址

vim /etc/docker/daemon.json

内容如下：

{
        "insecure-registries": ["192.168.10.122"]}

重新加载docker配置

/etc/init.d/docker reload

测试登录

root@ubuntu:~/docker_dir# docker login 192.168.10.122 -u jenkins -p Jenkins@1234
    WARNING! Using --password via the CLI is insecure. Use --password-stdin.
    WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
    Configure a credential helper to remove this warning. See
    https://docs.docker.com/engine/reference/commandline/login/#credentials-store
    
    Login Succeeded

如果出现

Error response from daemon: Get https://192.168.10.122/v2/: dial tcp 192.168.10.122:443: connect: connection refused

表示 /etc/docker/daemon.json 文件修改不正确

# 七、jenkins创建maven项目 #

## 新建任务 ##

登录到jenkins后台，点击 新建Item

输入名称，选择构建一个maven项目

![1341090-20190830152824173-1543808612.png][]

## 设置参数化构建 ##

丢弃旧的构建，保留2天

![1341090-20190830160714676-903962613.png][]

## 设置源代码 ##

输入仓库下载地址，选择用户root

![1341090-20190830160846246-1529003885.png][]

## 设置maven构建命令 ##

输入命令：clean package

![1341090-20190830161436026-1661645941.png][]

mvn clean package依次执行了clean、resources、compile、testResources、testCompile、test、jar(打包)等７个阶段。

package命令完成了项目编译、单元测试、打包功能，但没有把打好的可执行jar包（war包或其它形式的包）布署到本地maven仓库和远程maven私服仓库

## jar包打包成docker镜像并推送 ##

### 执行shell命令 ###

build完成之后，选择执行shell命令

![1341090-20190830161827563-1877510157.png][]

将下面的内容，贴过去

#!/bin/bash
    
    jarName=spring-boot-demo-0.0.1-SNAPSHOT.jar
    jarFolder=ph
    harborPro=java
    projectName=ph
    
    docker_path=${WORKSPACE}
    
    cp ${WORKSPACE}/target/${jarName} ${docker_path}
    bash -x /root/docker_dir/deploy_docker.sh ${harborPro} ${projectName} ${docker_path} ${jarName}

内容解释：

jarName jar包的文件名，名字随便

jarFolder jar包的文件夹

harborPro harbor里面的项目，在上面步骤中，创建java私有项目。

projectName 项目名，名字随便

docker\_path docker命令的工作目录

最后一步，会执行一个shell脚本。下面说到！

效果如下：

![1341090-20190830162247220-1433750981.png][]

### 定义shell脚本 ###

登录到jenkins服务器，创建目录

mkdir /root/docker_dir

编辑deploy\_docker.sh脚本

cd /root/docker_dir/
    vim deploy_docker.sh

内容如下：

#!/bin/bash
    # maven $workspace $jarname
    # ${harborPro} ${projectName} ${docker_path} ${jarName}
    
    set -e
    harbor_project=$1
    projectName=$2
    docker_path=$3
    appName=$4
    
    # harbor认证用户
    user_name=jenkins
    password=Jenkins@1234
    
    # harbor地址以及tag
    tag=$(date +%s)
    harbor_server=192.168.10.122
    server_path=${harbor_server}
    taget_image=${projectName}:${tag}
    #${BUILD_NUMBER}
    echo ${taget_image}
    
    # 登录docker
    cd ${docker_path}
    sudo docker login ${harbor_server} -u ${user_name} -p ${password}
    
    # 生成镜像并推送到harbor,最后删除本地镜像
    sudo docker build --build-arg app=${appName} -t ${taget_image} .
    sudo docker tag ${taget_image} ${server_path}/${harbor_project}/${projectName}
    echo "The name of image is ${server_path}/${harbor_project}/${projectName}"
    sudo docker push ${server_path}/${harbor_project}/${projectName}:latest
    sudo docker rmi -f $(docker images|grep ${projectName}|grep ${tag}|awk '{print $3}'|head -n 1)

添加执行权限

chmod 755 /root/docker_dir/deploy_docker.sh

## yaml文件拷贝到k8s master上并运用应用 ##

## k8s master 操作 ##

登录到k8s master，创建目录

mkdir kube-conf

修改gitlab springbootdemo项目中的kube.yaml，将其重命名为springbootdemo.yaml

修改内容，添加秘钥

apiVersion: v1
    kind: Service
    metadata:
      name: maven-service
    spec:
      type: NodePort
      ports:
      - name: maven
        port: 8080
        nodePort: 31002
        targetPort: 8080
        protocol: TCP
      selector:
        app: maven
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: maven-deployment
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: maven
        spec:
          imagePullSecrets:
            - name: harborsecret
          containers:
            - name: maven
              image: 192.168.10.122/java/ph:latest
              imagePullPolicy: Always
              ports:
              - containerPort: 8080
              env:
                - name: key
                  value: "value"

将springbootdemo.yaml 提交到gitlab

查看内容是否更改

![1341090-20190830175711396-609261942.png][]

k8s 生成imagePullSecrets，请参考以下链接

[https://www.cnblogs.com/xiao987334176/p/11434326.html][https_www.cnblogs.com_xiao987334176_p_11434326.html]

注意：使用docker登录用户时，指定jenkins

docker login 192.168.10.122 -u jenkins -p Jenkins@1234

## jenkins操作 ##

添加执行shell命令

![1341090-20190830164037444-1906051491.png][]

将下面的内容，贴过去

#!/bin/bash
    
    set -e
    echo ok
    echo ${WORKSPACE}
    k8sMaster=192.168.10.130
    scp ${WORKSPACE}/*.yaml ${k8sMaster}:/root/kube-conf
    ssh ${k8sMaster} 'kubectl apply -f /root/kube-conf/springbootdemo.yaml'
    ssh ${k8sMaster} 'cd /root/kube-conf && kubectl delete -f springbootdemo.yaml && kubectl apply -f springbootdemo.yaml'
    ssh ${k8sMaster} 'kubectl get svc|grep maven'

命令解释：

这里主要就是将项目下的yaml文件复制到k8s master /root/kube-conf 目录

并应用yaml文件，最后查看svc信息。

效果如下：

![1341090-20190830183401539-771186968.png][]

## 手动构建应用 ##

点击Build Now

![1341090-20190830170223299-1877562678.png][]

点击\#1

![1341090-20190830170259234-2143843476.png][]

点击控制台输出

![1341090-20190830170317766-1867994989.png][]

它会自己下载一些组件，需要等待一段时间。

![1341090-20190830170400658-1481506468.png][]

出现spring图标，表示应用开始启动构建了。

![1341090-20190830171817637-312729402.png][]

最后出现错误

![1341090-20190830172059103-471160791.png][]

为啥呢？ 查看springbootdemo 的Dockerfile文件，查看第一行

FROM registry-scu.cloudtogo.cn/ubuntu:jdk

需要把 registry-scu.cloudtogo.cn 添加到 /etc/docker/daemon.json中

登录到jenkins服务器，修改 /etc/docker/daemon.json

vim /etc/docker/daemon.json

内容如下：

{
        "insecure-registries": ["192.168.10.122","registry-scu.cloudtogo.cn"]}

重新加载docker

/etc/init.d/docker reload

再次手动构建一次，点击Build Now

查看构建过程

出现以下提示，说明正在下载镜像 registry-scu.cloudtogo.cn/ubuntu:jdk

![1341090-20190830172646755-1334482697.png][]

最后提示执行完成

![1341090-20190830173351651-236877884.png][]

## 查看harbor仓库 ##

会发现多了一个镜像，这个镜像，就是刚刚构建时，提交的。

![1341090-20190830173634356-2095239764.png][]

# 八、访问k8s 应用 #

## 查看Pod状态 ##

root@k8s-master:~# kubectl get pods -o wide
    NAME                                READY   STATUS             RESTARTS   AGE     IP              NODE        NOMINATED NODE   READINESS GATES
    maven-deployment-7bd87867f8-6tlgp   0/1     ImagePullBackOff   0          5m19s   192.168.36.68   k8s-node1   <none>           <none>

发现pod运行在k8s-node1 这台服务器，状态是ImagePullBackOff

查看pod详细信息

kubectl describe po maven-deployment-7bd87867f8-6tlgp

输出：

Events:
      Type     Reason     Age                     From                Message
      ----     ------     ----                    ----                -------
      Normal   Scheduled  7m15s                   default-scheduler   Successfully assigned default/maven-deployment-7bd87867f8-6tlgp to k8s-node1
      Normal   Pulling    5m49s (x4 over 7m14s)   kubelet, k8s-node1  Pulling image "192.168.10.122/maven:latest"
      Warning  Failed     5m49s (x4 over 7m14s)   kubelet, k8s-node1  Failed to pull image "192.168.10.122/maven:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://192.168.10.122/v2/: dial tcp 192.168.10.122:443: connect: connection refused
      Warning  Failed     5m49s (x4 over 7m14s)   kubelet, k8s-node1  Error: ErrImagePull
      Normal   BackOff    5m36s (x6 over 7m13s)   kubelet, k8s-node1  Back-off pulling image "192.168.10.122/maven:latest"
      Warning  Failed     2m11s (x20 over 7m13s)  kubelet, k8s-node1  Error: ImagePullBackOff

出现

Get https://192.168.10.122/v2/: dial tcp 192.168.10.122:443: connect: connection refused

说明，k8s-node1 这台服务器的 /etc/docker/daemon.json 没有更改

登录到k8s-node1 这台服务器，修改文件

vim /etc/docker/daemon.json

内容如下：

{
        "insecure-registries": ["192.168.10.122"]}

重新加载docker

/etc/init.d/docker reload

## 重新加载pod ##

登录到k8s master服务器，删除pod，重新应用

root@k8s-master:~# cd /root/kube-conf/
    root@k8s-master:~/kube-conf# kubectl delete -f springbootdemo.yaml
    service "maven-service" deleted
    deployment.extensions "maven-deployment" deleted
    root@k8s-master:~/kube-conf# kubectl apply -f springbootdemo.yaml
    service/maven-service created
    deployment.extensions/maven-deployment created

再次查看状态

root@k8s-master:~/kube-conf# kubectl get pods -o wide
    NAME                                READY   STATUS    RESTARTS   AGE   IP              NODE        NOMINATED NODE   READINESS GATES
    maven-deployment-7bd7475544-q9b6j   1/1     Running   0          27s   192.168.36.77   k8s-node1   <none>           <none>

发现处于Running状态了

## 访问应用页面 ##

查看svc暴露的端口

root@k8s-master:~/kube-conf# kubectl get svc|grep maven
    maven-service   NodePort    10.98.152.31   <none>        8080:31002/TCP   4m

可以发现，暴露的端口是31002

使用谷歌访问页面

http://192.168.10.130:31002/

效果如下：

![1341090-20190830180026770-969223909.png][]

使用k8s-node1 的ip

http://192.168.10.131:31002/

效果同上！

# 九、更新应用代码 #

打开本地的项目springbootdemo，修改文件IndexController.java

路径如下：

springbootdemo\src\main\java\com\example\demo\web\controller

修改下面2行内容，把数字加1

jsonObject.put("welcome2", "2");
    jsonObject.put("welcome7", "7");

重新提交到gitlab，然后重新构建一次，点击Build Now

等待构建成功后，刷新页面

![1341090-20190830183934374-1500952612.png][]

发现内容已经更新了！

文本参考链接：

[https://linux265.com/news/3465.html][https_linux265.com_news_3465.html]

[https://www.cnblogs.com/aguncn/p/9789320.html][https_www.cnblogs.com_aguncn_p_9789320.html]

[https://juejin.im/post/5c07b1126fb9a049e82b4cfe][https_juejin.im_post_5c07b1126fb9a049e82b4cfe]

[1341090-20190830144305422-1548071491.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/178bd472daca4e68b0ed50a1fa7a196f.png
[https_www.cnblogs.com_xiao987334176_p_11317844.html]: https://www.cnblogs.com/xiao987334176/p/11317844.html
[https_www.cnblogs.com_xiao987334176_p_11329018.html]: https://www.cnblogs.com/xiao987334176/p/11329018.html
[https_www.cnblogs.com_xiao987334176_p_11326467.html]: https://www.cnblogs.com/xiao987334176/p/11326467.html
[https_www.cnblogs.com_xiao987334176_p_11323795.html]: https://www.cnblogs.com/xiao987334176/p/11323795.html
[https_www.cnblogs.com_xiao987334176_p_11338827.html]: https://www.cnblogs.com/xiao987334176/p/11338827.html
[https_www.cnblogs.com_xiao987334176_p_11425560.html]: https://www.cnblogs.com/xiao987334176/p/11425560.html
[https_www.cnblogs.com_xiao987334176_p_11433636.html]: https://www.cnblogs.com/xiao987334176/p/11433636.html
[https_github.com_solochen84_SpringBootDemo]: https://github.com/solochen84/SpringBootDemo
[1341090-20190830150610885-307913411.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/2a0ccbbe049246c08116fb979c076b61.png
[1341090-20190830150722172-689417445.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/9f9115be34b44871bdd69f35fcbb2a01.png
[1341090-20190830151101456-2050835692.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/ada808a5e3b54462a19e3f614b350d9f.png
[1341090-20190830151534477-154384474.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/d777681892bd4e989843dbef8a0d4ba4.png
[1341090-20190830151842364-829260934.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/10d822a4d8974d46b16caeb2d323b5e3.png
[1341090-20190830151950418-386778551.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/fb376b01ba7c4e8d8a00e5fc1feb3346.png
[1341090-20190830152112666-265413376.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/67ae82cb03ba4443a462c90dc5606af4.png
[1341090-20190830152146386-1047807459.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/bcbd82c445424c0590ab74f18b31a700.png
[1341090-20190830152321789-304974899.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/2be6c06d9c984531a82a20e81b35515a.png
[1341090-20190830152353667-1507706210.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/898ed09832ae45db8d172dde31d0d68b.png
[1341090-20190830152432727-1932372690.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/ec122fae3fb542eea31500c990943195.png
[1341090-20190830152824173-1543808612.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/cb35ef614ca7447fabf0513e7bcab247.png
[1341090-20190830160714676-903962613.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/c75341237f1147b980c48b9d7d5c3104.png
[1341090-20190830160846246-1529003885.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/87dc08c43bcd4570bb295f3f604bb4b8.png
[1341090-20190830161436026-1661645941.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/88f9575b5b7c407d9f1e78caaac4a7c0.png
[1341090-20190830161827563-1877510157.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/bb3ec2a09c824d248f8a3ec0e45e9fa5.png
[1341090-20190830162247220-1433750981.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/ce1320bdee1145c4a8e1802d6179dc46.png
[1341090-20190830175711396-609261942.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/b1cd1af40b314cb19ceccab3b230d756.png
[https_www.cnblogs.com_xiao987334176_p_11434326.html]: https://www.cnblogs.com/xiao987334176/p/11434326.html
[1341090-20190830164037444-1906051491.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/772cbb184a6d4e848fde5326b960ad83.png
[1341090-20190830183401539-771186968.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/0f3cbb606a5040889fb531faa100e05d.png
[1341090-20190830170223299-1877562678.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/047de8a4bccf42a6bb92d7adf3d66f82.png
[1341090-20190830170259234-2143843476.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/0a3174abae3e4699a1ab0ef0ccdd5847.png
[1341090-20190830170317766-1867994989.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/9b785742fe5c4194a3b37392c12fec4b.png
[1341090-20190830170400658-1481506468.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/1af0ece407d14a02a812570de706cbd0.png
[1341090-20190830171817637-312729402.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/79e99581f78e4fb9bcdfb8c2fc2b7c91.png
[1341090-20190830172059103-471160791.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/aacb707a1fa5421bbb294f21c49fc8b2.png
[1341090-20190830172646755-1334482697.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/2cacfb0fbffd4d02a4a59b992fc1c3d5.png
[1341090-20190830173351651-236877884.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/7a622cbd31bd4799bb721b03fb0e24b3.png
[1341090-20190830173634356-2095239764.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/cc8b4c16415c45d3a66b8434b6b5d9bd.png
[1341090-20190830180026770-969223909.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/4a0b0023d030430eabeed1ca0f766be1.png
[1341090-20190830183934374-1500952612.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/65e3380532444909a4189a4d2e6a8f50.png
[https_linux265.com_news_3465.html]: https://linux265.com/news/3465.html
[https_www.cnblogs.com_aguncn_p_9789320.html]: https://www.cnblogs.com/aguncn/p/9789320.html
[https_juejin.im_post_5c07b1126fb9a049e82b4cfe]: https://juejin.im/post/5c07b1126fb9a049e82b4cfe