工作流调度器azkaban 我会带着你远行 2022-12-18 02:56 118阅读 0赞 # 一、Azkaban概述 # `Azkaban`是由`Linkedin`公司推出的一个批量工作流任务调度器,主要用于在一个工作流内以一个特定的顺序运行一组工作和流程,它的配置是通过简单的`key:value`对的方式,通过配置中的`dependencies`来设置依赖关系。`Azkaban`使用`job`配置文件建立任务之间的依赖关系,并提供一个易于使用的`web`用户界面维护和跟踪你的工作流。 ## 1.1 为什么需要工作流调度系统 ## * 一个完整的数据分析系统通常都是由大量任务单元组成:`shell`脚本程序,`java`程序,`MapReduce`程序、`hive`脚本等; * 各任务单元之间存在时间先后及前后依赖关系; * 为了很好地组织起这样的复杂执行计划,需要一个工作流调度系统来调度执行。 ## 1.2 Azkaban特点 ## 1. 兼容任何版本的`Hadoop` 2. 易于使用的`Web`用户界面 3. 简单的工作流的上传 4. 方便设置任务之间的关系 5. 调度工作流 6. 模块化和可插拔的插件机制 7. 认证/授权(权限的工作) 8. 能够杀死并重新启动工作流 9. 有关失败和成功的电子邮件提醒 ## 1.3 常见工作流调度系统 ## * 简单的任务调度:直接使用`crontab`实现; * 复杂的任务调度:开发调度平台或使用现成的开源调度系统,比如`ooize`、`azkaban`等。 ## 1.4 Azkaban的架构 ## ![在这里插入图片描述][20200703173002896.png] `Azkaban`由三个关键组件构成: * `AzkabanWebServer`:`AzkabanWebServer`是整个`Azkaban`工作流系统的主要管理者,它用户登录认证、负责`project`管理、定时执行工作流、跟踪工作流执行进度等一系列任务。 * `AzkabanExecutorServer`:负责具体的工作流的提交、执行,它们通过mysql数据库来协调任务的执行。 * 关系型数据库(`MySQL`):存储大部分执行流状态,`AzkabanWebServer`和`AzkabanExecutorServer`都需要访问数据库。 ## 1.5 在3.0后,Azkaban提供了两种部署模式: ## ### 1.5.1、solo-server模式 ### DB使用的是一个内嵌的H2,Web Server和Executor Server运行在同一个进程里。这种模式包含Azkaban的所有特性,但一般用来学习和测试,也可以用于小的应用。 ### 1.5.2、distributed multiple-executor模式 ### DB使用的是MySQL,MySQL最好使用master-slave架构,Web Server和Executor Server运行在不同机器上,且有多个Executor Server;这为Azkaban提供了很强的扩展性。 ## 1.6 启动Azkaban ## ## 编译安装 ## ### 1、下载源码 ### 从github上下载azkaban官方源码,目前最新release为3.90.0 下载链接:[https://github.com/azkaban/azkaban/releases][https_github.com_azkaban_azkaban_releases] ### 2、编译 ### 进入到下载的azkaban目录中进行编译:(提前安装好git,gradle,jdk8) ./gradlew build -x test BUILD SUCCESSFUL in 32m 7s 73 actionable tasks: 73 executed ### 3、解压 ### ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70][] /mine/azkaban-3.90.0/azkaban-web-server/build/distributions /mine/azkaban-3.90.0/azkaban-exec-server/build/distributions /mine/azkaban-3.90.0/azkaban-solo-server/build/distributions /mine/azkaban-3.90.0/azkaban-db/build/distributions azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz azkaban-web-server-0.1.0-SNAPSHOT.tar.gz azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz azkaban-db-0.1.0-SNAPSHOT.tar.gz ### 4、**azkaban web服务器安装** ### 解压azkaban-web-server-0.1.0-SNAPSHOT.tar.gz、azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz、azkaban-db-0.1.0-SNAPSHOT.tar.gz tar –zxvf azkaban-web-server-0.1.0-SNAPSHOT.tar.gz tar -zxvf azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz tar -zxvf azkaban-db-0.1.0-SNAPSHOT.tar.gz tar -zxvf azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz mv ../azkaban-db-0.1.0-SNAPSHOT sql ### 5、solo-server模式 ### 编译完成后, cd /mine/azkaban-3.90.0/azkaban-solo-server/build/distributions **5.1、修改时区** conf/azkaban.properties: #default.timezone.id=America/Los_Angeles default.timezone.id=Asia/Shanghai .default.servlet.path=/index web.resource.dir=/mine/azkaban/solo/web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/mine/azkaban/solo/conf/azkaban-users.xml # Loader for projects executor.global.properties=/mine/azkaban/solo/conf/global.properties azkaban.project.dir=projects database.type=h2 h2.path=./h2 h2.create.tables=true # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Azkaban Executor settings executor.port=12321 # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=plugins/jobtypes # Number of executions to be displayed azkaban.display.execution_page_size=16 azkaban.use.multiple.executors=true # Azkaban Ramp Feature Configuration #Ramp Feature Related azkaban.ramp.enabled=true azkaban.ramp.status.polling.enabled=true azkaban.ramp.status.polling.interval.min=30 azkaban.ramp.status.push.interval.threshold=15 azkaban.ramp.status.pull.interval.threshold=100 vim conf/azkaban-users.xml <azkaban-users> <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/> <user password="metrics" roles="metrics" username="metrics"/> <user username="admin" password="admin" roles="admin,metrics" /> <role name="admin" permissions="ADMIN"/> <role name="metrics" permissions="METRICS"/> </azkaban-users> ### 6、distributed multiple-executor模式 ### 假设Mysql已经安装完成,这里只考虑Web Server和Executor Server;源代码的编译在**192.168.60.101**上,目录为/mine/azkaban-3.90.0。 <table> <tbody> <tr> <td>192.168.60.101</td> <td>Web Server</td> </tr> <tr> <td>192.168.60.102</td> <td>Executor Server</td> </tr> <tr> <td>192.168.60.103</td> <td>Executor Server</td> </tr> </tbody> </table> **6.1、创建数据库** # ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 1][] # # ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 2][] # ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 3][] **6.2 部署Executor Server** 从192.168.60.101上拷贝部署包到192.168.60.102上 ![20201029200607111.png][] scp azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz hadoop02:/mine **192.168.60.102上部署Executor Server:** 修改conf/azkaban.properties文件: # Azkaban Personalization Settings azkaban.name=Test azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=conf/azkaban-users.xml # Loader for projects executor.global.properties=conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Where the Azkaban web server is located azkaban.webserver.url=http://hadoop01:8081 # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=plugins/jobtypes # Azkaban mysql settings by default. Users should configure their own username and password. database.type=mysql mysql.port=3306 mysql.host=ljxwtl.cn mysql.database=azkaban mysql.user=root mysql.password=wtl199201180271 mysql.numconnections=100 # Azkaban Executor settings executor.maxThreads=50 executor.flow.threads=30 executor.port=6666 启动: bin/start-exec.sh ** 问题:** **如果出现连接MySQL 有问题,而且报错里有caused by:java.lang.NullPointerException** 是因为我用的mysql是8.\*版本的,然而azkaban自带的mysql的jdbc包是mysql-connector-java-5\*.jar,不支持,需要删掉原来的jdbc的驱动包,用mysql-connector-java-8\*.jar。 ![20201029202617363.png][] rm -rf mysql-connector-java-5.1.28.jar ![20201029203406753.png][] 启动完之后,还需要激活: curl -G "localhost:12321/executor?action=activate" && echo ![20201029211730240.png][] **192.168.60.103上部署Executor Server(同192.168.60.102(hadoop02))的操作一致!!!** **6.3、部署Web Server** 可以直接使用192.168.600.101(hadoop01)上编译好的部署目录/mine/azkaban-web-server-0.1.0-SNAPSHOT ![20201029204348432.png][] ** 问题:** **如果出现连接MySQL 有问题,而且报错里有caused by:java.lang.NullPointerException** 是因为我用的mysql是8.\*版本的,然而azkaban自带的mysql的jdbc包是mysql-connector-java-5\*.jar,不支持,需要删掉原来的jdbc的驱动包,用mysql-connector-java-8\*.jar。 ![20201029202617363.png][] rm -rf mysql-connector-java-5.1.28.jar ![20201029203406753.png][] 修改conf/azkaban.properties文件: # Azkaban Personalization Settings azkaban.name=Test azkaban.label=My Local Azkaban azkaban.color=#FF3601 azkaban.default.servlet.path=/index web.resource.dir=/mine/azkaban-web-server-0.1.0-SNAPSHOT/web/ default.timezone.id=Asia/Shanghai # Azkaban UserManager class user.manager.class=azkaban.user.XmlUserManager user.manager.xml.file=/mine/azkaban-web-server-0.1.0-SNAPSHOT/conf/azkaban-users.xml # Loader for projects executor.global.properties=/mine/azkaban-web-server-0.1.0-SNAPSHOT/conf/global.properties azkaban.project.dir=projects # Velocity dev mode velocity.dev.mode=false # Azkaban Jetty server properties. jetty.use.ssl=false jetty.maxThreads=25 jetty.port=8081 # Azkaban Executor settings # mail settings mail.sender= mail.host= # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 # when this parameters set then these parameters are used to generate email links. # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. # azkaban.webserver.external_hostname=myazkabanhost.com # azkaban.webserver.external_ssl_port=443 # azkaban.webserver.external_port=8081 job.failure.email= job.success.email= lockdown.create.projects=false cache.directory=cache # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban mysql settings by default. Users should configure their own username and password. database.type=mysql mysql.port=3306 mysql.host=ljxwtl.cn mysql.database=azkaban mysql.user=root mysql.password=wtl199201180271 mysql.numconnections=100 #Multiple Executor azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1 vim azkaban-users.xml <azkaban-users> <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/> <user password="metrics" roles="metrics" username="metrics"/> <user username="admin" password="admin" roles="admin,metrics" /> <role name="admin" permissions="ADMIN"/> <role name="metrics" permissions="METRICS"/> </azkaban-users> 问题: 2020/10/29 09:27:49.902 -0400 INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Using dispatcher for execution id :20 2020/10/29 09:27:49.907 -0400 INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Reached handleNoExecutorSelectedCase stage for exec 20 with error count 0 2020/10/29 09:27:49.907 -0400 INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Using dispatcher for execution id :20 2020/10/29 09:27:49.908 -0400 INFO [ExecutorManager] [AzkabanWebServer-QueueProcessor-Thread] [Azkaban] Reached handleNoExecutorSelectedCase stage for exec 20 with error count 0 ![20201029212834356.png][] 解决方案: 可能原因: MinimumFreeMemory过滤器会检查executor主机空余内存是否会大于6G,如果不足6G,则web-server不会将任务交由该主机执行。 处理方法: 修改conf/azkaban.properties文件,去除MinimumFreeMemory过滤器 #azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus ![20201029213119846.png][] 问题: ![2020103107243283.png][] 原因: azkaban要求执行主机可用内存必须大于3G才能满足执行任务的条件。 处理方法: Executor Server部署目录下plugins/jobtypes/commonprivate.properties文件增加: memCheck.enabled=false # 二、Azkaban实战 # **Azkaba内置的任务类型支持command、java** ## 1、Command类型单一job示例 ## ### (1)创建job描述文件 ### vim command.job ### (2) 将job资源文件打包成zip文件 ### #command.job type=command command=echo 'hello world' zip command.job.zip command.job ### (3)通过azkaban的web管理平台创建project并上传job压缩包 ### 首先创建project ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 4][] 上传zip包 ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 5][] **问题:** ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 6][] **原因:** azkaban要求执行主机可用内存必须大于3G才能满足执行任务的条件。 plugins/jobtypes/commonprivate.properties文件增加: memCheck.enabled=false ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 7][] ## 2、**Command类型多job工作流flow** ## ### (1)创建有依赖关系的多个job描述 ### **第一个job:foo.job** # foo.job type=command command=echo foo ** 第二个job:bar.job依赖foo.job** # bar.job type=command dependencies=foo command=echo bar ** 将所有job资源文件打到一个zip包中** ![20201026212116675.png][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 8][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 9][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 10][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 11][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 12][] ## 3、**HDFS操作任务** ## **(1)创建job描述文件** # fs.job type=command command=hadoop fs -mkdir /azkaban **(2)将job资源文件打包成zip文件 ** ![20201031073423764.png][] **(3)通过azkaban的web管理平台创建project并上传job压缩包 (4)启动执行该job** ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 13][] ## 4、**MAPREDUCE任务** ## **Mr任务依然可以使用command的job类型来执行** **(1)创建job描述文件,及mr程序jar包(示例中直接使用hadoop自带的example jar)** # mrwc.job type=command command=hadoop jar /mine/hadoop-3.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /wordcount/input /wordcount/azout **(2)将所有job资源文件打到一个zip包中** ![20201031074542332.png][] **(3)在azkaban的web管理界面创建工程并上传zip包** ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 14][] **(4)启动job** ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 15][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 16][] ## 4、**HIVE脚本任务** ## **(1)创建job描述文件和hive脚本** Hive脚本: test.sql use default; drop table aztest; create table azkaban_student(id string,name string,sex string,age int,address string) row format delimited fields terminated by ',' stored as textfile; load data local inpath '/mine/students.txt' into table azkaban_student; create table azres as select * from azkaban_student; insert overwrite directory '/aztest/hiveoutput' select count(1) from azkaban_student; Job描述文件:hivef.job # hivef.job type=command command=hive -f 'test.sql' **(2) 将所有job资源文件打到一个zip包中** ![20201031105057657.png][] **(3) 在azkaban的web管理界面创建工程并上传zip包** **(4) 启动job** ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 17][] **报错原因:** **在hadoop02里没有hive的程序,所以报错!!!** **解决方案:** **在每台executor机器里都安装hive。** ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 18][] ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 19][] [20200703173002896.png]: /images/20221122/95468d6471694d7681664a9d3e553130.png [https_github.com_azkaban_azkaban_releases]: https://github.com/azkaban/azkaban/releases [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70]: /images/20221122/841b41cefe084b958f0410d4bd88416b.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 1]: /images/20221122/90b2debb1386418b992d460495b5b184.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 2]: /images/20221122/c870cdb39bc84ae3b1381cc086e2e708.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 3]: /images/20221122/2a359ebbe5164c65b8e896ee282c95c4.png [20201029200607111.png]: /images/20221122/57455fe8328040f4b647188e965e3453.png [20201029202617363.png]: /images/20221122/802eda139cc147959392ad6cd806b588.png [20201029203406753.png]: /images/20221122/69c1399cde4344cdb0d44f60bde2684e.png [20201029211730240.png]: /images/20221122/c893b7b8bf9b44ffae67844f878cc6a9.png [20201029204348432.png]: /images/20221122/568bacaa0e204be394564751cb4ec519.png [20201029212834356.png]: /images/20221122/7c0ea5a2799d495987c4f0e0bc6bfb68.png [20201029213119846.png]: /images/20221122/21ecfaf94bbe42a7b262d2a2cbacd968.png [2020103107243283.png]: /images/20221122/14e7a83e91c64840970f753a65b73303.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 4]: /images/20221122/923f733baa24454b87a17ce26ca2f045.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 5]: /images/20221122/a62a6651f52440cc8eee636be840f714.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 6]: /images/20221122/07415e7d015b4f61bcb4b71ce22489c1.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 7]: /images/20221122/50ae7afaced0437c875a96b1015bff31.png [20201026212116675.png]: /images/20221122/d364514616104fedae6d652fcef93b67.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 8]: /images/20221122/5d5f90d04d7641f28945517b427dbf8c.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 9]: /images/20221122/a180daca26264a35a299115c5b31b7b6.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 10]: /images/20221122/111667bcd3c1480c94f4aa01327b6d77.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 11]: /images/20221122/4dfd8bcbd3c04561a517a402445d7153.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 12]: /images/20221122/e19e7caf9718471f8e3a4fb96a5a949e.png [20201031073423764.png]: /images/20221122/3880f2abf67b45e1ad6d7d6ab873de6f.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 13]: /images/20221122/32761abc33b6460395b30d1ee921d2db.png [20201031074542332.png]: /images/20221122/8cd31a47f4b042a2976d4be48510f013.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 14]: /images/20221122/b665b3b49618464c862a861ebfe84e18.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 15]: /images/20221122/c7cfe72909e845698655f820673e44cb.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 16]: /images/20221122/1907f5819a384901a0c8c17c065e63ec.png [20201031105057657.png]: /images/20221122/7e14e8f4a3364caf85e7c5d69c7b8de9.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 17]: /images/20221122/1b2a9564be01440d91d3828c52eea85d.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 18]: /images/20221122/df6c22945d114e118df4945d50177fbb.png [watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 19]: /images/20221122/0f947e6ddfbe4cd9a3f6ab644c1afaac.png
还没有评论,来说两句吧...