最新版海豚调度dolphinscheduler-3.1.3安装部署详细教程

绝地灬酷狼 2024-03-30 17:41 91阅读 0赞

## 0 背景 ##

本文基于Ambari集群搭建最新版本的海豚调度dolphinscheduler-3.1.3版本，后续会尝试整合到Ambari中。

## 1 安装准备 ##

安装dolphinscheduler需要在环境中安装如下依赖  
① JDK8  
下载JDK (1.8+)，安装并配置 JAVA\_HOME 环境变量，并将其下的 bin 目录追加到 PATH 环境变量中  
② 数据库：[PostgreSQL (8.2.15+)][PostgreSQL _8.2.15] 或者 [MySQL (5.7+)][MySQL _5.7]，两者任选其一即可，如 MySQL 则需要 [JDBC Driver 8.0.16][]  
③ 注册中心：ZooKeeper (3.4.6+)

## 2 安装集群 ##

<table> 
 <thead> 
  <tr> 
   <th>节点</th> 
   <th>ip</th> 
   <th>服务角色</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Ambari-1</td> 
   <td>192.168.x.31</td> 
   <td>ds-master ds-api ds-alert</td> 
  </tr> 
  <tr> 
   <td>Ambari-2</td> 
   <td>192.168.x.32</td> 
   <td>ds-master</td> 
  </tr> 
  <tr> 
   <td>Ambari-3</td> 
   <td>192.168.x.33</td> 
   <td>ds-worker</td> 
  </tr> 
  <tr> 
   <td>Ambari-4</td> 
   <td>192.168.x.34</td> 
   <td>ds-worker</td> 
  </tr> 
  <tr> 
   <td>Ambari-5</td> 
   <td>192.168.x.35</td> 
   <td>ds-worker</td> 
  </tr> 
 </tbody> 
</table>

### 2.1 配置dolphinscheduler用户 ###

新建dolphinscheduler用户，并且一定要配置 sudo 免密.

# 创建用户需使用 root 登录
    useradd dolphinscheduler
    
    # 添加密码
    echo "dolphinscheduler" | passwd --stdin dolphinscheduler
    
    # 配置 sudo 免密
    sed -i '$adolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
    sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
    
    # 修改目录权限，使得部署用户对二进制包解压后的 apache-dolphinscheduler-*-bin 目录有操作权限
    chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin

配置免密

su dolphinscheduler
    ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    chmod 600 ~/.ssh/authorized_keys

集群之间完成免密

ssh-copy-id -i ~/.ssh/id_rsa.pub dolphinscheduler@主机

### 2.2 配置mysql ###

DolphinScheduler 元数据存储在关系型数据库中，故需创建相应的数据库和用户。注：这里根据Mysql安装版本的不同选择不同的执行方式。

mysql5.6/5.7

mysql -uroot -p
    mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
    # 修改 {user} 和 {password} 为你希望的用户名和密码
    mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
    mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}';
    mysql> flush privileges;

mysql8:  
创建数据库并指定编码格式

mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

开启权限

# 修改 {user} 和 {password} 为你希望的用户名和密码
    mysql> CREATE USER '{user}'@'%' IDENTIFIED BY '{password}';
    mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%';
    mysql> CREATE USER '{user}'@'localhost' IDENTIFIED BY '{password}';
    mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost';
    mysql> FLUSH PRIVILEGES;

执行过程中可能出现如下报错：

> ERROR 1819 (HY000): Your password does not satisfy the current policy requirements  
> ![在这里插入图片描述][19f3572402394a91b202b90639d41205.png]  
> 执行如下命令查看mysql密码要求规则

show variables like 'validate_password%';

![在这里插入图片描述][7417de1487d544abab84d25788b89767.png]  
修改校验登记为LOW,重新设置即可

set global validate_password.policy=LOW;

![在这里插入图片描述][55cb365597c740a988570f197bafbc61.png]

### 2.3 启动zk ###

### 2.4 修改ds配置文件 ###

#### 2.4.1 mysql的JDBC驱动放入ds的lib目录下 ####

包括 api-server/libs 和 alert-server/libs 和 master-server/libs 和 worker-server/libs

cp ../mysql-connector-java-8.0.16.jar ./lib
    cp ../mysql-connector-java-8.0.16.jar ./api-server/libs/
    cp ../mysql-connector-java-8.0.16.jar ./alert-server/libs/
    cp ../mysql-connector-java-8.0.16.jar ./master-server/libs/
    cp ../mysql-connector-java-8.0.16.jar ./worker-server/libs/

#### 2.4. 2 修改bin/env目录下的install\_env.sh配置文件 ####

我们仅需修改INSTALL MACHINE，DolphinScheduler ENV、Database、Registry Server  
![在这里插入图片描述][98b6f440d3034212a1b6cbd298ce854f.png]

#### 2.4.3 修改dolphinscheduler\_env.sh 文件 ####

bin/env/dolphinscheduler\_env.sh 中配置JAVA\_HOME、数据库、注册中心等相关配置  
![在这里插入图片描述][0ff37e4978494637b8caa4f650312c9f.png]

### 2.6 初始化数据库 ###

bash tools/bin/upgrade-schema.sh

![在这里插入图片描述][e550e8f4e18e4bc5aa380ad843fe4c8a.png]

### 2.7 启动ds ###

sh install.sh

报错如下：

> 9:32:47.555 \[main\] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation  
> 09:32:47.561 \[main\] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes  
> 09:32:47.578 \[main\] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false  
> ZooKeeper -server host:port -client-configuration properties-file cmd args  
> addWatch \[-m mode\] path \# optional mode is one of \[PERSISTENT, PERSISTENT\_RECURSIVE\] - default is PERSISTENT\_RECURSIVE  
> addauth scheme auth  
> close  
> config \[-c\] \[-w\] \[-s\]  
> connect host:port  
> create \[-s\] \[-e\] \[-c\] \[-t ttl\] path \[data\] \[acl\]  
> delete \[-v version\] path  
> deleteall path \[-b batch size\]  
> delquota \[-n|-b|-N|-B\] path  
> get \[-s\] \[-w\] path  
> getAcl \[-s\] path  
> getAllChildrenNumber path  
> getEphemerals path  
> history  
> listquota path  
> ls \[-s\] \[-w\] \[-R\] path  
> printwatches on|off  
> quit  
> reconfig \[-s\] \[-v version\] \[\[-file path\] | \[-members serverID=host:port1:port2;port3\[,…\]*\]\] | \[-add serverId=host:port1:port2;port3\[,…\]\]* \[-remove serverId\[,…\]\*\]  
> redo cmdno  
> removewatches path \[-c|-d|-a\] \[-l\]  
> set \[-s\] \[-v version\] path data  
> setAcl \[-s\] \[-v version\] \[-R\] path acl  
> setquota -n|-b|-N|-B val path  
> stat \[-w\] path  
> sync path  
> version  
> whoami  
> Command not found: Command not found rmr  
> 09:32:47.584 \[main\] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 127

![在这里插入图片描述][3c9826ab4b8345eea1631248cbea17d2.png]  
查看具体生成日志发现如下报错：

> Caused by: org.apache.dolphinscheduler.registry.api.RegistryException: Failed to put registry key: /nodes/master  
> 239 at org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperRegistry.put(ZookeeperRegistry.java:186)  
> 240 at org.apache.dolphinscheduler.service.registry.RegistryClient.initNodes(RegistryClient.java:247)  
> 241 at org.apache.dolphinscheduler.service.registry.RegistryClient.afterConstruct(RegistryClient.java:77)  
> 242 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
> 243 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
> 244 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
> 245 at java.lang.reflect.Method.invoke(Method.java:498)  
> 246 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor L i f e c y c l e E l e m e n t . i n v o k e ( I n i t D e s t r o y A n n o t a t i o n B e a n P o s t P r o c e s s o r . j a v a : 389 ) 247 a t o r g . s p r i n g f r a m e w o r k . b e a n s . f a c t o r y . a n n o t a t i o n . I n i t D e s t r o y A n n o t a t i o n B e a n P o s t P r o c e s s o r LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) 247 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389)247atorg.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessorLifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333)  
> 248 at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157)  
> 249 … 46 common frames omitted  
> 250 Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /dolphinscheduler/nodes/master  
> 251 at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)  
> 252 at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)  
> 253 at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1450)  
> 254 at org.apache.curator.framework.imps.CreateBuilderImpl$18.call(CreateBuilderImpl.java:1216)  
> 255 at org.apache.curator.framework.imps.CreateBuilderImpl$18.call(CreateBuilderImpl.java:1193)  
> 256 at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)  
> 257 at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1190)  
> 258 at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:605)  
> 259 at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:595)  
> 260 at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:48)  
> 261 at org.apache.dolphinscheduler.plugin.registry.zookeeper.ZookeeperRegistry.put(ZookeeperRegistry.java:184)

![在这里插入图片描述][48c403d3606b403ebed81e96b7495213.png]  
这里最新版本的ds与zk版本不兼容，最新版本的ds需要最新版本的zk，即3.8版本，当前环境安装的是3.4.6。通过查看ds依赖的zk的jarb包也能验证这一点  
![在这里插入图片描述][e3231a27199d4cc0b20d41ec18583058.png]

解决思路：下载3.8版本的zk并安装  
下载地址:  
https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz  
安装方法可以看这篇文章[《zookeeper快速入门》][zookeeper]  
安装完成并启动zk，然后再重新启动ds成功。  
![在这里插入图片描述][e0e9c61fd5034defbae808db42a096ec.png]  
访问master节点url连接：

http://<your_ip>:12345/dolphinscheduler/ui/login

默认账户密码：admin/dolphinscheduler123

![在这里插入图片描述][5cb1031eb2914c0cbd51bb04c4babb72.png]

## 3 \[附加\]ds源码修改与打包 ##

当前ds-3.1.3版本数据源还不支持trino，可以修改源码，增加对trino数据源的支持，修改完成后使用如下命令重新编译并完成打包。

./mvnw clean install -Prelease -DskipTests=true

这里的-DskipTests=true指跳过测试，如果不加该参数，整个耗时会特别长。编译完成后可看到如下成功页面。  
![在这里插入图片描述][edbff972a4e142c1b95c68e011aed7fb.png]  
然后进入你的maven仓库即可发现已经打包的ds-3.1.3安装包

cd /root/.m2/repository/org/apache/dolphinscheduler/dolphinscheduler-dist/3.1.3
    ll -h

![在这里插入图片描述][41b68e0561714e3e97aafb6dd5d16256.png]

[PostgreSQL _8.2.15]: https://www.postgresql.org/download/
[MySQL _5.7]: https://dev.mysql.com/downloads/mysql/
[JDBC Driver 8.0.16]: https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.16.zip
[19f3572402394a91b202b90639d41205.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/f45699ae87804c438c29ef4d4794f87f.png
[7417de1487d544abab84d25788b89767.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/60cca328730742ea9affe0f416c68c9e.png
[55cb365597c740a988570f197bafbc61.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/ac2893723b704d12937b69d48dc44d3f.png
[98b6f440d3034212a1b6cbd298ce854f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/b66a7ecffab34de58ba152f1d95ccd8a.png
[0ff37e4978494637b8caa4f650312c9f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/7b4f240ec95247768c958e5db52e34c0.png
[e550e8f4e18e4bc5aa380ad843fe4c8a.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/a6350e71c3e24e93bfbfe625ad22f127.png
[3c9826ab4b8345eea1631248cbea17d2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/e5c3ee8170964e3e883ec398d21122a5.png
[48c403d3606b403ebed81e96b7495213.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/3516e1850be14422ace0009c5ab21235.png
[e3231a27199d4cc0b20d41ec18583058.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/4281c721f4f442ab91c80bf2ba33b154.png
[zookeeper]: https://blog.csdn.net/Keyuchen_01/article/details/108372008
[e0e9c61fd5034defbae808db42a096ec.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/a7536cec6cae4c6bb17ea7681e7dcaa6.png
[5cb1031eb2914c0cbd51bb04c4babb72.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/7378e7f5b7534ab2ab45091d5bd52a8f.png
[edbff972a4e142c1b95c68e011aed7fb.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/881f6d69727e4c43bc473769470dc813.png
[41b68e0561714e3e97aafb6dd5d16256.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/30/99021bc2d2a543bcb3b5d4803205bc17.png