azkaban3.7编译安装

青旅半醒 2023-10-10 08:54 11阅读 0赞

# azkaban3.7编译安装 #

## 1 编译安装 ##

**环境**

mysql 8.0
    azkaban 3.75
    jdk 1.8
    git 2.3

**编译**

进入解压后的目录，编译
     
     ./gradelw distTar
    
    大概30分钟左右，挂上代理会快点

**安装**

（1）创建azkaban目录
    （2）解压azkaban-db的压缩包
     
     tar -zxvf ./azkaban-db/build/distributions/azkaban-db-0.1.0-SNAPSHOT.tar.gz -C ../azkaban
     
    （3）解压azkaban-exec-server的压缩包
     
     tar -zxvf ./azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz -C ../azkaban
     
    （4）解压azkaban-web-server的压缩包
     
     tar -zxvf ./azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz -C ../azkaban
     
    （5）解压azkaban-solo-server的压缩包
     
     tar -zxvf ./azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz -C ../azkaban
    （6）重命名
     mv azkaban-db-0.1.0-SNAPSHOT azkaban-db
     mv azkaban-exec-server-0.1.0-SNAPSHOT azkaban-exec-server
     mv azkaban-web-server-0.1.0-SNAPSHOT azkaban-web-server
     mv azkaban-solo-server-0.1.0-SNAPSHOT azkaban-solo-server

## 创建数据库 ##

（1）创建数据库
     create database azkaban;
    （2）创建azkaban用户，并分配权限（此处我掠过直接用root）
     CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban';
     
     GRANT ALL ON azkaban.* to 'azkaban'@'localhost' IDENTIFIED BY 'azkaban';
     
     flush privileges;
    （3）导入数据
     use azkaban;
     source /usr/local/azkaban/azkaban-db/create-all-sql-0.1.0-SNAPSHOT.sql;

## 创建密钥 ##

keytool -keystore keystore -alias jetty -genkey -keyalg RSA

把生成的keystore 放到azkaban-web-server/bin下

## 修改azkaban-exec-server下配置文件，引入mysql的依赖包 ##

（1）将azkaban-solo-server下conf、plugins、sql目录复制到azkaban-exec-server目录下
     cd /usr/local/azkaban/azkaban-solo-server
     cp -a ./conf ./plugins ./sql /usr/local/azkaban-exec-server
    （2）修改azkaban-exec-server的azkaban.properties文件
    
    
    
    # Azkaban Personalization Settings
    azkaban.name=GPY
    azkaban.label=My Local Azkaban
    azkaban.color=#FF3601
    azkaban.default.servlet.path=/index
    web.resource.dir=/usr/azkaban/azkaban-web-server/web/
    default.timezone.id=Asia/Shanghai
    # Azkaban UserManager class
    user.manager.class=azkaban.user.XmlUserManager
    user.manager.xml.file=/usr/azkaban/azkaban-web-server/conf/azkaban-users.xml
    # Loader for projects
    executor.global.properties=/usr/azkaban/azkaban-exec-server/conf/global.properties
    azkaban.project.dir=projects
    database.type=h2
    h2.path=./h2
    h2.create.tables=true
    # Velocity dev mode
    velocity.dev.mode=false
    # Azkaban Jetty server properties.
    jetty.use.ssl=false
    jetty.maxThreads=25
    jetty.port=8081
    jetty.ssl.port=8443
    # Azkaban Executor settings
    executor.port=12321
    # mail settings
    mail.sender=
    mail.host=
    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    # when this parameters set then these parameters are used to generate email links. 
    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    # azkaban.webserver.external_hostname=myazkabanhost.com
    # azkaban.webserver.external_ssl_port=443
    # azkaban.webserver.external_port=8081
    job.failure.email=
    job.success.email=
    lockdown.create.projects=false
    cache.directory=cache
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban plugin settings
    azkaban.jobtype.plugin.dir=/usr/azkaban/azkaban-web-server/plugins/jobtypes
    # Number of executions to be displayed
    azkaban.display.execution_page_size=16
    azkaban.use.multiple.executors=true
    
    #添加mysql配置信息
     database.type=mysql
     mysql.port=3306
     mysql.host=hadoop
     mysql.database=azkaban
     mysql.user=root
     mysql.password=123456
     mysql.numconnections=100
    
    
    
    （4）在azkaban-exec-server目录下创建extlib目录，将mysql的驱动拷贝到该目录和lib目录下
     mkdir ./extlib
     cp ~/softdirs/mysql-connector-java-8.0.17.jar ./extlib
      cp ~/softdirs/mysql-connector-java-8.0.17.jar ./lib
     
    （5）将azkaban-exec-server目录下conf、extlib、lib目录复制到azkaban-web-server目录下
     cp ./conf ./extlib ./lib  ../azkaban-web-server

登录mysql，将azkaban数据库下的executors表的active值改为1；否者启动程序会报azkaban.executor.ExecutorManagerException: No active executors found的异常。

use azkaban;
     
    update executors set active=1 where id=1

启动程序

cd azkaban-exec-server/bin
    ./start-exec.sh
     
    cd azkaban-web-server/bin
    ./start-web.sh
     
    jps 查看启动进程

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NTI1OTA2_size_16_color_FFFFFF_t_70]  
如果查看日志后出现，端口未激活,手动激活

curl http://hadoop:12321/executor?action=activate

错误：

29-05-2018 15:30:13 CST test ERROR - Job run failed!
    azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib'
        at azkaban.utils.Props.getString(Props.java:421)
        at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:234)
        at azkaban.execapp.JobRunner.runJob(JobRunner.java:752)
        at azkaban.execapp.JobRunner.doRun(JobRunner.java:591)
        at azkaban.execapp.JobRunner.run(JobRunner.java:552)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
    29-05-2018 15:30:13 CST test ERROR - Missing required property 'azkaban.native.lib' cause: null
    29-05-2018 15:30:13 CST test INFO - Finishing job test at 1527579013756 with status FAILED

这个问题很奇怪，第一次部署的时候没有出现，换了个环境部署就出现了，如果是老版本的话呢

`exec-server/plugins/jobtypes/commonprivate.properties`下把 azkaban.native.lib=false就行了

commonprivate.properties 中有两条记录：

execute.as.user=false
    azkaban.native.lib=false

但是我再3.7的那个配置文件下并没有找到这个可以改的地方

最后的解决方法是：

exec-server/plugins/jobtypes/common/java/azkaban/jobExeccutor/utils/ProcessJob.java里  
    
    
    boolean isExecuteAsUser = sysProps.getBoolean(EXECUTE_AS_USER, true);
    改成
    
    boolean isExecuteAsUser = sysProps.getBoolean(EXECUTE_AS_USER, false);

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NTI1OTA2_size_16_color_FFFFFF_t_70 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NTI1OTA2_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/2019092509074830.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NTI1OTA2,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NTI1OTA2_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20190925090917278.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM2NTI1OTA2,size_16,color_FFFFFF,t_70