azkanban简介与安装部署

Love The Way You Lie 2022-11-07 13:21 164阅读 0赞

## azkanban架构 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2MjgxNDY_size_16_color_FFFFFF_t_70]  
**WebServer**:整个调度集群的核心，负责所有作业的管理和调度。  
**ExecutorServer**：整个调度集群中实际运行作业的节点。  
**DB(MYSQL)**:是集群中所有节点运行公用的数据存储，包含作业信息、各种调度元数据等等。**默认不支持mysql8.x**

## 调度原理 ##

WebServer根据Executor server 的运行状态信息，选择一个合适的Executor来运行WorkFlow, 然后会将提交到队列中的WorkFlow调度到选定的Executor Server上运行。  
**通信**：WebServer与Executor之间的交互方式是通过REST API的方式进行的，WebServer根据调度需要，主动调用ExecutorServer暴露的REST API来获取相应的资源信息，比如Executor Server的状态信息、分配WorkFlow到指定ExectorServer上运行。

> WebServer上有一个核心线程、主要方法selectExecutorAndDispatchFlow()用来负责如何选择一个合适的Executor去处理作业。

## 部署 ##

WebServer:10.5.25.53  
ExecutorServer:10.3.17.174

安装包下载：https://github.com/azkaban/azkaban(需翻墙)

git clone https://github.com/azkaban/azkaban.git
    cd azkaban 
    ./gradlew build installDist

假设相关机器上已有安装包：  
executor拷入安装包：`scp -r root@10.5.25.53:/root/azkaban-exec-server-3.80.0-11-g82923e2/ .`  
webServer拷入安装包：`scp -r algo@10.3.17.173:/home/algo/azkaban_package/azkaban-web-server .`

**webServer部署**

scp -r algo@10.3.17.173:/home/algo/azkaban_package/azkaban-db-3.80.0-11-g82923e2 .

web端安装mysql：

docker search mysql
    docker pull mysql
    cd /opt/
    mkdir mysql_docker
    cd mysql_docker/
    echo $PWD
    docker run --name mysqlserver -v $PWD/conf:/etc/mysql/conf.d -v $PWD/logs:/logs -v $PWD/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=123456 -d -i -p 3306:3306 mysql:latest
    docker exec -it mysqlserver bash
    mysql -uroot -p
    use mysql;
    select host,user from user;
    ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
    flush privileges;
    mysql -h10.5.25.53 -P3306 -uroot -p"123456"

创建azkaban数据库相关

CREATE DATABASE azkaban;
    set global max_allowed_packet = 210241024；
    SET GLOBAL innodb_file_format = BARRACUDA;
    use azkaban；
    source /root/azkaban-db-3.80.0-11-g82923e2/create-all-sql-3.80.0-11-g82923e2.sql;
    SET @@global.innodb_large_prefix = 1;

配置webServer:/root/azkaban-web-server/conf

# Azkaban Personalization Settings
    azkaban.name=Test
    azkaban.label=My Local Azkaban
    azkaban.color=#FF3601
    azkaban.default.servlet.path=/index
    web.resource.dir=web/
    default.timezone.id=Asia/Shanghai
    # Azkaban UserManager class
    user.manager.class=azkaban.user.XmlUserManager
    user.manager.xml.file=conf/azkaban-users.xml<绝对路径>
    # Loader for projects
    executor.global.properties=conf/global.properties
    azkaban.project.dir=projects
    # Velocity dev mode
    velocity.dev.mode=false
    # Azkaban Jetty server properties.
    jetty.use.ssl=false
    jetty.maxThreads=25
    jetty.port=8081
    # Azkaban Executor settings
    # mail settings
    mail.host=192.168.80.26
    mail.port=25
    mail.sender=azkaban@dangdang.com
    mail.useAuth=false
    # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    # when this parameters set then these parameters are used to generate email links.
    # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    # azkaban.webserver.external_hostname=myazkabanhost.com
    # azkaban.webserver.external_ssl_port=443
    # azkaban.webserver.external_port=8081
    job.failure.email=haoshuwen@dangdang.com
    job.success.email=haoshuwen@dangdang.com
    lockdown.create.projects=false
    cache.directory=cache
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban mysql settings by default. Users should configure their own username and password.
    database.type=mysql
    mysql.port=3306
    mysql.host=10.5.31.68
    mysql.database=azkaban
    mysql.user=algo
    mysql.password=algo
    mysql.numconnections=100
    #Multiple Executor
    azkaban.use.multiple.executors=true
    azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
    azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
    azkaban.executorselector.comparator.Memory=1
    azkaban.executorselector.comparator.LastDispatched=1
    azkaban.executorselector.comparator.CpuUsage=1

**ExecutorServer部署**

cd /root/azkaban-exec-server-3.80.0-11-g82923e2/conf
    azkaban.properties 配置内容同webServer配置
    cd /root/azkaban-exec-server-3.80.0-11-g82923e2
    bash bin/start-exec.sh

executor激活(executor必须被激活才能被webServer感知)：`curl -G "localhost:$(<./executor.port)/executor?action=activate" && echo`

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2MjgxNDY_size_16_color_FFFFFF_t_70]: /images/20221023/5432d703cb344dbf884738fd96968bfb.png