Apache Doris集群搭建（FE扩容）

清疚 2022-09-03 10:28 313阅读 0赞

参考官方文档[http://doris.apache.org/master/zh-CN/installing/install-deploy.html\#%E6%89%A9%E5%AE%B9%E7%BC%A9%E5%AE%B9][http_doris.apache.org_master_zh-CN_installing_install-deploy.html_E6_89_A9_E5_AE_B9_E7_BC_A9_E5_AE_B9]

### 1、现有FE节点 ###

（1）[前面博文][Link 1]已经介绍搭建了一个FE节点

[root@node3 fe]# sh bin/start_fe.sh --daemon
    [root@node3 fe]# jps
    7234 Jps
    7160 PaloFe
    [root@node3 fe]#

（2）FE节点配置如下

JAVA_OPTS="-Xmx512m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

priority_networks = 192.168.9.101/24

（3）登录客户端

[root@node3 fe]# mysql -h10.17.12.160 -P9030 -uroot -p123456
    Welcome to the MariaDB monitor.  Commands end with ; or \g.
    Your MySQL connection id is 3
    Server version: 5.1.0 Doris version 0.12.0-rc03
    
    Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    MySQL [(none)]> SHOW PROC '/frontends'\G
    *************************** 1. row ***************************
                 Name: 192.168.122.1_9010_1628234030903
                   IP: 192.168.122.1
             HostName: 192.168.122.1
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: true
            ClusterId: 2007251588
                 Join: true
                Alive: true
    ReplayedJournalId: 8078
        LastHeartbeat: 2021-08-06 23:11:09
             IsHelper: true
               ErrMsg: 
    1 row in set (20.05 sec)
    
    MySQL [(none)]>

### 2、FE集群介绍 ###

FE 分为 Leader，Follower 和 Observer 三种角色。 默认一个集群，只能有一个 Leader，可以有多个 Follower 和 Observer。其中 Leader 和 Follower 组成一个 Paxos 选择组，如果 Leader 宕机，则剩下的 Follower 会自动选出新的 Leader，保证写入高可用。Observer 同步 Leader 的数据，但是不参加选举。如果只部署一个 FE，则 FE 默认就是 Leader。

第一个启动的 FE 自动成为 Leader，上面输出可以看到`IsMaster: true`。在此基础上，可以添加若干 Follower 和 Observer。

### 3、FE集群扩容 ###

（1）使用 mysql-client 连接到已启动的 FE，这一步上面已经完成。  
（2）添加 Follower 或 Observer  
`ALTER SYSTEM ADD FOLLOWER "host:port";`  
`ALTER SYSTEM ADD OBSERVER "host:port";`

其中，其中 host 为 Follower 或 Observer 所在节点 ip，port 为其配置文件 fe.conf 中的 edit\_log\_port，默认值是**9010**。

操作如下：

MySQL [(none)]> ALTER SYSTEM ADD FOLLOWER "10.17.12.158:9010";
    Query OK, 0 rows affected (0.03 sec)
    
    MySQL [(none)]> ALTER SYSTEM ADD OBSERVER "10.17.12.159:9010";
    Query OK, 0 rows affected (0.01 sec)
    
    MySQL [(none)]>

（3）FOLLOWER节点部署

[root@node1 app]# tar -zxvf apache-doris_apache-doris-0.12.0-incubating-bin.tar.gz
    [root@node1 app]# cd apache-doris-0.12.0-incubating-bin/
    [root@node1 apache-doris-0.12.0-incubating-bin]# ls
    apache_hdfs_broker  be  fe  LICENSE.txt  NOTICE.txt  quickstart  README.md
    [root@node1 apache-doris-0.12.0-incubating-bin]# mv fe /app/fe
    [root@node1 apache-doris-0.12.0-incubating-bin]# mv be /app/be
    [root@node1 apache-doris-0.12.0-incubating-bin]# cd /app/fe
    [root@node1 fe]# mkdir doris-meta
    [root@node1 fe]# ./bin/start_fe.sh --helper 10.17.12.160:9010 --daemon
    ./bin/start_fe.sh:行111: /app/fe/log/fe.out: 没有那个文件或目录
    ./bin/start_fe.sh:行112: /app/fe/log/fe.out: 没有那个文件或目录
    [root@node1 fe]# jps
    6148 PaloFe
    6233 Jps
    [root@node1 fe]#

–helper 参数仅在 follower 和 observer 第一次启动时才需要。  
（4）OBSERVER节点部署

[root@node2 app]# tar -zxvf apache-doris_apache-doris-0.12.0-incubating-bin.tar.gz
    [root@node2 app]# cd apache-doris-0.12.0-incubating-bin/
    [root@node2 apache-doris-0.12.0-incubating-bin]# ls
    apache_hdfs_broker  be  fe  LICENSE.txt  NOTICE.txt  quickstart  README.md
    [root@node2 apache-doris-0.12.0-incubating-bin]# mv fe /app/fe
    [root@node2 apache-doris-0.12.0-incubating-bin]# mv be /app/be
    [root@node2 apache-doris-0.12.0-incubating-bin]# cd /app/fe
    [root@node2 fe]# mkdir doris-meta
    [root@node2 fe]# ./bin/start_fe.sh --helper 10.17.12.160:9010 --daemon
    ./bin/start_fe.sh:行111: /app/fe/log/fe.out: 没有那个文件或目录
    ./bin/start_fe.sh:行112: /app/fe/log/fe.out: 没有那个文件或目录
    [root@node2 fe]# jps
    6148 PaloFe
    6233 Jps
    [root@node1 fe]#

（5）查看FE集群状态

MySQL [(none)]> SHOW PROC '/frontends'\G
    *************************** 1. row ***************************
                 Name: 10.17.12.159_9010_1628262806951
                   IP: 10.17.12.159
             HostName: node2
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 0
              RpcPort: 0
                 Role: OBSERVER
             IsMaster: false
            ClusterId: 2007251588
                 Join: false
                Alive: false
    ReplayedJournalId: 0
        LastHeartbeat: N/A
             IsHelper: false
               ErrMsg: invalid cluster id: 1461718723
    *************************** 2. row ***************************
                 Name: 192.168.122.1_9010_1628234030903
                   IP: 192.168.122.1
             HostName: 192.168.122.1
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: true
            ClusterId: 2007251588
                 Join: true
                Alive: true
    ReplayedJournalId: 8179
        LastHeartbeat: 2021-08-06 23:16:39
             IsHelper: true
               ErrMsg: 
    *************************** 3. row ***************************
                 Name: 10.17.12.158_9010_1628262775800
                   IP: 10.17.12.158
             HostName: node1
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 0
              RpcPort: 0
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 2007251588
                 Join: false
                Alive: false
    ReplayedJournalId: 0
        LastHeartbeat: N/A
             IsHelper: true
               ErrMsg: got exception
    3 rows in set (20.05 sec)
    
    MySQL [(none)]>

### 4、存在的问题 ###

上面输出可以看到node1和node2的fe状态为`Alive: false`

问题原因是服务器具有多个IP地址，参考官方文档  
[http://doris.apache.org/master/zh-CN/installing/install-deploy.html\#%E8%BD%AF%E7%A1%AC%E4%BB%B6%E9%9C%80%E6%B1%82][http_doris.apache.org_master_zh-CN_installing_install-deploy.html_E8_BD_AF_E7_A1_AC_E4_BB_B6_E9_9C_80_E6_B1_82]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXVxaWFuZw_size_16_color_FFFFFF_t_70]  
解决方案：  
（1）停止服务

[root@node3 fe]# jps
    6576 Worker
    30467 Jps
    7160 PaloFe
    [root@node3 fe]# bin/stop_fe.sh 
    stop java, and remove pid file. 
    [root@node3 fe]# jps
    6576 Worker
    32086 Jps
    [root@node3 fe]# v

（2）查看IP地址

[root@node3 fe]# ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    
    2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
        link/ether 00:50:56:98:94:d9 brd ff:ff:ff:ff:ff:ff
        inet 10.17.12.160/24 brd 10.17.12.255 scope global ens192
           valid_lft forever preferred_lft forever
    
    3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
        link/ether 52:54:00:2e:e3:a4 brd ff:ff:ff:ff:ff:ff
        inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
           valid_lft forever preferred_lft forever
    
    [root@node3 fe]#

（3）配置priority\_networks

[root@node3 fe]# vi conf/fe.conf

配置如下

http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    mysql_service_nio_enabled = true
    
    # Choose one if there are more than one ip except loopback address.
    # Note that there should at most one ip match this list.
    # If no ip match this rule, will choose one randomly.
    # use CIDR format, e.g. 10.10.10.0/24
    # Default value is empty.
    priority_networks = 10.17.12.160/24

[root@node3 fe]# rm -rf doris-meta/*
    [root@node3 fe]# ls doris-meta/
    [root@node3 fe]# ls
    bin  conf  lib  webroot
    [root@node3 fe]#

（5）查看单节点FE状态

（6）同样操作处理node2和node3  
node2:priority\_networks = 10.17.12.159/24  
node3:priority\_networks = 10.17.12.160/24

MySQL [(none)]> SHOW PROC '/frontends'\G
    *************************** 1. row ***************************
                 Name: 10.17.12.160_9010_1628735497652
                   IP: 10.17.12.160
             HostName: node3
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: true
            ClusterId: 1912164111
                 Join: true
                Alive: true
    ReplayedJournalId: 579
        LastHeartbeat: 2021-08-12 11:03:08
             IsHelper: true
               ErrMsg: 
    *************************** 2. row ***************************
                 Name: 10.17.12.159_9010_1628736841249
                   IP: 10.17.12.159
             HostName: node2
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 0
              RpcPort: 0
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 1912164111
                 Join: true
                Alive: false
    ReplayedJournalId: 0
        LastHeartbeat: N/A
             IsHelper: true
               ErrMsg: got exception
    *************************** 3. row ***************************
                 Name: 10.17.12.158_9010_1628736836696
                   IP: 10.17.12.158
             HostName: node1
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 1912164111
                 Join: true
                Alive: true
    ReplayedJournalId: 579
        LastHeartbeat: 2021-08-12 11:03:08
             IsHelper: true
               ErrMsg: 
    3 rows in set (0.05 sec)
    
    MySQL [(none)]>

[http_doris.apache.org_master_zh-CN_installing_install-deploy.html_E6_89_A9_E5_AE_B9_E7_BC_A9_E5_AE_B9]: http://doris.apache.org/master/zh-CN/installing/install-deploy.html#%E6%89%A9%E5%AE%B9%E7%BC%A9%E5%AE%B9
[Link 1]: https://blog.csdn.net/chengyuqiang/article/details/119455109
[http_doris.apache.org_master_zh-CN_installing_install-deploy.html_E8_BD_AF_E7_A1_AC_E4_BB_B6_E9_9C_80_E6_B1_82]: http://doris.apache.org/master/zh-CN/installing/install-deploy.html#%E8%BD%AF%E7%A1%AC%E4%BB%B6%E9%9C%80%E6%B1%82
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NoZW5neXVxaWFuZw_size_16_color_FFFFFF_t_70]: /images/20220829/c02e5d8525344b569e4168e7dc3969e2.png