Doris集群环境搭建

川长思鸟来 2022-09-09 06:16 498阅读 0赞

本文记录按照官网文档进行操作过程

### 1、下载预编译包 ###

（1）百度Doris官方网址  
2021年，百度网址doris.baidu.com/doc已经调整为了  
[https://cloud.baidu.com/doc/PALO/s/Wksis5irl][https_cloud.baidu.com_doc_PALO_s_Wksis5irl]  
![在这里插入图片描述][watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA56iL6KOV5by6_size_20_color_FFFFFF_t_70_g_se_x_16]  
请注意官方提示：

*  适用于CentOS 7.3
 *  依赖Oracle JDK 1.8 版本

（2）确定CPU是否支持avx2指令  
![在这里插入图片描述][watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA56iL6KOV5by6_size_20_color_FFFFFF_t_70_g_se_x_16 1]  
检查如下，我使用的是虚拟机，暂不支持avx2指令，因此需要下载对应的no-avx2版本的预编译

[root@node3 ~]# cat /proc/cpuinfo | grep flags
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon pebs bts nopl tsc_reliable nonstop_tsc aperfmperf pni ssse3 cx16 hypervisor lahf_lm ida arat epb dtherm
    [root@node3 ~]# cat /proc/cpuinfo | grep avx2
    [root@node3 ~]#

（3）下载  
如上图，单击`0.14.13-no-avx2`后面的Link，进行下载  
[https://palo-cloud-repo-bd.bd.bcebos.com/baidu-doris-release/PALO-0.14.13-release-no-avx2-binary.tar.gz][https_palo-cloud-repo-bd.bd.bcebos.com_baidu-doris-release_PALO-0.14.13-release-no-avx2-binary.tar.gz]

### 2、FE基本配置 ###

（1）将软件包上传到服务器  
此处我上传到了自己创建的/app目录下  
![在这里插入图片描述][watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA56iL6KOV5by6_size_20_color_FFFFFF_t_70_g_se_x_16 2]  
（2）加压并重命名

[root@node3 app]# tar -zxvf PALO-0.14.13-release-no-avx2-binary.tar.gz
    [root@node3 app]# mv PALO-0.14.13-release-no-avx2-binary doris-0.14.13
    [root@node3 app]# cd doris-0.14.13/
    [root@node3 doris-0.14.13]# ls
    apache_hdfs_broker  auditloader.zip  be  doris-flink-1.0-SNAPSHOT.jar  doris-spark-1.0.0-SNAPSHOT.jar  fe  udf
    [root@node3 doris-0.14.13]#

（3）创建FE节点元数据目录

[root@node3 doris-0.14.13]# cd fe
    [root@node3 fe]# mkdir doris-meta
    [root@node3 fe]#

（4）配置参数

[root@node3 fe]# vi conf/fe.conf

修改JAVA\_OPTS参数，默认内存是4G，根据具体情况修改。生产环境建议8G以上，我这里是测试环境，修改为了1G（1024M）

JAVA_OPTS="-Xmx1024m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

配置priority\_networks

priority_networks = 10.17.12.160/24

### 3、BE基本配置 ###

（1）配置环境参数/etc/security/limits.conf  
如果这一步已经配置过了，可以跳过。

[root@node1 be]# vi /etc/security/limits.conf

在文件末尾添加如下内容

# End of file
    
    * hard nofile 65536
    * soft nofile 65536

（2）创建数据目录

[root@node3 fe]# cd ../be
    [root@node3 be]# mkdir storage
    [root@node3 be]#

（3）修改配置文件

[root@node3 be]# vi conf/be.conf

priority_networks = 10.17.12.160/24

### 4、配置集群节点 ###

（1）复制软件包到集群节点

[root@node3 app]# scp -r doris-0.14.13/ node1:/app
    [root@node3 app]# scp -r doris-0.14.13/ node2:/app

（2）修改FE配置  
先修改node1

[root@node1 ~]# cd /app/doris-0.14.13/
    [root@node1 doris-0.14.13]# vi fe/conf/fe.conf 
    [root@node1 doris-0.14.13]#

priority_networks = 10.17.12.158/24

在修改node2

[root@node2 ~]# cd /app/doris-0.14.13/
    [root@node2 doris-0.14.13]# vi fe/conf/fe.conf 
    [root@node2 doris-0.14.13]#

priority_networks = 10.17.12.159/24

（3）修改BE配置

[root@node1 doris-0.14.13]# vi be/conf/be.conf

[root@node2 doris-0.14.13]# vi be/conf/be.conf

修改内容：  
node1:

priority_networks = 10.17.12.158/24

node2

priority_networks = 10.17.12.159/24

**注意**：如果是第一次配置了/etc/security/limits.conf，请重启机器。

### 5、 启动FE 集群 ###

（1）启动第一个FE节点

[root@node1 app]# cd doris-0.14.13/fe
    [root@node1 fe]# sh bin/start_fe.sh --daemon
    [root@node1 fe]# jps
    10400 PaloFe
    12744 Worker
    12249 Master
    10475 Jps
    [root@node1 fe]#

（2）登录客户端  
参考[https://blog.csdn.net/chengyuqiang/article/details/119974926][https_blog.csdn.net_chengyuqiang_article_details_119974926]按照mysql-client客户端

[root@node1 fe]# mysql -h10.17.12.158 -P9030 -uroot
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 0
    Server version: 5.1.0
    
    Copyright (c) 2000, 2021, Oracle and/or its affiliates. All rights reserved.
    
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql>

（3）添加FE节点

mysql> ALTER SYSTEM ADD FOLLOWER "node2:9010";
    Query OK, 0 rows affected (0.04 sec)
    
    mysql> ALTER SYSTEM ADD FOLLOWER "node3:9010";
    Query OK, 0 rows affected (0.01 sec)
    
    mysql>

（4）查看FE集群状态

mysql> show frontends \G
    *************************** 1. row ***************************
                 Name: 10.17.12.158_9010_1630292462734
                   IP: 10.17.12.158
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: true
            ClusterId: 1583270145
                 Join: true
                Alive: true
    ReplayedJournalId: 219
        LastHeartbeat: 2021-08-30 11:13:04
             IsHelper: true
               ErrMsg: 
              Version: 0.14.13-release-Unknown
    *************************** 2. row ***************************
                 Name: 10.17.12.160_9010_1630293081245
                   IP: 10.17.12.160
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 0
              RpcPort: 0
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 1583270145
                 Join: false
                Alive: false
    ReplayedJournalId: 0
        LastHeartbeat: NULL
             IsHelper: true
               ErrMsg: got exception
              Version: NULL
    *************************** 3. row ***************************
                 Name: 10.17.12.159_9010_1630293077685
                   IP: 10.17.12.159
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 0
              RpcPort: 0
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 1583270145
                 Join: false
                Alive: false
    ReplayedJournalId: 0
        LastHeartbeat: NULL
             IsHelper: true
               ErrMsg: got exception
              Version: NULL
    3 rows in set (0.06 sec)
    
    mysql>

（5）启动其他两个FE节点

[root@node2 fe]# ./bin/start_fe.sh --helper node1:9010 --daemon
    [root@node2 fe]# jps
    6144 Worker
    18516 PaloFe
    18568 Jps
    [root@node2 fe]#

[root@node3 fe]# ./bin/start_fe.sh --helper node1:9010 --daemon
    [root@node3 fe]# jps
    5971 PaloFe
    1940 Worker
    6025 Jps
    [root@node3 fe]#

mysql> show frontends \G
    *************************** 1. row ***************************
                 Name: 10.17.12.158_9010_1630292462734
                   IP: 10.17.12.158
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: true
            ClusterId: 1583270145
                 Join: true
                Alive: true
    ReplayedJournalId: 286
        LastHeartbeat: 2021-08-30 11:16:50
             IsHelper: true
               ErrMsg: 
              Version: 0.14.13-release-Unknown
    *************************** 2. row ***************************
                 Name: 10.17.12.160_9010_1630293081245
                   IP: 10.17.12.160
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 1583270145
                 Join: true
                Alive: true
    ReplayedJournalId: 285
        LastHeartbeat: 2021-08-30 11:16:50
             IsHelper: true
               ErrMsg: 
              Version: 0.14.13-release-Unknown
    *************************** 3. row ***************************
                 Name: 10.17.12.159_9010_1630293077685
                   IP: 10.17.12.159
          EditLogPort: 9010
             HttpPort: 8030
            QueryPort: 9030
              RpcPort: 9020
                 Role: FOLLOWER
             IsMaster: false
            ClusterId: 1583270145
                 Join: true
                Alive: true
    ReplayedJournalId: 285
        LastHeartbeat: 2021-08-30 11:16:50
             IsHelper: true
               ErrMsg: 
              Version: 0.14.13-release-Unknown
    3 rows in set (0.03 sec)
    
    mysql>

### 6、启动BE集群 ###

（1）添加BE节点

mysql> ALTER SYSTEM ADD BACKEND "node1:9050";
    Query OK, 0 rows affected (0.04 sec)
    
    mysql> ALTER SYSTEM ADD BACKEND "node2:9050";
    Query OK, 0 rows affected (0.02 sec)
    
    mysql> ALTER SYSTEM ADD BACKEND "node3:9050";
    Query OK, 0 rows affected (0.01 sec)
    
    mysql>

（2）查看BE集群状态

mysql> show backends \G
    *************************** 1. row ***************************
                BackendId: 10002
                  Cluster: default_cluster
                       IP: 10.17.12.158
            HeartbeatPort: 9050
                   BePort: -1
                 HttpPort: -1
                 BrpcPort: -1
            LastStartTime: NULL
            LastHeartbeat: NULL
                    Alive: false
     SystemDecommissioned: false
    ClusterDecommissioned: false
                TabletNum: 0
         DataUsedCapacity: .000 
            AvailCapacity: 1.000 B
            TotalCapacity: .000 
                  UsedPct: 0.00 %
           MaxDiskUsedPct: 0.00 %
                   ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)
                  Version: 
                   Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
    *************************** 2. row ***************************
                BackendId: 10003
                  Cluster: default_cluster
                       IP: 10.17.12.159
            HeartbeatPort: 9050
                   BePort: -1
                 HttpPort: -1
                 BrpcPort: -1
            LastStartTime: NULL
            LastHeartbeat: NULL
                    Alive: false
     SystemDecommissioned: false
    ClusterDecommissioned: false
                TabletNum: 0
         DataUsedCapacity: .000 
            AvailCapacity: 1.000 B
            TotalCapacity: .000 
                  UsedPct: 0.00 %
           MaxDiskUsedPct: 0.00 %
                   ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)
                  Version: 
                   Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
    *************************** 3. row ***************************
                BackendId: 10004
                  Cluster: default_cluster
                       IP: 10.17.12.160
            HeartbeatPort: 9050
                   BePort: -1
                 HttpPort: -1
                 BrpcPort: -1
            LastStartTime: NULL
            LastHeartbeat: NULL
                    Alive: false
     SystemDecommissioned: false
    ClusterDecommissioned: false
                TabletNum: 0
         DataUsedCapacity: .000 
            AvailCapacity: 1.000 B
            TotalCapacity: .000 
                  UsedPct: 0.00 %
           MaxDiskUsedPct: 0.00 %
                   ErrMsg: java.net.ConnectException: 拒绝连接 (Connection refused)
                  Version: 
                   Status: {"lastSuccessReportTabletsTime":"N/A","lastStreamLoadTime":-1}
    3 rows in set (0.01 sec)
    
    mysql>

这是因为当前3个BE节点都没有启动

（3）启动BE节点

[root@node1 ~]# cd /app/doris-0.14.13/be
    [root@node1 be]# sh bin/start_be.sh --daemon
    [root@node1 be]#  ps -aux |grep _be
    root     11402  2.3  1.3 2875316 111728 pts/2  Sl   11:23   0:00 /app/doris-0.14.13/be/lib/palo_be
    root     12344  0.0  0.0 112664   956 pts/2    S+   11:23   0:00 grep --color=auto _be
    [root@node1 be]#

[root@node2 be]# pwd
    /app/doris-0.14.13/be
    [root@node2 be]# sh bin/start_be.sh --daemon
    [root@node2 be]# ps -aux |grep _be
    root     27454  0.6  1.4 2876336 114208 pts/0  Sl   11:21   0:00 /app/doris-0.14.13/be/lib/palo_be
    root     29140  0.0  0.0 112664   960 pts/0    S+   11:22   0:00 grep --color=auto _be
    [root@node2 be]#

[root@node3 be]# pwd
    /app/doris-0.14.13/be
    [root@node3 be]# sh bin/start_be.sh --daemon
    [root@node3 be]# ps -aux |grep _be
    root     15805  1.6  1.1 2876336 95692 pts/0   Sl   11:22   0:00 /app/doris-0.14.13/be/lib/palo_be
    root     16473  0.0  0.0 112668   960 pts/0    S+   11:22   0:00 grep --color=auto _be
    [root@node3 be]#

（4）**稍等片刻**，再次查看BE集群状态

mysql> show backends \G
    *************************** 1. row ***************************
                BackendId: 10002
                  Cluster: default_cluster
                       IP: 10.17.12.158
            HeartbeatPort: 9050
                   BePort: 9060
                 HttpPort: 8040
                 BrpcPort: 8060
            LastStartTime: 2021-08-30 11:23:50
            LastHeartbeat: 2021-08-30 11:25:56
                    Alive: true
     SystemDecommissioned: false
    ClusterDecommissioned: false
                TabletNum: 0
         DataUsedCapacity: .000 
            AvailCapacity: 6.456 GB
            TotalCapacity: 49.090 GB
                  UsedPct: 86.85 %
           MaxDiskUsedPct: 86.85 %
                   ErrMsg: 
                  Version: 0.14.13-release-Unknown
                   Status: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:52","lastStreamLoadTime":-1}
    *************************** 2. row ***************************
                BackendId: 10003
                  Cluster: default_cluster
                       IP: 10.17.12.159
            HeartbeatPort: 9050
                   BePort: 9060
                 HttpPort: 8040
                 BrpcPort: 8060
            LastStartTime: 2021-08-30 11:21:53
            LastHeartbeat: 2021-08-30 11:25:56
                    Alive: true
     SystemDecommissioned: false
    ClusterDecommissioned: false
                TabletNum: 0
         DataUsedCapacity: .000 
            AvailCapacity: 9.921 GB
            TotalCapacity: 49.090 GB
                  UsedPct: 79.79 %
           MaxDiskUsedPct: 79.79 %
                   ErrMsg: 
                  Version: 0.14.13-release-Unknown
                   Status: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:56","lastStreamLoadTime":-1}
    *************************** 3. row ***************************
                BackendId: 10004
                  Cluster: default_cluster
                       IP: 10.17.12.160
            HeartbeatPort: 9050
                   BePort: 9060
                 HttpPort: 8040
                 BrpcPort: 8060
            LastStartTime: 2021-08-30 11:22:04
            LastHeartbeat: 2021-08-30 11:25:56
                    Alive: true
     SystemDecommissioned: false
    ClusterDecommissioned: false
                TabletNum: 0
         DataUsedCapacity: .000 
            AvailCapacity: 7.602 GB
            TotalCapacity: 49.090 GB
                  UsedPct: 84.51 %
           MaxDiskUsedPct: 84.51 %
                   ErrMsg: 
                  Version: 0.14.13-release-Unknown
                   Status: {"lastSuccessReportTabletsTime":"2021-08-30 11:25:07","lastStreamLoadTime":-1}
    3 rows in set (0.01 sec)
    
    mysql>

[https_cloud.baidu.com_doc_PALO_s_Wksis5irl]: https://cloud.baidu.com/doc/PALO/s/Wksis5irl
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA56iL6KOV5by6_size_20_color_FFFFFF_t_70_g_se_x_16]: /images/20220829/d287b22c09054b84b3078c1ba5221c46.png
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA56iL6KOV5by6_size_20_color_FFFFFF_t_70_g_se_x_16 1]: /images/20220829/b79f799774ab4051bc2452a98e487cf1.png
[https_palo-cloud-repo-bd.bd.bcebos.com_baidu-doris-release_PALO-0.14.13-release-no-avx2-binary.tar.gz]: https://palo-cloud-repo-bd.bd.bcebos.com/baidu-doris-release/PALO-0.14.13-release-no-avx2-binary.tar.gz
[watermark_type_ZHJvaWRzYW5zZmFsbGJhY2s_shadow_50_text_Q1NETiBA56iL6KOV5by6_size_20_color_FFFFFF_t_70_g_se_x_16 2]: /images/20220829/82414a2baa764458a2d52ec5edc1b791.png
[https_blog.csdn.net_chengyuqiang_article_details_119974926]: https://blog.csdn.net/chengyuqiang/article/details/119974926