3.2.5 spark体系之分布式计算-spark-core之离线计算-HA-Spark集群环境搭建(Yarn模式)

╰+哭是因爲堅強的太久メ 2022-12-28 08:19 144阅读 0赞

**目录**

1.概述

1.1 Yarn的架构（spark放到container中）

1.2 Yarn运行模式介绍

1.3 Yarn部署spark流程图

2.安装并使用

一、集群规划

二、集群配置

2.1 spark-env.sh

2.2 spark-defaults.conf

2.3 slaves

2.4 yarn-site.xml

2.5 分发

三、启动集群

3.1 启动ha-hadoop集群

3.2 在hadoop01上启动spark集群

3.3 在hadoop02上启动备Master

3.4 在hadoop01上启动任务历史服务器

四、查看集群

4.1 jps进程查看

4.2 Web UI查看

五、计算 PI (测试高可用)

--------------------

# 1.概述 #

Spark客户端直接连接Yarn，不需要额外构建Spark集群。有yarn-client和yarn-cluster两种模式，主要区别在于：Driver程序的运行节点。

*  yarn-client：Driver程序运行在客户端，适用于交互、调试，希望立即看到app的输出
 *  yarn-cluster：Driver程序运行在由RM（ResourceManager）启动的AP（APPMaster）适用于生产环境。

## **1.1 Yarn的架构（spark放到container中）** ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70][]

## **1.2 Yarn运行模式介绍** ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70 1][]

## **1.3 Yarn部署spark流程图** ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70 2][]

# 2.安装并使用 #

前置准备

CentOS7、jdk1.8、scala-2.11.12、spark-2.4.5、hadoop-2.7.7、zookeeper-3.5.7

## 一、集群规划 ##

![图片][99297ec93b6d09aef9b6c9800b1bf98f.png]

## 二、集群配置 ##

### 2.1 spark-env.sh ###

[xiaokang@hadoop01 conf]$ cp spark-env.sh.template spark-env.sh
    
    export JAVA_HOME=/opt/moudle/jdk1.8.0_191
    export SCALA_HOME=/opt/moudle/scala-2.11.12
    YARN_CONF_DIR=/opt/software/hadoop-2.7.7/etc/hadoop
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop01:2181,hadoop02:2181,hadoop03:2181 -Dspark.deploy.zookeeper.dir=/ha-spark"
    export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=24 -Dspark.history.fs.logDirectory=hdfs://hadoop01:9000/spark-jobhistory"

### 2.2 spark-defaults.conf ###

[xiaokang@hadoop01 conf]$ cp spark-defaults.conf.template spark-defaults.conf
    
    #spark.master                     spark://hadoop01:7077
    spark.master                     spark://hadoop01:7077,hadoop02:7077
    spark.eventLog.enabled           true
    spark.eventLog.dir               hdfs://hadoop01:9000/spark-jobhistory
    spark.yarn.historyServer.address               hadoop01:18080

### 2.3 slaves ###

[xiaokang@hadoop01 conf]$ cp slaves.template slaves
    
    hadoop01
    hadoop02
    hadoop03

### 2.4 yarn-site.xml ###

<property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    
    <property>
        <name>yarn.log.server.url</name>
        <value>http://hadoop01:19888/jobhistory/logs</value>
    </property>

### 2.5 分发 ###

[xiaokang@hadoop01 ~]$ distribution.sh /opt/software/spark-2.4.5
    [xiaokang@hadoop01 ~]$ distribution.sh /opt/software/hadoop-2.7.7/etc/hadoop/yarn-site.xml

## 三、启动集群 ##

### 3.1 启动ha-hadoop集群 ###

[xiaokang@hadoop01 ~]$ ha-hadoop.sh start

### 3.2 在hadoop01上启动spark集群 ###

# 进入/opt/software/spark-2.4.5/sbin目录并启动集群
    [xiaokang@hadoop01 sbin]$ ./start-all.sh

### 3.3 在hadoop02上启动备Master ###

# 进入/opt/software/spark-2.4.5/sbin目录并启动备Master
    [xiaokang@hadoop02 sbin]$ ./start-master.sh

### 3.4 在hadoop01上启动任务历史服务器 ###

# 进入/opt/software/spark-2.4.5/sbin目录并启动任务历史服务器
    [xiaokang@hadoop01 sbin]$ ./start-history-server.sh

## 四、查看集群 ##

### 4.1 jps进程查看 ###

[xiaokang@hadoop01 ~]$ call-cluster.sh jps
    --------hadoop01--------
    10881 NodeManager
    11265 HistoryServer
    13058 Master
    10131 NameNode
    10470 JournalNode
    11176 Worker
    10250 DataNode
    10683 DFSZKFailoverController
    10796 JobHistoryServer
    12973 Jps
    9935 QuorumPeerMain
    --------hadoop02--------
    7232 QuorumPeerMain
    7648 DFSZKFailoverController
    7505 JournalNode
    7831 NodeManager
    8183 Worker
    7321 NameNode
    7402 DataNode
    9099 Jps
    7743 ResourceManager
    8255 Master
    --------hadoop03--------
    7313 DataNode
    8273 Jps
    7234 QuorumPeerMain
    7843 Worker
    7545 ResourceManager
    7419 JournalNode
    7663 NodeManager

### 4.2 Web UI查看 ###

[xiaokang@hadoop01 sbin]$ cat /opt/software/spark-2.4.5/logs/spark-xiaokang-org.apache.spark.deploy.master.Master-1-hadoop01.out | grep MasterWebUI
    
    20/05/31 08:32:08 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hadoop01:8081

通过启动日志可以看到hadoop01的`MasterWebUI`的端口号为8081

![图片][acb0dee03481dc624e88931b646bc82e.png]

[xiaokang@hadoop02 sbin]$ cat /opt/software/spark-2.4.5/logs/spark-xiaokang-org.apache.spark.deploy.master.Master-1-hadoop02.out | grep MasterWebUI
    
    20/05/31 08:32:43 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://hadoop02:8082

通过启动日志可以看到hadoop02的`MasterWebUI`的端口号为8082

![图片][99fafadbcf722ebfb9263aa24c8cb6d1.png]

hadoop01上的任务历史服务器端口号为18080

![图片][e47af5aa02b25e92e7c97413fac0d29e.png]

## 五、计算 PI (测试高可用) ##

[xiaokang@hadoop01 ~]$ spark-submit --master yarn --class org.apache.spark.examples.SparkPi /opt/software/spark-2.4.5/examples/jars/spark-examples_2.11-2.4.5.jar 1000

运行过程中将主Master给kill掉，测试是否高可用

[xiaokang@hadoop01 ~]$ kill -9 11091

杀掉主Master之后，程序还是同样在执行，可以看到hadoop02的`MasterWebUI`中状态显示为recovering

![图片][a212fbbc3c6ea17f1a3a516d4a0689e3.png]

最终计算结果如下：

![图片][c545e7c0cc4f69320b77ef6491e79fae.png]

通过hadoop03上YARN的`8088`可以查看历史任务

![图片][fadcbf2efc447bbeca75c93a3758ef55.png]

![图片][d4337c9c8f6e68f981271f95dc848f9d.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70]: /images/20221120/ede0d2d8019c42eeb3badec45f8e7527.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70 1]: /images/20221120/6a434406b8034983a36b72d0fb803930.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L1N1eWViaXViaXU_size_16_color_FFFFFF_t_70 2]: /images/20221120/0a130c0044644db2b1eae29130c46e66.png
[99297ec93b6d09aef9b6c9800b1bf98f.png]: /images/20221120/8b9ff57d979e436b9705495b9850076a.png
[acb0dee03481dc624e88931b646bc82e.png]: /images/20221120/c4a75f8c6c094aed8bb7411f0f8e52fb.png
[99fafadbcf722ebfb9263aa24c8cb6d1.png]: /images/20221120/15444c910bcc4bc4af0bbc843fc1a6ec.png
[e47af5aa02b25e92e7c97413fac0d29e.png]: /images/20221120/4c8528e06091496289578886f1926d04.png
[a212fbbc3c6ea17f1a3a516d4a0689e3.png]: /images/20221120/9f0762a5ad03497fb367515057e43298.png
[c545e7c0cc4f69320b77ef6491e79fae.png]: /images/20221120/1808149feb094a419744acc92121bd79.png
[fadcbf2efc447bbeca75c93a3758ef55.png]: /images/20221120/54669a565fe24857abfe43584e76c5e7.png
[d4337c9c8f6e68f981271f95dc848f9d.png]: /images/20221120/bc4c61ea897d4d95b2ecb6560fb6e9cb.png