Spark 安装 Standalone模式 (单机与集群)

浅浅的花香味﹌ 2022-02-26 11:26 586阅读 0赞

### 前言 ###

今天我们了解下`Spark`的安装. Spark的安装模式有很多模式, 主要包括  
[Spark Standalone][]、[Running Spark on YARN][]、[Running Spark on Kubernetes][]、[Running Spark on Mesos][]. 其中`YARN/Mesos/Kubernetes`都是资源的调度器. 本章我们将简单的介绍下`Spark Standalone`在`单机`与`集群`模式下的安装.

本文主要包括如下几个部分:

*  安装前置条件
 *  安装 - 单机
 *  安装 - 集群

--------------------

### 前置条件 ###

与`Hadoop`的安装相似, `Spark`的安装前置条件为:

*  JDK
 *  防火墙
 *  免密

--------------------

### 安装 - 单机 ###

Spark的安装较为简单. 我们先将下载的`Spark安装包`解压.(我的为`spark-2.2.1-bin-hadoop2.7.tgz`). 注意安装包版本的选择要根据集群内的其他组件相结合(主要是`scala/jdk/hadoop`的版本).

*  步骤1: 解压安装包

tar -zxvf spark-2.2.1-bin-hadoop2.7.tgz

*  步骤2: 进入`conf`配置文件目录. 修改`slaves`和`spark-env.sh`这两个配置文件.

# spark-env.sh
    export SCALA_HOME=/Users/Sean/Software/Scala/scala-2.11.8/bin/scala
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home
    # 本地安装绑定
    export SPARK_MASTER_IP=127.0.0.1
    export SPARK_LOCAL_IP=127.0.0.1
    export SPARK_MASTER_PORT=7077
    export SPARK_WORKER_MEMORY=1G
    export SPARK_EXECUTOR_CORES=2
    # export master=spark://192.168.100.80:7077

# slaves
    # A Spark Worker will be started on each of the machines listed below.
    localhost

*  步骤3: 使用启动`start-all.sh`脚本进行启动.

localhost:sbin Sean$ ./start-all.sh
    starting org.apache.spark.deploy.master.Master, logging to /Users/Sean/Software/Spark/spark-2.2.1-bin-hadoop2.7/logs/spark-Sean-org.apache.spark.deploy.master.Master-1-localhost.out
    localhost: starting org.apache.spark.deploy.worker.Worker, logging to /Users/Sean/Software/Spark/spark-2.2.1-bin-hadoop2.7/logs/spark-Sean-org.apache.spark.deploy.worker.Worker-1-localhost.out

从上方的日志可以看到, 我们在本地启动了一个`master节点`与一个`salve节点`.

*  步骤4: 验证

localhost:sbin Sean$ jps -lm
    40460 org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://localhost:7077
    40428 org.apache.spark.deploy.master.Master --host localhost --port 7077 --webui-port 8080

可以看到我们在本地启动了一个`master`与`worker`.

访问WebUI地址: [http://localhost:8080/][http_localhost_8080]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly95YW54bWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]

--------------------

### 安装 - 集群 ###

集群的安装与单机类似. 首先我们有三台虚拟机:

*  master(192.168.31.60 - sean60)
 *  worker1(192.168.31.61 - sean61)
 *  woeker2(192.168.31.62 - sean62)

分别在上方完成JDK安装、免密、防火墙这三个前置条件.

随后的安装步骤如下:

*  步骤1: 将压缩包, 通过集群脚本分别传输到三台机器上, 并且解压.

#!/bin/bash
    # 需要安装的机器(也可以从配置文件读取)
    cluster_nodes="192.168.31.60,192.168.31.61,192.168.31.62"
    cluster_nodes=${cluster_nodes//,/ }
    # 安装名字
    app_tar_name="spark-2.2.1-bin-hadoop2.7.tgz";
    # 安装目录
    install_folder="/opt/apps/spark"
    
    # 循环所有的结点
    for i in ${cluster_nodes}; do 
    # make dir
    	ssh root@${i} "mkdir -p ${install_folder};" 
    # tar -zxvf spark-2.2.1-bin-hadoop2.7.tgz
    # 分发
    	scp ${app_tar_name} root@${i}:${install_folder}/
    # 解压
    	ssh root@${i} "cd ${install_folder};tar -zxvf spark-2.2.1-bin-hadoop2.7.tgz; mv spark-2.2.1-bin-hadoop2.7 spark-2.2.1;"
    done

*  步骤2: 更改`spark-env.sh`与`slaves`文件并分发到三台机器上.

# spark-env.sh
    export SCALA_HOME=/Users/Sean/Software/Scala/scala-2.11.7/bin/scala
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home
    export SPARK_MASTER_IP=192.168.31.60
    export SPARK_WORKER_MEMORY=1g
    export SPARK_EXECUTOR_CORES=2
    export master=spark://192.168.100.60:7077

# slaves
    # A Spark Worker will be started on each of the machines listed below.
    192.168.31.61
    192.168.31.62

# 传输配置文件(此操作可以在脚本内完成)
    scp spark-env.sh root@192.168.31.60:/opt/apps/spark/spark-2.2.1/conf/
    scp spark-env.sh root@192.168.31.61:/opt/apps/spark/spark-2.2.1/conf/
    scp spark-env.sh root@192.168.31.62:/opt/apps/spark/spark-2.2.1/conf/
    
    scp slaves root@192.168.31.60:/opt/apps/spark/spark-2.2.1/conf/
    scp slaves root@192.168.31.61:/opt/apps/spark/spark-2.2.1/conf/
    scp slaves root@192.168.31.62:/opt/apps/spark/spark-2.2.1/conf/

*  步骤3: 通过`start-all.sh`脚本启动`Spark集群`.
 *  步骤4: 通过访问`master节点`的`8080`端口进行访问.  
    [http://192.168.31.60:8080][http_192.168.31.60_8080]

--------------------

### Tips ###

> 修改Web UI端口  
> 生产有时需要更改默认的WebUI端口`8080`.

修改Spark Web UI默认的8080端口  
修改SPARK\_HOME/sbin目录下`start-master.sh`，查找8080定位并修改为你想要的端口即可.

if [ "$SPARK_MASTER_WEBUI_PORT" = "" ]; then
      SPARK_MASTER_WEBUI_PORT=8087
    fi

> 修改PID记录地址  
> [Spark集群无法停止:“no org.apache.spark.deploy.master.Master to stop”][Spark_no org.apache.spark.deploy.master.Master to stop]

> 修改Logs地址  
> 更改log4j.properties文件即可.  
> [修改spark日志配置][spark]

> 升级Log4j2  
> 待补充

> 日志内Spark的真实启动命令
> 
>     Spark Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java -cp /Users/Sean/Software/Spark/spark-2.2.1-bin-hadoop2.7/conf/:/Users/Sean/Software/Spark/spark-2.2.1-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host localhost --port 7077 --webui-port 8080

--------------------

### Reference ###

\[1\] [Spark Standalone Mode][Spark Standalone]  
\[2\] [修改HBase或者Spark默认的8080端口][HBase_Spark_8080]

[Spark Standalone]: http://spark.apache.org/docs/latest/spark-standalone.html
[Running Spark on YARN]: http://spark.apache.org/docs/latest/running-on-yarn.html
[Running Spark on Kubernetes]: http://spark.apache.org/docs/latest/running-on-kubernetes.html
[Running Spark on Mesos]: http://spark.apache.org/docs/latest/running-on-mesos.html
[http_localhost_8080]: http://localhost:8080/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly95YW54bWwuYmxvZy5jc2RuLm5ldA_size_16_color_FFFFFF_t_70]: /images/20220226/397a6ee6bc7b459795ed190d91694392.png
[http_192.168.31.60_8080]: http://192.168.31.60:8080
[Spark_no org.apache.spark.deploy.master.Master to stop]: https://yanxml.blog.csdn.net/article/details/80137353
[spark]: https://blog.csdn.net/javamoo/article/details/77266956
[HBase_Spark_8080]: https://blog.csdn.net/u011095110/article/details/84322266