Hadoop集群搭建及配置⑦—— Spark&Scala安装配置

电玩女神 2022-11-06 04:57 249阅读 0赞

### Spark&Scala安装配置 ###

*   *  1.1 用FTP把压缩包发送到linux master
     *  1.2 linux系统下的安装scala
     *   *   *  1.2.1创建scala文件
             *  1.2.2 解压缩
             *  1.2.3 配置Scala环境变量
             *  1.2.4 分发文件
     *  1.3 安装Spark
     *   *   *  1.3.1 创建文件
             *  1.3.2 解压并配置文件
             *  1.3.3 配置spark从节点，修改slaves文件
             *  1.3.4 配置spark环境（三个节点）
             *  1.3.5 分发spark文件
             *  1.3.6 启动spark集群（master）
             *  1.3.7 进入spark-shell交互

--------------------

[Hadoop集群搭建及配置⑤ —— Zookeeper 讲解及安装][Hadoop_ _ Zookeeper]

[Hadoop集群搭建及配置⑥ —— Hadoop组件安装及配置][Hadoop_ _ Hadoop]

[Hadoop集群搭建及配置⑦—— Spark&Scala安装配置][Hadoop_ Spark_Scala]

[Hadoop集群搭建及配置⑧——Hbase的安装配置][Hadoop_Hbase]

[解决spark单点故障问题——Spark HA部署][spark_Spark HA]

--------------------

## 1.1 用FTP把压缩包发送到linux master ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]

--------------------

## 1.2 linux系统下的安装scala ##

#### 1.2.1创建scala文件 ####

mkdir /usr/scala

#### 1.2.2 解压缩 ####

tar -zxvf /home/master/Downloads/scala-2.11.12.tgz -C /usr/scala

#### 1.2.3 配置Scala环境变量 ####

vi /etc/profile
    #scala home(三个节点都要)
    export SCALA_HOME=/usr/scala/scala-2.11.12 # 与Scala文件夹路径一致
    export PATH=$SCALA_HOME/bin:$PATH

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]

# 保存退出后，生效环境，再查看Scala版本
    source /etc/profile 
    scala -version

--------------------

#### 1.2.4 分发文件 ####

scp -r /usr/scala root@slave1:/usr
    scp -r /usr/scala root@slave2:/usr

![在这里插入图片描述][20210308095313310.png]

--------------------

## 1.3 安装Spark ##

#### 1.3.1 创建文件 ####

mkdir -p /usr/spark/

#### 1.3.2 解压并配置文件 ####

tar -zxvf /home/master/Downloads/spark-2.4.0-bin-hadoop2.7.tgz -C /usr/spark/
    
    cd /usr/spark/spark-2.4.0-bin-hadoop2.7/conf
    
    # 把配置文件spark-env.sh.template复制为spark-env.sh并编辑
    cp spark-env.sh.template spark-env.sh
    
    vi spark-env.sh
    # spark配置
    export SPARK_MASTER_IP=master
    export SCALA_HOME=/usr/scala/scala-2.11.12
    export SPARK_WORKER_MEMORY=8g
    export JAVA_HOME=/usr/java/jdk1.8.0_171
    export HADOOP_HOME=/usr/hadoop/hadoop-2.7.3
    export HADOOP_CONF_DIR=/usr/hadoop/hadoop-2.7.3/etc/hadoop

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]

#### 1.3.3 配置spark从节点，修改slaves文件 ####

cp slaves.template slaves
    
    vi slaves
    slave1
    slave2

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]  
![在这里插入图片描述][20210308095419610.png]

#### 1.3.4 配置spark环境（三个节点） ####

vi /etc/profile
    
    #spark home
    export SPARK_HOME=/usr/spark/spark-2.4.0-bin-hadoop2.7
    export PATH=$SPARK_HOME/bin:$PATH
    source /etc/profile

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]

#### 1.3.5 分发spark文件 ####

scp -r /usr/spark root@slave1:/usr

#### 1.3.6 启动spark集群（master） ####

在 /usr/spark/spark-2.4.0-bin-hadoop2.7/ 目录下:
    sbin/start-all.sh
    # 可以用jps 查看启动情况

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]  
![在这里插入图片描述][20210308095542767.png]  
启动成功 打开浏览器 `192.168.142.128:8080`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 6]

--------------------

#### 1.3.7 进入spark-shell交互 ####

spark-shell
    # 输入命令，执行scala
    println("hello,world")

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 7]  
配置完成。

[Hadoop_ _ Zookeeper]: https://blog.csdn.net/weixin_44775255/article/details/113772666
[Hadoop_ _ Hadoop]: https://blog.csdn.net/weixin_44775255/article/details/113775449
[Hadoop_ Spark_Scala]: https://blog.csdn.net/weixin_44775255/article/details/114518207
[Hadoop_Hbase]: https://blog.csdn.net/weixin_44775255/article/details/114676669
[spark_Spark HA]: https://blog.csdn.net/weixin_44775255/article/details/114796573
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]: /images/20221023/56e38531b37043d9ae7111eea501a670.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]: /images/20221023/dd849299dc3e4d7abec96c953824ace4.png
[20210308095313310.png]: /images/20221023/a9b5dd7a63124b188a28ec4fc121fadc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]: /images/20221023/86de2fd831ec47089bc3f9e190876448.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]: /images/20221023/00b15f35637044a88989db260eb1c62b.png
[20210308095419610.png]: /images/20221023/df07a5733a93412c8ca35b2b13fdc3ea.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]: /images/20221023/fb75561837e6449ca979e32d752ef320.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]: /images/20221023/a2f310f1b589492fa29bac0ad599f285.png
[20210308095542767.png]: /images/20221023/f09353d71e6c422db9239ad14bbc2543.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 6]: /images/20221023/32719899b52e4a8cab9ea42d2b4c5482.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 7]: /images/20221023/beaf0f00d3014bd38769af2702b9c1f3.png