记一次基于Hadoop 3.3.0 安装部署 Spark 3.0.0 分布式集群

以你之姓@ 2022-12-29 14:06 241阅读 0赞

# 一、基本信息 #

官网  
[http://spark.apache.org/][http_spark.apache.org]  
Apache Spark 官方文档中文版（Spark 2.2.0）  
[http://spark.apachecn.org/\#/][http_spark.apachecn.org]  
Spark 中文文档（Spark 2.2.0）  
[http://doc.codingdict.com/spark/1/][http_doc.codingdict.com_spark_1]  
《Spark 官方文档》Spark快速入门（Spark 1.6.0）  
[http://ifeve.com/spark-quick-start/][http_ifeve.com_spark-quick-start]  
官网实例  
[http://spark.apache.org/examples.html][http_spark.apache.org_examples.html]  
官网新发布  
[http://spark.apache.org/news/spark-3-0-0-released.html][http_spark.apache.org_news_spark-3-0-0-released.html]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70]  
官网下载页面  
[http://spark.apache.org/downloads.html][http_spark.apache.org_downloads.html]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 1]  
各个版本下载列表  
[https://archive.apache.org/dist/spark/][https_archive.apache.org_dist_spark]  
本文使用版本下载目录  
[https://archive.apache.org/dist/spark/spark-3.0.0/][https_archive.apache.org_dist_spark_spark-3.0.0]  
Spark基本架构及原理  
[https://www.cnblogs.com/Mayny/p/9330436.html][https_www.cnblogs.com_Mayny_p_9330436.html]

# 二、基础集群环境 #

1、Centos7.x 安装OpenJDK1.8  
[https://blog.csdn.net/llwy1428/article/details/111144659][https_blog.csdn.net_llwy1428_article_details_111144659]  
2、Centos7.x 配置 Scala2.12.12  
[https://blog.csdn.net/llwy1428/article/details/111601924][https_blog.csdn.net_llwy1428_article_details_111601924]  
3、Centos7.x 安装部署 Hadoop 3.x HDFS基础环境  
[https://blog.csdn.net/llwy1428/article/details/111144524][https_blog.csdn.net_llwy1428_article_details_111144524]  
4、Centos7.x 安装部署 Zookeeper 3.6.2 集群  
[https://blog.csdn.net/llwy1428/article/details/111601567][https_blog.csdn.net_llwy1428_article_details_111601567]  
5、Centos7.x Hadoop 3.x 部署 YARN 集群  
[https://blog.csdn.net/llwy1428/article/details/111464707][https_blog.csdn.net_llwy1428_article_details_111464707]  
6、Hadoop 3.x 集群 YARN 启动 timelineserver（ApplicationHistoryServer）  
[https://blog.csdn.net/llwy1428/article/details/112417384][https_blog.csdn.net_llwy1428_article_details_112417384]

# 三、安装、部署 #

1、下载文件 spark-3.0.0-bin-hadoop3.2.tgz

[root@master ~]# wget -P /usr/bigdata/ https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop3.2.tgz

![在这里插入图片描述][20201222234310336.png]  
2、进入文件的下载目录

[root@master ~]# cd /usr/bigdata/

3、解压缩、查看文件

[root@master bigdata]# tar zxf spark-3.0.0-bin-hadoop3.2.tgz

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 2]  
4、进入 spark 的配置文件目录

[root@master ~]# cd /usr/bigdata/spark-3.0.0-bin-hadoop3.2/conf/

![在这里插入图片描述][20201223001005298.png]  
5、复制、重命名配置文件 slaves.template

[root@master conf]# cp slaves.template slaves

6、编辑配置文件 slaves

[root@master conf]# vim slaves

7、编辑、写入内容 配置 子节点的主机名列表

slave1
    slave2
    slave3

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 3]  
8、复制、重命名配置文件 spark-env.sh.template

[root@master conf]# cp spark-env.sh.template spark-env.sh

9、编辑配置文件

[root@master conf]# vim spark-env.sh

10、编辑内容

HADOOP_CONF_DIR=/usr/bigdata/hadoop-3.3.0/etc/hadoop/

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 4]  
11、进入 spark 的 sbin 目录

[root@master ~]# cd /usr/bigdata/spark-3.0.0-bin-hadoop3.2/sbin/

12、编辑配置文件 spark-config.sh

[root@master sbin]# vim spark-config.sh

13、编辑内容

export JAVA_HOME="/usr/lib/jvm/java-1.8.0"

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 5]  
14、把编辑、配置完毕的目录及文件发送给指定的 spark 的各个子节点

[root@master ~]# scp -r /usr/bigdata/spark-3.0.0-bin-hadoop3.2 root@slave1:/usr/bigdata/
    [root@master ~]# scp -r /usr/bigdata/spark-3.0.0-bin-hadoop3.2 root@slave2:/usr/bigdata/
    [root@master ~]# scp -r /usr/bigdata/spark-3.0.0-bin-hadoop3.2 root@slave3:/usr/bigdata/

15、启动 Hadoop 的 HDFS 服务

[root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/start-dfs.sh

16、启动 Zookeeper 集群

[root@slave1 ~]# /usr/bigdata/apache-zookeeper-3.6.2-bin/bin/zkServer.sh start
    [root@slave2 ~]# /usr/bigdata/apache-zookeeper-3.6.2-bin/bin/zkServer.sh start
    [root@slave3 ~]# /usr/bigdata/apache-zookeeper-3.6.2-bin/bin/zkServer.sh start

17、启动 YARN 的相关服务

[root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/start-yarn.sh
    [root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/yarn-daemon.sh start timelineserver

18、启动 jobhistoryserver

[root@master ~]# /usr/bigdata/hadoop-3.3.0/sbin/mr-jobhistory-daemon.sh start historyserver

19、进入 Spark 的 sbin 目录启动 spark

[root@master ~]# cd /usr/bigdata/spark-3.0.0-bin-hadoop3.2/sbin/

启动 spark

[root@master sbin]# ./start-all.sh

或

[root@master ~]# /usr/bigdata/spark-3.0.0-bin-hadoop3.2/sbin/start-all.sh

![在这里插入图片描述][20210111230204588.png]  
20、查看各个节点服务的启动状态

[root@master ~]# jps
    [root@slave1 ~]# jps
    [root@slave2 ~]# jps
    [root@slave3 ~]# jps

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 6]  
21、各个节点服务启动及进程

[root@master ~]# netstat -lntp
    [root@slave1 ~]# netstat -lntp
    [root@slave2 ~]# netstat -lntp
    [root@slave3 ~]# netstat -lntp

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 7]  
22、停止 Spark 服务

[root@master ~]# /usr/bigdata/spark-3.0.0-bin-hadoop3.2/sbin/stop-all.sh

![在这里插入图片描述][20201223010940310.png]

# 四、浏览器查看 Spark 的 Web UI #

1、在浏览器地址栏录入 http://192.168.11.21:8080/  
说明：192.168.11.21 是 master 节点的 IP ，Spark Web UI 的默认端口 8080 ，如需修改端口可更改其配置文件  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 8]  
2、点击各个 Worker 查看各个节点的详情 例如节点 slave1  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 9]

至此，基于Hadoop 3.3.0 安装部署 Spark 3.0.0 分布式集群操作完毕，希望能够对您有所帮助！

[http_spark.apache.org]: http://spark.apache.org/
[http_spark.apachecn.org]: http://spark.apachecn.org/#/
[http_doc.codingdict.com_spark_1]: http://doc.codingdict.com/spark/1/
[http_ifeve.com_spark-quick-start]: http://ifeve.com/spark-quick-start/
[http_spark.apache.org_examples.html]: http://spark.apache.org/examples.html
[http_spark.apache.org_news_spark-3-0-0-released.html]: http://spark.apache.org/news/spark-3-0-0-released.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70]: /images/20221120/8269364d587044a689475e2dbf8a65d4.png
[http_spark.apache.org_downloads.html]: http://spark.apache.org/downloads.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 1]: /images/20221120/d1df57a9f7d84c4da84a00d64ab49480.png
[https_archive.apache.org_dist_spark]: https://archive.apache.org/dist/spark/
[https_archive.apache.org_dist_spark_spark-3.0.0]: https://archive.apache.org/dist/spark/spark-3.0.0/
[https_www.cnblogs.com_Mayny_p_9330436.html]: https://www.cnblogs.com/Mayny/p/9330436.html
[https_blog.csdn.net_llwy1428_article_details_111144659]: https://blog.csdn.net/llwy1428/article/details/111144659
[https_blog.csdn.net_llwy1428_article_details_111601924]: https://blog.csdn.net/llwy1428/article/details/111601924
[https_blog.csdn.net_llwy1428_article_details_111144524]: https://blog.csdn.net/llwy1428/article/details/111144524
[https_blog.csdn.net_llwy1428_article_details_111601567]: https://blog.csdn.net/llwy1428/article/details/111601567
[https_blog.csdn.net_llwy1428_article_details_111464707]: https://blog.csdn.net/llwy1428/article/details/111464707
[https_blog.csdn.net_llwy1428_article_details_112417384]: https://blog.csdn.net/llwy1428/article/details/112417384
[20201222234310336.png]: /images/20221120/1845af71a4d94453810f6ff7770da969.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 2]: /images/20221120/78ab8060b40547c489840301d49f669e.png
[20201223001005298.png]: /images/20221120/f16d1d4d0d634158a36625f5bfe1a9fc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 3]: /images/20221120/e72b92f8e9864f058398e3fae8432e9b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 4]: /images/20221120/d12dd43f986747f698e1c142b0ee027c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 5]: /images/20221120/6844e70a00fb48e8978335752695f5e2.png
[20210111230204588.png]: /images/20221120/a800f336024b4b05892a4e217a7c9336.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 6]: /images/20221120/4c6b3b16e41d40cd81af1965209471a8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 7]: /images/20221120/d6b452d60d8c4e6690845ab83c61d2ed.png
[20201223010940310.png]: /images/20221120/93e59a889d2148c6a6ee1109891704d5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 8]: /images/20221120/6211e77f0c3048fa82e3af375129044f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xsd3kxNDI4_size_16_color_FFFFFF_t_70 9]: /images/20221120/778b70f63c0749eba32b3a1579c23f41.png