解决spark单点故障问题——Spark HA部署

刺骨的言语ヽ痛彻心扉 2022-10-23 05:58 8阅读 0赞

### Spark HA的部署过程以及测试结果 ###

*   *  1. 查看zookeeper 的zoo.cfg文件
     *  2. 配置spark-env.sh 文件（三个节点)
     *  3. 单点故障操作

[简单易懂，手把手带小白用VMware虚拟机安装Linux centos7系统][VMware_Linux centos7]

[Hadoop集群搭建及配置〇 —— Hadoop组件获取 & 传输文件][Hadoop_ _ Hadoop_ _]

[Hadoop集群搭建及配置① —— 克隆节点][Hadoop_ _]

[Hadoop集群搭建及配置② —— 网络IP配置，连接网络][Hadoop_ _ _IP]

[Hadoop集群搭建及配置③ —— 基础环境搭建][Hadoop_ _ 1]

[Hadoop集群搭建及配置④ —— JDK简介及其安装][Hadoop_ _ JDK]

[Hadoop集群搭建及配置⑤ —— Zookeeper 讲解及安装][Hadoop_ _ Zookeeper]

[Hadoop集群搭建及配置⑥ —— Hadoop组件安装及配置][Hadoop_ _ Hadoop]

[Hadoop集群搭建及配置⑦—— Spark&Scala安装配置][Hadoop_ Spark_Scala]

[Hadoop集群搭建及配置⑧——Hbase的安装配置][Hadoop_Hbase]

[HDFS的Java API编程配置][HDFS_Java API]

[Hdfs的 java API 编程实现基本功能][Hdfs_ java API]

[HBase API java表操作][HBase API java]

[Spark RDD弹性分布式数据集——理论][Spark RDD]

[Spark RDD弹性分布式数据集——实践][Spark RDD 1]

## 1. 查看zookeeper 的zoo.cfg文件 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]

## 2. 配置spark-env.sh 文件（三个节点) ##

vi spark-env.sh
    注释掉masterIP，添加：
    
    export SPARK_MASTER_PORT=7077
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER
    -Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181
    -Dspark.deploy.zookeeper.dir=/usr/spark/spark-2.4.0-bin-hadoop2.7"
    
    保存退出后，在salve1的spark目录：
    cd /usr/spark/spark-2.4.0-bin-hadoop2.7/下，
    启动：sbin/start-master.sh

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]  
可以看到slave1已经启动spark–Master，网页查看状态  
![在这里插入图片描述][20210314182017313.png]

## 3. 单点故障操作 ##

可以看到**slave1:8080**状态 Status:STANDBY。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]

在master节点停止掉spark，可以看到，**slave1:8080**的状态变为**ALIVE**，表示解决了单点故障问题。![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]

[VMware_Linux centos7]: https://blog.csdn.net/weixin_44775255/article/details/113652758
[Hadoop_ _ Hadoop_ _]: https://blog.csdn.net/weixin_44775255/article/details/113783563
[Hadoop_ _]: https://blog.csdn.net/weixin_44775255/article/details/113754483
[Hadoop_ _ _IP]: https://blog.csdn.net/weixin_44775255/article/details/113754813
[Hadoop_ _ 1]: https://blog.csdn.net/weixin_44775255/article/details/113755997
[Hadoop_ _ JDK]: https://blog.csdn.net/weixin_44775255/article/details/113771546
[Hadoop_ _ Zookeeper]: https://blog.csdn.net/weixin_44775255/article/details/113772666
[Hadoop_ _ Hadoop]: https://blog.csdn.net/weixin_44775255/article/details/113775449
[Hadoop_ Spark_Scala]: https://blog.csdn.net/weixin_44775255/article/details/114518207
[Hadoop_Hbase]: https://blog.csdn.net/weixin_44775255/article/details/114676669
[HDFS_Java API]: https://blog.csdn.net/weixin_44775255/article/details/114799064
[Hdfs_ java API]: https://blog.csdn.net/weixin_44775255/article/details/114809298
[HBase API java]: https://blog.csdn.net/weixin_44775255/article/details/114821297
[Spark RDD]: https://blog.csdn.net/weixin_44775255/article/details/115536606
[Spark RDD 1]: https://blog.csdn.net/weixin_44775255/article/details/115560667
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]: /images/20221023/94a02c9c86204ac39d63db3c2cb4b147.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]: /images/20221023/db129aa453544766b852b85039f6fb4c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20210314181952898.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ==,size_16,color_FFFFFF,t_70
[20210314182017313.png]: https://img-blog.csdnimg.cn/20210314182017313.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2021031418204696.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/2021031418205845.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20210314182123151.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ==,size_16,color_FFFFFF,t_70