解决spark单点故障问题——Spark HA部署

「爱情、让人受尽委屈。」 2022-11-09 06:04 168阅读 0赞

### Spark HA的部署过程以及测试结果 ###

*   *  1. 查看zookeeper 的zoo.cfg文件
     *  2. 配置spark-env.sh 文件（三个节点)
     *  3. 单点故障操作

[简单易懂，手把手带小白用VMware虚拟机安装Linux centos7系统][VMware_Linux centos7]

[Hadoop集群搭建及配置〇 —— Hadoop组件获取 & 传输文件][Hadoop_ _ Hadoop_ _]

[Hadoop集群搭建及配置① —— 克隆节点][Hadoop_ _]

[Hadoop集群搭建及配置② —— 网络IP配置，连接网络][Hadoop_ _ _IP]

[Hadoop集群搭建及配置③ —— 基础环境搭建][Hadoop_ _ 1]

[Hadoop集群搭建及配置④ —— JDK简介及其安装][Hadoop_ _ JDK]

[Hadoop集群搭建及配置⑤ —— Zookeeper 讲解及安装][Hadoop_ _ Zookeeper]

[Hadoop集群搭建及配置⑥ —— Hadoop组件安装及配置][Hadoop_ _ Hadoop]

[Hadoop集群搭建及配置⑦—— Spark&Scala安装配置][Hadoop_ Spark_Scala]

[Hadoop集群搭建及配置⑧——Hbase的安装配置][Hadoop_Hbase]

[HDFS的Java API编程配置][HDFS_Java API]

[Hdfs的 java API 编程实现基本功能][Hdfs_ java API]

[HBase API java表操作][HBase API java]

[Spark RDD弹性分布式数据集——理论][Spark RDD]

[Spark RDD弹性分布式数据集——实践][Spark RDD 1]

## 1. 查看zookeeper 的zoo.cfg文件 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]

## 2. 配置spark-env.sh 文件（三个节点) ##

vi spark-env.sh
    注释掉masterIP，添加：
    
    export SPARK_MASTER_PORT=7077
    export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER
    -Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181
    -Dspark.deploy.zookeeper.dir=/usr/spark/spark-2.4.0-bin-hadoop2.7"
    
    保存退出后，在salve1的spark目录：
    cd /usr/spark/spark-2.4.0-bin-hadoop2.7/下，
    启动：sbin/start-master.sh

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]  
可以看到slave1已经启动spark–Master，网页查看状态  
![在这里插入图片描述][20210314182017313.png]

## 3. 单点故障操作 ##

可以看到**slave1:8080**状态 Status:STANDBY。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]

在master节点停止掉spark，可以看到，**slave1:8080**的状态变为**ALIVE**，表示解决了单点故障问题。![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]

[VMware_Linux centos7]: https://blog.csdn.net/weixin_44775255/article/details/113652758
[Hadoop_ _ Hadoop_ _]: https://blog.csdn.net/weixin_44775255/article/details/113783563
[Hadoop_ _]: https://blog.csdn.net/weixin_44775255/article/details/113754483
[Hadoop_ _ _IP]: https://blog.csdn.net/weixin_44775255/article/details/113754813
[Hadoop_ _ 1]: https://blog.csdn.net/weixin_44775255/article/details/113755997
[Hadoop_ _ JDK]: https://blog.csdn.net/weixin_44775255/article/details/113771546
[Hadoop_ _ Zookeeper]: https://blog.csdn.net/weixin_44775255/article/details/113772666
[Hadoop_ _ Hadoop]: https://blog.csdn.net/weixin_44775255/article/details/113775449
[Hadoop_ Spark_Scala]: https://blog.csdn.net/weixin_44775255/article/details/114518207
[Hadoop_Hbase]: https://blog.csdn.net/weixin_44775255/article/details/114676669
[HDFS_Java API]: https://blog.csdn.net/weixin_44775255/article/details/114799064
[Hdfs_ java API]: https://blog.csdn.net/weixin_44775255/article/details/114809298
[HBase API java]: https://blog.csdn.net/weixin_44775255/article/details/114821297
[Spark RDD]: https://blog.csdn.net/weixin_44775255/article/details/115536606
[Spark RDD 1]: https://blog.csdn.net/weixin_44775255/article/details/115560667
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]: /images/20221023/94a02c9c86204ac39d63db3c2cb4b147.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]: /images/20221023/db129aa453544766b852b85039f6fb4c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]: /images/20221023/b128b1829098480a9683697ec9ee4eec.png
[20210314182017313.png]: /images/20221023/8fb5be146ad9434d974da2bf7428f7ae.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]: /images/20221023/ceb6e9da002c44adbbb213dd883bb5cc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]: /images/20221023/8f8d66255a0a43f988425b5dcd61b0c6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]: /images/20221023/ebccb27061f6460fa3ad76443d12c139.png