Hadoop——MR与RM HA部署

落日映苍穹つ 2022-05-26 01:41 187阅读 0赞

本节目标：

1、掌握MR的搭建步骤

2、掌握RM HA的部署

本节在[HDFS HA部署][HDFS HA]基础上完成。

节点部署情况：

<table> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> NN </td> 
 <td style="vertical-align:top;"> DN </td> 
 <td style="vertical-align:top;"> JN </td> 
 <td style="vertical-align:top;"> ZK </td> 
 <td style="vertical-align:top;"> ZKFC </td> 
 <td style="vertical-align:top;"> RM </td> 
 <td style="vertical-align:top;"> NM </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> master(192.168.85.10) </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> slave1(192.168.85.11) </td> 
 <td style="vertical-align:top;"> l&nbsp;&nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> slave2(192.168.85.12) </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;"> slave3(192.168.85.13) </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> &nbsp; </td> 
 <td style="vertical-align:top;"> 1 </td> 
 <td style="vertical-align:top;"> l&nbsp; &nbsp; </td> 
 </tr> 
 </tbody> 
</table>

1、将mapred-site.xml模板改名 
2、mapred-site.xml 
    <property> 
        <name>mapreduce.framework.name</name> 
        <value>yarn</value> 
    </property> 
3、yarn-site.xml

<property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
 </property>
 <property>
 <name>yarn.resourcemanager.ha.enabled</name>
 <value>true</value>
 </property>
 <property>
 <name>yarn.resourcemanager.cluster-id</name>
 <value>RMHA</value>
 </property>
 <property>
 <name>yarn.resourcemanager.ha.rm-ids</name>
 <value>rm1,rm2</value>
 </property>
 <property>
 <name>yarn.resourcemanager.hostname.rm1</name>
 <value>slave2</value>
 </property>
 <property>
 <name>yarn.resourcemanager.hostname.rm2</name>
 <value>slave3</value>
 </property>
 <property>
 <name>yarn.resourcemanager.webapp.address.rm1</name>
 <value>slave2:8088</value>
 </property>
 <property>
 <name>yarn.resourcemanager.webapp.address.rm2</name>
 <value>slave3:8088</value>
 </property>
 <property>
 <name>yarn.resourcemanager.zk-address</name>
 <value>master:2181,slavee1:2181,slave2:2181</value>
 </property>

4、同步到其他节点  
        scp ./\*.xml slave1:/usr/local/src/hadoop-2.7.6/etc/hadoop  
5.启动集群

先启动zk  
start-all.sh   
6、RM节点启动  
yarn-daemon.sh start resourcemanager  
7、查看RM后台8088端口是否开启  
netstat -npl

![20180423082050100][]

8、webUI检查

slave2:8088

slave3:8088

![20180423082336485][]

8、测试RM HA是否有效：分别在slave2，slave3节点上执行，yarn-daemon.sh stop resourcemanager，检测RM是否自动切换.

也可以查看当前 RM 的 active 和 standby 状态  
yarn rmadmin -getServiceState rm1

yarn rmadmin -getServiceState rm2

![20180423090217394][]

从以上配置过程可以看出，salves 文件既是DN的配置文件，也是NM的配置文件。

eclipse操作HDFS时注意：

![20180423084418891][]

8020端口是在hdfs-site.xml配置的。

![20180423090533304][]

更新src目录下的core-site.xml、hdfs-site.xml配置文件。

更多细节参考官方文档中[core-default.xml][]、[hdfs-default.xml][]、[mapred-default.xml][]、[yarn-default.xml][]、以及[RM HA][]配置

[HDFS HA]: https://blog.csdn.net/ym01213/article/details/80038652
[20180423082050100]: /images/20220526/f17002244ef340e4baaef255d45cab18.png
[20180423082336485]: /images/20220526/f66b40da4cd248138a24758958df2157.png
[20180423090217394]: /images/20220526/68994f9c5ebb459ea217d5dbc5914c6f.png
[20180423084418891]: /images/20220526/115494cc9a3f495cb16110ba9a451d7b.png
[20180423090533304]: /images/20220526/64142c83d9c944fabac35fae9f4aead8.png
[core-default.xml]: http://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-common/core-default.xml
[hdfs-default.xml]: http://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
[mapred-default.xml]: http://hadoop.apache.org/docs/r2.7.6/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
[yarn-default.xml]: http://hadoop.apache.org/docs/r2.7.6/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
[RM HA]: http://hadoop.apache.org/docs/r2.7.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html