Hadoop HA + Federation + YARN HA配置实现（九）

Love The Way You Lie 2022-04-25 06:52 271阅读 0赞

**集群规划**

![20170328104414639][]

**配置步骤**  
**一、 前提准备**  
1） 安装好Hadoop2.0集群 《==参考《HADOOP HA+Federation集群实现（七）》及以前的文章  
2） 安装好zookeeper集群  
**二、修改如下配置文件**  
1） yarn-site.xml  
2） mapred-site.xml  
3） yarn-env.sh  
**三、启动服务**  
1） 启动HDFS  
2） 启动YARN，通过yarn-daemons.sh来启动  
**四、测试验证**  
1） 查看状态  
yarn rmadmin -getServiceState rm1/rm2  
2） 模拟故障切换

**实施步骤**  
**1） 编辑yarn-env.sh**  
\[hadoop@master ~\]$ cd /opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ vi yarn-env.sh  
export JAVA\_HOME=/usr/java/default

**2）编辑mapred-site.xml**  
\[hadoop@master hadoop\]$ vi mapred-site.xml  
<property>  
   <name>mapreduce.framework.name</name>  
   <value>yarn</value>  
 </property>

**3）编辑yarn-site.xml**  
\[hadoop@master hadoop\]$ vi yarn-site.xml

<property>  
   <name>yarn.nodemanager.aux-services</name>  
   <value>mapreduce\_shuffle</value>  
  </property>  
  
<property>  
   <name>yarn.resourcemanager.cluster-id</name>  
   <value>hd260-yarn</value>  
  </property>  
<property>  
   <name>yarn.resourcemanager.ha.enabled</name>  
   <value>true</value>  
  </property>  
  <property>  
   <name>yarn.resourcemanager.ha.rm-ids</name>  
   <value>rm1,rm2</value>  
  </property>  
  <property>  
   <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>  
   <value>true</value>  
  </property>  
  <property>  
   <name>yarn.resourcemanager.hostname.rm1</name>  
   <value>master</value>  
  </property>  
  <property>  
   <name>yarn.resourcemanager.hostname.rm2</name>  
   <value>hadoop04</value>  
  </property>  
  
  <property>  
   <name>yarn.resourcemanager.recovery.enabled</name>  
   <value>true</value>  
  </property>  
  <property>  
   <name>yarn.resourcemanager.store.class</name>  
   <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>  
  </property>  
  <property>  
      <name>yarn.resourcemanager.zk-address</name>  
      <value>hadoop04:2181,slave1:2181,slave2:2181</value>  
   </property>

<property>  
      <name>yarn.resourcemanager.address.rm1</name>  
      <value>master:8032</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.scheduler.address.rm1</name>  
      <value>master:8030</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.webapp.address.rm1</name>  
      <value>master:8088</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.resource-tracker.address.rm1</name>  
      <value>master:8031</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.admin.address.rm1</name>  
      <value>master:8033</value>  
   </property>

<property>  
      <name>yarn.resourcemanager.address.rm2</name>  
      <value>hadoop04:8032</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.scheduler.address.rm2</name>  
      <value>hadoop04:8030</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.webapp.address.rm2</name>  
      <value>hadoop04:8088</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.resource-tracker.address.rm2</name>  
      <value>hadoop04:8031</value>  
   </property>  
  <property>  
      <name>yarn.resourcemanager.admin.address.rm2</name>  
      <value>hadoop04:8033</value>  
   </property>

**4）同步配置文件**  
\[hadoop@master hadoop\]$ scp yarn-env.sh slave1:/opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ scp yarn-env.sh slave2:/opt/hadoop/etc/hadoop/    
\[hadoop@master hadoop\]$ scp yarn-env.sh hadoop04:/opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ scp yarn-site.xml slave1:/opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ scp yarn-site.xml slave2:/opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ scp yarn-site.xml hadoop04:/opt/hadoop/etc/hadoop/     
\[hadoop@master hadoop\]$ scp mapred-site.xml hadoop04:/opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ scp mapred-site.xml slave1:/opt/hadoop/etc/hadoop/      
\[hadoop@master hadoop\]$ scp mapred-site.xml slave2:/opt/hadoop/etc/hadoop/  
**5）启动zookeeper**  
\[hadoop@slave1 ~\]$ zkServer.sh start  
\[hadoop@slave2 ~\]$ zkServer.sh start  
\[hadoop@hadoop04 ~\]$ zkServer.sh start

**6）启动journalnode**  
\[hadoop@master hadoop\]$ hadoop-daemons.sh --hostnames 'slave1 slave2 hadoop04' start journalnode

**7）启动namenode**  《==如果没有配置hdfs ha，那就不用执行第六步，直接启动namenode  
\[hadoop@master hadoop\]$ hadoop-daemons.sh --hostnames 'master slave1 slave2 hadoop04' start namenode

**8）启动datanode**  
\[hadoop@master ~\]$ hadoop-daemons.sh start datanode

**9）启动 zkfc**  
\[hadoop@master ~\]$ hadoop-daemons.sh --hostnames 'master slave1 slave2 hadoop04' start zkfc

**10）分别启动resource manager**  
\[hadoop@master ~\]$ yarn-daemon.sh start resourcemanager  
启动时报错，部分报错内容：  
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*/  
2017-03-27 03:26:23,310 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered UNIX signal handlers   
for \[TERM, HUP, INT\]  
2017-03-27 03:26:24,688 INFO org.apache.hadoop.conf.Configuration: found resource core-site.xml at file:/opt/hadoop-2.6.0/et  
c/hadoop/core-site.xml  
2017-03-27 03:26:25,580 FATAL org.apache.hadoop.conf.Configuration: error parsing conf java.io.BufferedInputStream@4a44bd92  
org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 'include' failed, and no 'fallback' element was found.  
        at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)  
        at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)  
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:121)  
        at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2352)  
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2425)  
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2374)  
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2281)  
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:888)  
        at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:907)  
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2068)  
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2098)  
        at org.apache.hadoop.security.Groups.<init>(Groups.java:70)  
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)  
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(Groups.java:295)  
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)  
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)  
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)  
2017-03-27 03:26:25,595 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state INITED; caus  
e: java.lang.RuntimeException: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30;   
An 'include' failed, and no 'fallback' element was found.  
java.lang.RuntimeException: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An   
'include' failed, and no 'fallback' element was found.  
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2106)  
        at org.apache.hadoop.security.Groups.<init>(Groups.java:70)  
        at org.apache.hadoop.security.Groups.<init>(Groups.java:66)  
        at org.apache.hadoop.security.Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(Groups.java:295)  
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:211)  
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)  
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1210)  
Caused by: java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 20; columnNumber: 30; An 'include' failed,  
 and no 'fallback' element was found.

**11）修改core-site.xml         <==解决报错内容**  
  
<configuration>  
 <property>  
  <name>fs.defaultFS</name>  
  <value>viewfs://nsX</value>  
 </property>

<property>  
  <name>hadoop.tmp.dir</name>  
  <value>/opt/hadoop/tmp</value>  
 </property>

<property>  
  <name>dfs.journalnode.edits.dir</name>  
  <value>/opt/hadoop/journalnode/data</value>  
 </property>  
 <property>  
  <name>ha.zookeeper.quorum</name>  
  <value>hadoop04:2181,slave1:2181,slave2:2181</value>  
 </property>  
   
<property>  
  <name>fs.viewfs.mounttable.nsX.link./view\_ns1</name>  
  <value>hdfs://ns1</value>  
  <description>Abasefor other temporary directories.</description>  
 </property>

<property>  
  <name>fs.viewfs.mounttable.nsX.link./view\_ns2</name>  
  <value>hdfs://ns2</value>  
  <description>Abasefor other temporary directories.</description>  
 </property>

**12）同步core-site.xml文件**  
\[hadoop@master hadoop\]$ scp core-site.xml slave1:/opt/hadoop/etc/hadoop/  
\[hadoop@master hadoop\]$ scp core-site.xml slave2:/opt/hadoop/etc/hadoop/     
\[hadoop@master hadoop\]$ scp core-site.xml hadoop04:/opt/hadoop/etc/hadoop/

**13）再次启动resourcemanager**  
\[hadoop@master hadoop\]$ yarn-daemon.sh start resourcemanager

**14）检查rm1的状态**  
\[hadoop@master hadoop\]$ yarn rmadmin -getServiceState rm1  
active  
\[hadoop@master hadoop\]$

**15）Hadoop04节点启动resourcemanager**  
\[hadoop@hadoop04 ~\]$ yarn-daemon.sh start resourcemanager  
\[hadoop@hadoop04 ~\]$ yarn rmadmin -getServiceState rm2  
standby  
\[hadoop@hadoop04 ~\]$

**16）启动nodemanager**  
\[hadoop@master hadoop\]$ yarn-daemons.sh start nodemanager

**自此 Hadoop HA + Federation + YARN HA配置实现完成**  
\---------------------   
作者：forever19870418   
来源：CSDN   
原文：https://blog.csdn.net/forever19870418/article/details/67632682   
版权声明：本文为博主原创文章，转载请附上博文链接！

[20170328104414639]: /images/20220126/508563fe9e58430d9f59a6ed8eff51e6.png