【Hadoop】YARN伪分布式部署和MapReduce案例

女爷i 2022-05-24 03:25 159阅读 0赞

**前言：**  
前几篇文章 我们介绍了HDFS组件的配置及启动，Yarn是Hadoop集群的资源与作业调度平台，下面介绍下Yarn的伪分布部署及MapReduce简单使用。

##### 1.配置文件及启动 #####

[hadoop@hadoop000 hadoop]$ pwd
    /opt/software/hadoop-2.8.1/etc/hadoop
    [hadoop@hadoop000 hadoop]$ cp mapred-site.xml.template mapred-site.xml
    [hadoop@hadoop000 hadoop]$ vi mapred-site.xml
    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>
    [hadoop@hadoop000 hadoop]$ vi yarn-site.xml
    <configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>
    
    [hadoop@hadoop000 hadoop]$ cd /opt/software/hadoop-2.8.1
    [hadoop@hadoop000 hadoop-2.8.1]$ sbin/start-yarn.sh
    starting yarn daemons
    starting resourcemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-hadoop000.out
    192.168.6.217: starting nodemanager, logging to /opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-hadoop000.out
    [hadoop@hadoop000 hadoop-2.8.1]$ jps
    10512 ResourceManager --Yarn的进程
    10004 SecondaryNameNode
    9655 NameNode
    10936 Jps
    9806 DataNode
    10623 NodeManager --Yarn的进程
    #Yarn的两个进程:
       ResourceManager （资源管理）
       NodeManager     （节点管理）
    #可参考官网：http://hadoop.apache.org/docs/r2.8.4/hadoop-project-dist/hadoop-common/SingleCluster.html

##### 2.Yarn网站测试 #####

![http://192.168.6.217:8088/cluster][http_192.168.6.217_8088_cluster]

##### 3.MapReduce Job测试 #####

[hadoop@hadoop000 hadoop-2.8.1]$ pwd
    /opt/software/hadoop-2.8.1
    [hadoop@hadoop000 hadoop-2.8.1]$ find ./ -name "*example*"
    ./lib/native/examples
    ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.1-test-sources.jar
    ./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.1-sources.jar
    ./share/hadoop/mapreduce/lib-examples
    ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar
    ./share/doc/hadoop/hadoop-mapreduce-examples
    ./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/webapp/example
    ./share/doc/hadoop/hadoop-auth-examples
    ./share/doc/hadoop/api/org/apache/hadoop/examples
    ./share/doc/hadoop/api/org/apache/hadoop/security/authentication/examples
    ./etc/hadoop/ssl-server.xml.example
    ./etc/hadoop/ssl-client.xml.example
    [hadoop@hadoop000 hadoop-2.8.1]$ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 5 10

##### 4.结果解读 #####

![运行结果][70]  
![网页端显示][70 1]

[http_192.168.6.217_8088_cluster]: /images/20220524/495587bf9ebd46438b740760c7a0ef8b.png
[70]: /images/20220524/2cdd5f9359ad46e889af747c92a305e9.png
[70 1]: /images/20220524/1dfe4728cf794d3d82611e9dc96e2fde.png