hadoop3 提交任务出错（找不到类）

约定不等于承诺〃 2022-05-09 00:14 721阅读 0赞

# hadoop3 提交任务出错（找不到类） #

hadoop3的配置和hadoop2的配置基本相同，但还是有一定的差距

hadoop3的配置注意事项

1.首先要一堆环境变量（HADOOP\_PREFIX不要配）

export JAVA_HOEM=/usr/java/default
    export HADOOP_HOME=/opt/hadoop
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_COMMON_HOME=$HADOOP_HOME
    export HADOOP_HDFS_HOME=$HADOOP_HOME
    export HADOOP_MAPRED_HOME=$HADOOP_HOME
    export HADOOP_YARN_HOME=$HADOOP_HOME
    export HADOOP_INSTALL=$HADOOP_HOME
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_CONF_DIR=$HADOOP_HOME
    export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
    export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
    export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
    export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
    export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH

2.所有的主机要写在workers文件中

3.mapred-site.xml要在hadoop2的基础上添加

<property>
      <name>yarn.app.mapreduce.am.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.map.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.reduce.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
     <property>
            <name>mapreduce.application.classpath</name>
            <value>
                ${HADOOP_HOME}/etc/hadoop,
                ${HADOOP_HOME}/share/hadoop/common/*,
                ${HADOOP_HOME}/share/hadoop/common/lib/*,
                ${HADOOP_HOME}/share/hadoop/hdfs/*,
                ${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
                ${HADOOP_HOME}/share/hadoop/mapreduce/*,
                ${HADOOP_HOME}/share/hadoop/mapreduce/lib/*,
                ${HADOOP_HOME}/share/hadoop/yarn/*,
                ${HADOOP_HOME}/share/hadoop/yarn/lib/*
            </value>
        </property>

4.在没有添加以上内容前的报错，找不到各种各样的类

hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar pi 1 1
    WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
    Number of Maps  = 1
    Samples per Map = 1
    2018-09-28 14:19:01,238 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Wrote input for Map #0
    Starting Job
    2018-09-28 14:19:02,297 INFO client.RMProxy: Connecting to ResourceManager at standlone/192.168.114.135:8032
    2018-09-28 14:19:02,779 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hdfs/.staging/job_1538169417840_0003
    2018-09-28 14:19:02,874 INFO input.FileInputFormat: Total input files to process : 1
    2018-09-28 14:19:02,918 INFO mapreduce.JobSubmitter: number of splits:1
    2018-09-28 14:19:02,953 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
    2018-09-28 14:19:03,447 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538169417840_0003
    2018-09-28 14:19:03,452 INFO mapreduce.JobSubmitter: Executing with tokens: []
    2018-09-28 14:19:03,642 INFO conf.Configuration: resource-types.xml not found
    2018-09-28 14:19:03,642 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
    2018-09-28 14:19:03,729 INFO impl.YarnClientImpl: Submitted application application_1538169417840_0003
    2018-09-28 14:19:03,770 INFO mapreduce.Job: The url to track the job: http://standlone:8088/proxy/application_1538169417840_0003/
    2018-09-28 14:19:03,771 INFO mapreduce.Job: Running job: job_1538169417840_0003
    2018-09-28 14:19:09,887 INFO mapreduce.Job: Job job_1538169417840_0003 running in uber mode : false
    2018-09-28 14:19:09,888 INFO mapreduce.Job:  map 0% reduce 0%
    2018-09-28 14:19:11,943 INFO mapreduce.Job: Task Id : attempt_1538169417840_0003_m_000000_0, Status : FAILED
    [2018-09-28 14:19:11.432]Exception from container-launch.
    Container id: container_1538169417840_0003_01_000002
    Exit code: 1
    
    [2018-09-28 14:19:11.434]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
    
    
    [2018-09-28 14:19:11.435]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
    
    
    
    2018-09-28 14:19:15,005 INFO mapreduce.Job: Task Id : attempt_1538169417840_0003_m_000000_1, Status : FAILED
    [2018-09-28 14:19:14.181]Exception from container-launch.
    Container id: container_1538169417840_0003_01_000003
    Exit code: 1
    
    [2018-09-28 14:19:14.182]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
    
    
    [2018-09-28 14:19:14.183]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
    
    
    
    2018-09-28 14:19:18,045 INFO mapreduce.Job: Task Id : attempt_1538169417840_0003_m_000000_2, Status : FAILED
    [2018-09-28 14:19:17.200]Exception from container-launch.
    Container id: container_1538169417840_0003_01_000004
    Exit code: 1
    
    [2018-09-28 14:19:17.202]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
    
    
    [2018-09-28 14:19:17.203]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
    Last 4096 bytes of prelaunch.err :
    Last 4096 bytes of stderr :
    Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
    
    
    
    2018-09-28 14:19:23,081 INFO mapreduce.Job:  map 100% reduce 100%
    2018-09-28 14:19:24,118 INFO mapreduce.Job: Job job_1538169417840_0003 failed with state FAILED due to: Task failed task_1538169417840_0003_m_000000
    Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
    
    2018-09-28 14:19:24,207 INFO mapreduce.Job: Counters: 13
    	Job Counters 
    		Failed map tasks=4
    		Killed reduce tasks=1
    		Launched map tasks=4
    		Other local map tasks=3
    		Data-local map tasks=1
    		Total time spent by all maps in occupied slots (ms)=3549
    		Total time spent by all reduces in occupied slots (ms)=0
    		Total time spent by all map tasks (ms)=3549
    		Total vcore-milliseconds taken by all map tasks=3549
    		Total megabyte-milliseconds taken by all map tasks=3634176
    	Map-Reduce Framework
    		CPU time spent (ms)=0
    		Physical memory (bytes) snapshot=0
    		Virtual memory (bytes) snapshot=0
    Job job_1538169417840_0003 failed!