Hadoop各个配置文件的作用

爱被打了一巴掌 2022-04-13 08:55 418阅读 0赞

### core-site.xml ###

其中 fs.defaultFS 配置的地址就是java代码访问的时候的路径，需要配置在java代码中代码中要用IP:9000不能用localhost

<configuration>
           <property>
             
    
              <name>fs.defaultFS</name>
              <value>hdfs://localhost:9000</value>
           </property>
           <property>
              
    
              <name>io.file.buffer.size</name>
              <value>131072</value>
            </property>
           <property>
             
    
             <name>hadoop.tmp.dir</name>
             <value>file:/usr/local/hadoop-2.4.1/tmp</value>
             <description>Abase for other temporary   directories.</description>
           </property>
            <property>
                
               <name>ipc.maximum.data.length</name>
               <value>134217728</value>
            </property>
    
    </configuration>

### hdfs-site.xml ###

dfs.namenode.secondary.http-address 这个地址就是文件系统浏览器访问地址

主namenode有个hdfs协议的访问地址：[http://192.168.187.128:50070][http_192.168.187.128_50070]

secondNamenode有个http协议的访问地址：[http://192.168.187.128:50071][http_192.168.187.128_50070]

<configuration>
    
           <property>
    
                
                    <name>dfs.namenode.secondary.http-address</name>
                   <value>localhost:50070</value>
           </property>
         <property>
                
    
                 <name>dfs.namenode.name.dir</name>
                 <value>file:/usr/local/hadoop-2.4.1/dfs/name</value>
           </property>
          <property>
                
    
                  <name>dfs.datanode.data.dir</name>
                  <value>file:/usr/local/hadoop-2.4.1/dfs/data</value>
           </property>
           <property>
                
    
                   <name>dfs.replication</name>
                   <value>3</value>
            </property>
            <property>
                
    
                    <name>dfs.webhdfs.enabled</name>
                    <value>true</value>
             </property>
    </configuration>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p4bDY0NjgwMTkyNA_size_16_color_FFFFFF_t_70][]

### yarn-site.xml ###

<property>
                   <name>yarn.nodemanager.aux-services</name>
                   <value>mapreduce_shuffle</value>
            </property>
            <property>                                                                
                   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
            </property>
            <property>
                
                   <name>yarn.resourcemanager.address</name>
                   <value>singlehost:8032</value>
           </property>
           <property>
                
                   <name>yarn.resourcemanager.scheduler.address</name>
                   <value>singlehost:8030</value>
           </property>
           <property>
                
                <name>yarn.resourcemanager.resource-tracker.address</name>
                 <value>singlehost:8031</value>
          </property>
          <property>
                
                  <name>yarn.resourcemanager.admin.address</name>
                   <value>singlehost:8033</value>
           </property>
           <property>
                
                   <name>yarn.resourcemanager.webapp.address</name>
                   <value>singlehost:8088</value>
           </property>

### mapred-site.xml ###

MapReduce内部的一个通讯地址：singlehost:10020

MapReduce外部web访问的地址：singlehost:19888

<configuration>
              <property> 
                
    
                 <name>mapreduce.framework.name</name>
                 <value>yarn</value>
               </property>
              <property>
            
    
                      <name>mapreduce.jobhistory.address</name>
                      <value>singlehost:10020</value>
              </property>
              <property>
    
            
                    <name>mapreduce.jobhistory.webapp.address</name>
                    <value>singleshost:19888</value>
           </property>
    </configuration>

[http_192.168.187.128_50070]: http://192.168.187.128:50070/explorer.html#/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p4bDY0NjgwMTkyNA_size_16_color_FFFFFF_t_70]: /images/20220413/110f6f9dee8b45a394b96fd4b5beeaa3.png