hadoop2.6.1 完全分布/伪分布部署

素颜马尾好姑娘i 2022-08-09 11:54 219阅读 0赞

**先写完全分布式，在写伪分布 ：**

**hadoop2.6.1 完全分布模式**

**三台虚拟机进行完全分布式部署：**

**master：ip 192.168.0.162  hostname  lin162**

**slaves：  ip 192.168.0.163  hostname  lin163**

**slaves：  ip 192.168.0.164  hostname  lin164**

--------------------

**安装步骤：**

1、安装jdk 配置环境变量

安装过程如下：  [http://blog.csdn.net/linlinv3/article/details/45060705][http_blog.csdn.net_linlinv3_article_details_45060705]

2、配置ssh

若系统没有装配ssh 则使用下面语句安装ssh：

$ sudo apt-get install openssh-server 
    $ ps -e|grep ssh

出现以下语句证明成功安装 2228 ? 00:00:00 ssh-agent 5027 ? 00:00:00 sshd

创建秘钥：

$ ssh-keygen -t rsa

$ cd .ssh
    $ cp id_rsa.pub authorized_keys

三台机器分别执行此命名 ，然后 把每个节点的authorized\_keys组合成一个大文件，然后分别覆盖到每个节点的authorized\_keys

3、配置hosts和hostname

$ sudo vim /etc/hostname

三个电脑分别修改为lin162，lin163 ，lin164 （根据自己情况进行修改）

配置host    把IP地址对应到hostname

$ sudo vim /etc/hosts

三个电脑分别修改：

127.0.0.1 localhost 192.168.0.162 lin162 192.168.0.163 lin163 192.168.0.164 lin164

4、配置hadoop 相关文件

hadoop版本下载地址： [http://hadoop.apache.org/releases.html][http_hadoop.apache.org_releases.html]

选择自己需要的版本进行下载：

需要修改的文件有：

(1) 、hadoop-env.sh 修改以下配置项 (java环境)

export JAVA_HOME=/usr/soft/jdk1.7.0_79

(2) 、core-site.xml

<configuration>
    
           
            	<property>
           
            		<name>fs.defaultFS</name>
           
            		<value>hdfs://lin162:8020</value>  
           
            	</property>
           
            	<property>
           
            		<name>hadoop.tmp.dir</name>
           
            		<value>file:///home/lin/hadoop/hadoop-2.6.1/data/tmp</value>
           
            	</property>
           
            </configuration>

(3)、hdfs-site.xml

<configuration> 	<property> 	  <name>dfs.namenode.name.dir</name> 	  <value>file:///home/lin/hadoop/hadoop-2.6.1/data/nn</value> 	</property> 	<property> 	  <name>dfs.datanode.data.dir</name> 	  <value>file:///home/lin/hadoop/hadoop-2.6.1/data/dn</value> 	</property> 	<property> 		<name>dfs.replication</name> 		<value>2</value> 	</property> </configuration>

(4) 、mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

<name>mapreduce.jobhistory.address</name>

<description>MapReduce JobHistory Server IPC host:port</description>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

<description>MapReduce JobHistory Server Web UI host:port</description>

</property>

</configuration>

(5)、 yarn-site.xml

<configuration>
    
           
            
           
            
    
           
              <property>
    
           
                <name>yarn.nodemanager.aux-services</name>
    
           
                <value>mapreduce_shuffle</value>
    
           
              </property>
    
           
              <property>
    
           
            	<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    
           
            	<value>org.apache.hadoop.mapred.ShuffleHandler</value>
    
           
              </property>
    
           
              <property>
    
           
                <description>The hostname of the RM.</description>
    
           
                <name>yarn.resourcemanager.hostname</name>
    
           
                <value>lin162</value>
    
           
              </property>    
    
           
              
    
           
              <property>
    
           
                <name>yarn.resourcemanager.address</name>
    
           
                <value>${yarn.resourcemanager.hostname}:8032</value>
    
           
              </property>
    
           
            
           
              <property>
    
           
                <name>yarn.resourcemanager.scheduler.address</name>
    
           
                <value>${yarn.resourcemanager.hostname}:8030</value>
    
           
              </property>
    
           
            
           
              <property>
    
           
                <name>yarn.resourcemanager.webapp.address</name>
    
           
                <value>${yarn.resourcemanager.hostname}:8088</value>
    
           
              </property>
    
           
            
           
              <property>
    
           
                <name>yarn.resourcemanager.webapp.https.address</name>
    
           
                <value>${yarn.resourcemanager.hostname}:8090</value>
    
           
              </property>
    
           
            
           
              <property>
    
           
                <name>yarn.resourcemanager.resource-tracker.address</name>
    
           
                <value>${yarn.resourcemanager.hostname}:8031</value>
    
           
              </property>
    
           
            
           
              <property>
    
           
                <name>yarn.resourcemanager.admin.address</name>
    
           
                <value>${yarn.resourcemanager.hostname}:8033</value>
    
           
              </property> </configuration>

(6) 、slaves文件

lin163

lin164

5、配置hadoop环境变量

$ sudo  vim /etc/profile

加入如下内容

export HADOOP_HOME=/home/lin/hadoop/hadoop-2.6.1 export PATH=$PATH:$HADOOP_HOME/bin

$ source /etc/profile

6、复制 hadoop2.6.1 到 lin163 和 lin164

$ scp -r /home/lin/hadoop/hadoop-2.6.1 lin@192.168.0.163:/home/lin/hadoop/hadoop-2.6.1 scp -r /home/lin/hadoop/hadoop-2.6.1 lin@192.168.0.164:/home/lin/hadoop/hadoop-2.6.1

6、在162格式化namenode

$ hdfs namenode -format

若出现

15/10/22 09:35:01 INFO common.Storage: Storage directory /home/lin/hadoop/hadoop-2.6.1/data/nn has been successfully formatted.

证明格式化成功

7、在162启动hadoop

$ hdfs dfs start-all.sh

每台机器 jps 后

lin162 出现

![Center][]

![Image 1][]

lin163 出现

![Center 1][]

![Image 1][]

lin164 出现

![Image 1][]

![Center 2][]

证明启动成功；

Web Interfaces

<table style="border-collapse:collapse; border:1px solid rgb(227,237,245); font-size:14px; line-height:21px; widows:auto; word-wrap:break-word; empty-cells:show; table-layout:auto; width:528px; color:rgb(68,68,68); font-family:Tahoma,'Microsoft Yahei',Simsun"> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word; background-color:rgb(223,227,219)"> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      Daemon 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      Web Interface 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      Notes 
    </div> </td> 
  </tr> 
  <tr style="word-wrap:break-word; background-color:rgb(238,238,238)"> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      NameNode 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word"> 
     <span style="word-wrap:break-word; color:rgb(102,136,170)">http://nn_host:port/</span> 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      Default HTTP port is 50070. 
    </div> </td> 
  </tr> 
  <tr style="word-wrap:break-word; background-color:rgb(221,221,221)"> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      ResourceManager 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word"> 
     <span style="word-wrap:break-word; color:rgb(102,136,170)">http://rm_host:port/</span> 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      Default HTTP port is 8088. 
    </div> </td> 
  </tr> 
  <tr style="word-wrap:break-word; background-color:rgb(238,238,238)"> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      MapReduce JobHistory Server 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word"> 
     <span style="word-wrap:break-word; color:rgb(102,136,170)">http://jhs_host:port/</span> 
    </div> </td> 
   <td style="border-collapse:collapse; border:1px solid rgb(227,237,245); word-wrap:break-word; padding:4px; overflow:hidden"> 
    <div style="word-wrap:break-word">
      Default HTTP port is 19888. 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

通过web 查看系统

**hadoop 2.x伪分布安装  [2.x伪分布安装][2.x]**

启动 history  与 WebAppProxyServer

./mr-jobhistory-daemon.sh start historyserver
    ./yarn-daemon.sh --config $HADOOP_CONF_DIR start proxyserver

[http_blog.csdn.net_linlinv3_article_details_45060705]: http://blog.csdn.net/linlinv3/article/details/45060705
[http_hadoop.apache.org_releases.html]: http://hadoop.apache.org/releases.html
[Center]: /images/20220731/c36c2183ac694dbabe92f37873d45a60.png
[Image 1]: 
[Center 1]: /images/20220731/5daefe1759864292809e2e5cda861bca.png
[Center 2]: /images/20220731/f5937a25f7a14045b1ae76d3f900295a.png
[2.x]: http://blog.csdn.net/linlinv3/article/details/42709589