[hadoop]hadoop2.4.0集群安装示例

た入场券 2022-08-27 11:43 127阅读 0赞

软件环境:  
linux:ubuntu1404  
jdk:jdk-6u45-linux-x64.bin  
hadoop:hadoop-2.4.0  
主机名称和IP  
192.168.233.129 master.hadoop master  
192.168.233.130 node1.hadoop node1  
192.168.233.131 node2.hadoop node2  
  
一、设置主机名称，配置hostname文件  
1.修改/etc/hostname文件，master增加所有node的IP，所有node增加master的IP  
二、建立用户群（hadoop）和用户(hadoop)，配置sudo权限

attwice@master:~$ sudo su
    root@master:~# groupadd hadoop 
    root@master:~# useradd -s /bin/bash -d /home/hadoop -m hadoop -g hadoop
    root@master:~# passwd hadoop
    Enter new UNIX password:
    Retype new UNIX password:
    passwd: password updated successfully
    root@master:~# ls /home
    attwice hadoop

配置sudo权限，修改文件/etc/sudoers，增加一行如下：

hadoop ALL=(ALL) ALL

三、配置SSH免密码登入  
配置master无密码登录所有的node，配置所有node无密码登录master  
安装ssh：sudo apt-get install ssh

hadoop@master:~$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
    Created directory '/home/hadoop/.ssh'.
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /home/hadoop/.ssh/id_rsa.
    Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
    The key fingerprint is:
    ec:dd:60:07:0b:f0:ea:a1:8f:d6:84:5e:f6:5c:0a:f0 hadoop@master
    The key's randomart image is:
    +--[ RSA 2048]----+
    | . |
    | o |
    | o . |
    | . o . o |
    | +o S + . |
    | .oEo o.+ |
    | ..=.+.o. . |
    | oo. + |
    | .. . |
    +-----------------+

进入.ssh目录，执行如下命令

[master上]
    cat id_rsa.pub >> authorized_keys
    scp id_rsa.pub hadoop@node1:~/.ssh/master.pub
    scp id_rsa.pub hadoop@node2:~/.ssh/master.pub
    scp hadoop@node1:~/.ssh/id_rsa.pub node1.pub
    scp hadoop@node2:~/.ssh/id_rsa.pub node2.pub    
    cat node1.pub >> authorized_keys
    cat node2.pub >> authorized_keys
    [node上]
    cat id_rsa.pub >> authorized_keys
    cat master.pub >> authorized_keys

用ssh命令测试各自的互通性：ssh localhost或ssh xxx  
四、安装JDK，设置环境变量  
1.将JDK安装到/usr/java下面  
2.设置环境变量，修改/etc/profile文件，增加如下内容：

JAVA_HOME=/usr/java/jdk1.6.0_45
    PATH=$JAVA_HOME/bin:$PATH
    export JAVA_HOME PATH

五、安装hadoop  
1.解压hadoop-2.4.0.tar.gz文件到/home/hadoop/apps/目录下，进入目录/home/hadoop/apps/hadoop-2.4.0/etc/hadoop，修改配置文件  
2.配置hadoop-env.sh（JAVA\_HOME）

export JAVA_HOME=/usr/java/jdk1.6.0_45

3.配置core-site.xml，configuration增加如下内容：

<property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadoop/apps/temp/hadoop-${user.name}</value>
      </property>
      <property>
        <name>fs.default.name</name>
        <value>hdfs://192.168.233.129:9000</value>
      </property>

master主机需创建temp文件夹  
4.配置yarn-site.xml，configuration增加如下内容：

<property>
        <name>yarn.resourcemanager.address</name>
        <value>192.168.233.129:8032</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>192.168.233.129:8030</value>
      </property>
      <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>192.168.233.129:8031</value>
      </property>

5.可选配置mapred-site.xml  
6.配置slaves（仅仅在master主机上，执行完第7步再修改）

node1
    node2

7.将配置好的hadoop复制到各个节点

scp -r hadoop-2.4.0 hadoop@node1:~/apps/
    scp -r hadoop-2.4.0 hadoop@node2:~/apps/

六、测试是否配置成功  
1.格式化及启动

bin/hadoop namenode -format
    sbin/start-dfs.sh
    sbin/start-yarn.sh

2.查看状态（后台和web方式）  
bin/hadoop dfsadmin -report  
http://192.168.233.129:50070/  查看信息  
http://192.168.233.129:8088/  查看resource manager  
七、辅助功能  
1.关闭hadoop

sbin/stop-dfs.sh
    sbin/stop-yarn.sh