Hadoop实验环境

约定不等于承诺〃 2022-07-14 20:16 143阅读 0赞

# 3台虚拟机 #

•OS：RHEL\_6.X\_X86-64

172.16.34.94 lsfvm01  
172.16.34.95 lsfvm02

172.16.34.96 lsfvm03

# 安装准备 #

•3台机器之间网络联通、主机名正确解析、关闭防火墙  
•配置ssh无密码访问  
•安装jdk

# 配置ssh无密码访问 #

1.登录lsfvm01, 做如下操作：

\[root@lsfvm01 .ssh\]\# cd /root/.ssh

\[root@lsfvm01 .ssh\]\# ssh-keygen -t rsa

\[root@lsfvm01 .ssh\]\# ls id\_rsa id\_rsa.pub

\[root@lsfvm01 .ssh\]\# cat id\_rsa.pub >> authorized\_keys

\[root@lsfvm01 .ssh\]\# ssh-copy-id -i id\_rsa.pub root@lsfvm02

\[root@lsfvm01 .ssh\]\# ssh-copy-id -i id\_rsa.pub root@lsfvm03

\[root@lsfvm01 .ssh\]\# chmod 700 /root/.ssh

\[root@lsfvm01 .ssh\]\# chmod 600 /root/.ssh/authorized\_keys

2. 依次登录lsfvm02, lsfvm03重复步骤1

# 安装jdk #

1. 安装jdk包  
\[root@lsfvm01 ~\]\# rpm -ivh jdk-7u79-linux-x64.rpm  
\[root@lsfvm01 ~\]\# ls /usr/java/jdk1.7.0\_79/

2. 设置java环境变量：  
在/etc/profile末尾加入：  
\[root@lsfvm01 ~\]\# vim /etc/profile

#for Java
    export JAVA_HOME=/usr/java/jdk1.7.0_79
    export PATH=$JAVA_HOME/bin:$PATH
    export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

使添加环境变量立即生效  
\[root@lsfvm01 ~\]\# source /etc/profile  
3. 验证安装  
\[root@lsfvm01 ~\]\# java -version  
java version "1.7.0\_79”  
4. 在lsfvm02和lsfvm03上重复上述工作

# 安装Hadoop (一) #

1. 准备hadoop 安装目录  
\[root@lsfvm01 .ssh\]\# mkdir /usr/hadoop  
\[root@lsfvm01 .ssh\]\# mkdir /usr/hadoop/tmp  
\[root@lsfvm01 .ssh\]\# mkdir /usr/hadoop/hdfs  
\[root@lsfvm01 .ssh\]\# mkdir /usr/hadoop/hdfs/data  
\[root@lsfvm01 .ssh\]\# mkdir /usr/hadoop/hdfs/name  
2. 复制hadoop安装包到/usr/hadoop，并解压  
\[root@lsfvm01 ~\]\# cp hadoop-2.7.1.tar.gz /usr/hadoop/  
\[root@lsfvm01 hadoop\]\# tar zxvf hadoop-2.7.1.tar.gz  
3. 设置环境变量

\[root@lsfvm01 hadoop\]\# vim /etc/profile

#for Hadoop
    export HADOOP_HOME=/usr/hadoop/hadoop-2.7.1
    export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

使环境变量立即生效  
\[root@lsfvm01 hadoop\]\# source /etc/profile

# 安装Hadoop(二) #

4. 修改hadoop配置 需修改7个配置文件

\[root@lsfvm01 hadoop\]\# cd $HADOOP\_HOME/etc/hadoop

hadoop-env.sh yarn-env.sh core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml slaves

1）\[root@lsfvm01 hadoop\]\# vim hadoop-env.sh

#export JAVA_HOME=${JAVA_HOME} 
    export JAVA_HOME=/usr/java/jdk1.7.0_79

2) \[root@lsfvm01 hadoop\]\# vim yarn-env.sh

#export JAVA_HOME=/home/y/libexec/jdk1.6.0/ 
    export JAVA_HOME=/usr/java/jdk1.7.0_79

# 安装Hadoop(三) #

3) \[root@lsfvm01 hadoop\]\# vim core-site.xml

<configuration>
    <property>
    	<name>fs.defaultFS</name>
    	<value>hdfs://172.16.34.94:9000</value>
    	<description>HDFS的URI，文件系统://namenode标识:端口号</description>
    </property>
    <property>
    	<name>hadoop.tmp.dir</name>
    	<value>/usr/hadoop/tmp</value>
    	<description>namenode上本地的hadoop临时文件夹</description>
    </property>
    </configuration>

4) \[root@lsfvm01 hadoop\]\# vim hdfs-site.xml

<configuration>
    <property>
    <name>dfs.name.dir</name>
    <value>/usr/hadoop/hdfs/name</value>
    <description>namenode上存储hdfs名字空间元数据 </description>
    </property>
    <property>
    <name>dfs.data.dir</name>
    <value>/usr/hadoop/hdfs/data</value>
    <description>datanode上数据块的物理存储位置</description>
    </property>
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    <description>副本个数，配置默认是3,应小于datanode机器数量</description>
    </property>
    </configuration>

5) \[root@lsfvm01 hadoop\]\# vim mapred-site.xml

6) \[root@lsfvm01 hadoop\]\# vim yarn-site.xml

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>172.16.34.94:8099</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>172.16.34.94:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>172.16.34.94:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>172.16.34.94:8031</value> </property> </configuration>

7) \[root@lsfvm01 hadoop\]\# vim slaves

lsfvm01 lsfvm02 lsfvm03

# 安装Hadoop(八) #

5. 将Hadoop的安装包复制到其他节点：lsfvm02 lsfvm03  
1) 将lsfvm01上的$HADOOP\_HOME目录下所有内容复制到其他节点相同目录  
2) 复制设置环境变量（Java和Hadoop）  
例如以lsfvm02为例：  
\[root@lsfvm01 hadoop\]\# scp -r /usr/hadoop/ root@lsfvm02:/usr/  
\[root@lsfvm01 hadoop\]\# ssh lsfvm02

\[root@lsfvm02 ~\]\# vim /etc/profile

#for Hadoop
    export HADOOP_HOME=/usr/hadoop/hadoop-2.7.1
    PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
    CLASSPATH=$CLASSPATH:$HADOOP_HOME/share/hadoop/common/hadoop-common-2.7.1.jar:$HADOOP_HOME/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar
    PATH=$PATH:$HOME/bin
    export PATH
    export CLASSPATH

\[root@lsfvm02 ~\]\# source /etc/profile

# 启动并检查hadoop状态 #

1）启动  
\#第一次启动格式化namenode  
\[root@lsfvm01 hadoop-2.7.1\]\# hdfs namenode -format  
\[root@lsfvm01 hadoop-2.7.1\]\# start-dfs.sh  
\[root@lsfvm01 hadoop-2.7.1\]\# start-yarn.sh  
2）检查状态  
1）namenode  
\[root@lsfvm01 hadoop-2.7.1\]\# jps  
17148 ResourceManager  
18147 Jps  
10920 DataNode  
17247 NodeManager  
15807 SecondaryNameNode  
11793 NameNode  
2) datanode  
\[root@lsfvm02 ~\]\# jps  
24008 NodeManager  
29349 Jps  
23721 DataNode  
3) 查看web  
lsfvm01 172.16.34.94  
http://172.16.34.94:50070  
http://172.16.34.94:8099

# 提交测试作业 #

1. 新建一个测试文本文档

\[root@lsfvm01 hadoop\]\# vim /tmp/world.txt

2. 在HDFS创建一个目录，并将测试文档传送到HDFS目录中

\[root@lsfvm01 hadoop\]\# hdfs dfs -mkdir /test

\[root@lsfvm01 hadoop\]\# hdfs dfs -ls /

\[root@lsfvm01 hadoop\]\# hdfs dfs -put /tmp/world.txt /test

\[root@lsfvm01 hadoop\]\# hdfs dfs -ls /test

3.提交作业

\[root@lsfvm01 hadoop\]\#hadoop jar /usr/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount /test /out

4.查看结果 \[root@lsfvm01 hadoop\]\# hdfs dfs -ls /out

Found 2 items