Hadoop测试搭建环境

太过爱你忘了你带给我的痛 2022-04-08 09:45 364阅读 0赞

[ 安装java][_java]

Hadoop是使用JAVA写的，所以需要先安装JAVA环境。

本次安装的是hadoop-2.7.0，需要JDK 7以上版本。

yum install java-1.7.0-openjdk
    yum install java-1.7.0-openjdk-devel

安装后确认

java –version

Hadoop安装

从官网下载Hadoop最新版2.7.0

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.0/hadoop-2.7.0.tar.gz

将hadoop解压到/usr/local/下

cd /usr/local/
    tar zxvf /root/hadoop-2.7.0.tar.gz

[设置环境变量][Link 1]

# vi /etc/profile
    export PATH=/usr/local/hadoop-2.7.0/bin:$PATH
    export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64
    
    source /etc/profile

进入Hadoop文件夹下面设置jdk环境变量

cd hadoop-2.7.0/
    vi etc/hadoop/hadoop-env.sh
    export JAVA_HOME=替换位置
    /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.3.el7.x86_64

单机测试节点  
standalone mode（本地单独模式）

进入Hadoop的安装目录

mkdir input
     cp etc/hadoop/*.xml input
    
    hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep input output 'dfs[a-z.]+'
     cat output/*

![这里写图片描述][868522-20151230154637151-747029866.png]

pseudo-distributed mode（伪分布模式）

这种模式是，1个节点上运行，HDFS daemon的 NameNode 和 DataNode、YARN daemon的 ResourceManger 和 NodeManager，分别启动单独的java进程，这个是不是模式比较适合做个人测试

还是在Hadoop的目录文件夹下执行以下命令

vi etc/hadoop/core-site.xml
     <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
    </configuration>
    vi etc/hadoop/hdfs-site.xml
    <configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

[ 设定本机的无密码ssh登陆][_ssh]

ssh-keygen -t rsa
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

格式化文件系统

hdfs namenode -format

启动名称节点和数据节点后台进程

sbin/start-dfs.sh

![这里写图片描述][868522-20151230155950651-1494084439.png]  
在localhost启动一个1个NameNode和1个DataNode，在0.0.0.0启动第二个NameNode

用jps命令确认下  
![这里写图片描述][868522-20151230160053307-321146360.png]

访问NameNode的web页面  
![这里写图片描述][868522-20151230160142807-1000295571.png]

创建HDFS

hdfs dfs -mkdir /user
     hdfs dfs -mkdir /user/test

拷贝input文件到HDFS目录下

hdfs dfs -put etc/hadoop /user/test/input

确认，查看

hadoop fs -ls /user/test/input

执行Hadoop job

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep /user/test/input output 'dfs[a-z.]+'

确认执行结果

hdfs dfs -cat output/*

![这里写图片描述][868522-20151230160746089-740880709.png]

或者从HDFS拷贝到本地查看

bin/hdfs dfs -get output output
     cat output/*

停止daemon

sbin/stop-dfs.sh

执行YARN job

修改设定文件

cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
    vi etc/hadoop/mapred-site.xml
    <configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>
     vi etc/hadoop/yarn-site.xml
    <configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>

启动ResourceManger和NodeManager后台进程

sbin/start-yarn.sh

![这里写图片描述][868522-20151230161420354-522989004.png]

用jps确认  
![这里写图片描述][868522-20151230161441604-292381601.png]

访问ResourceManger的web页面

[http://localhost:8088/][http_localhost_8088]  
![这里写图片描述][868522-20151230161511339-1601485235.png]

执行hadoop job

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar grep /user/test/input output 'dfs[a-z.]+'

确认执行结果

hdfs dfs -cat output/*

停止daemon

sbin/stop-yarn.sh

出现警告

java.io.IOException: java.net.ConnectException: Call From test166/10.86.255.166 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused;

原因是没有启动job history server  
启动jobhistory daemon

sbin/mr-jobhistory-daemon.sh start historyserver

jps确认  
![这里写图片描述][868522-20151230162529073-874732914.png]

访问Job History Server的web页面

[http://localhost:19888/][http_localhost_19888]  
![这里写图片描述][868522-20151231114335307-1037127755.png]

都搭建完事之后正常使用的命令是

[更多命令][Link 2]

sbin/start-all.sh sbin/stop-all.sh

[_java]: http://blog.csdn.net/xupeng874395012/article/details/52447484
[Link 1]: http://blog.csdn.net/xupeng874395012/article/details/65443539
[868522-20151230154637151-747029866.png]: /images/20220407/504dac55bab04c1ab89afadcd0757507.png
[_ssh]: http://blog.csdn.net/xupeng874395012/article/details/52447253
[868522-20151230155950651-1494084439.png]: /images/20220407/255f6225daa041208dadcc480db6f4ca.png
[868522-20151230160053307-321146360.png]: /images/20220407/fcecfa564ee04fada454e613a3f4c362.png
[868522-20151230160142807-1000295571.png]: /images/20220407/ea659ea345054b9e9b8e2722a7018744.png
[868522-20151230160746089-740880709.png]: /images/20220407/dafaedcee978416992026ead296cec27.png
[868522-20151230161420354-522989004.png]: /images/20220407/a073b1cada92478287c56b53635aac24.png
[868522-20151230161441604-292381601.png]: /images/20220407/986ac77fccae402792377ce638230382.png
[http_localhost_8088]: http://localhost:8088/
[868522-20151230161511339-1601485235.png]: /images/20220407/e7834d6552044d95bb9039af97807b21.png
[868522-20151230162529073-874732914.png]: /images/20220407/3606154279224337920bdd185053d206.png
[http_localhost_19888]: http://localhost:19888/
[868522-20151231114335307-1037127755.png]: /images/20220407/f4d5b1fdd7f0448d9a5b6e45319a5203.png
[Link 2]: http://blog.csdn.net/xupeng874395012/article/details/53159449