hadoop使用docker安装和使用(单节点适合开发环境)

亦凉 2022-11-07 00:56 167阅读 0赞

安装环境

[http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html][http_hadoop.apache.org_docs_stable_hadoop-project-dist_hadoop-common_SingleCluster.html]

[https://hub.docker.com/r/sequenceiq/hadoop-docker/tags?page=1&ordering=last\_updated][https_hub.docker.com_r_sequenceiq_hadoop-docker_tags_page_1_ordering_last_updated]  dockerhub  
centos7.2  
docker 18.06.0-ce(这个安装参考我的另外一篇博客https://blog.csdn.net/qq\_16563637/article/details/81699251)  
首先关闭防火墙

systemctl stop firewalld.service #停止firewall
    systemctl disable firewalld.service #禁止firewall开机启动

下载hadoop镜像

docker pull sequenceiq/hadoop-docker:2.7.0

查看镜像

docker images

运行hadoop

docker run -it -p 50070:50070 -p 9000:9000 -p 8088:8088 -p 8040:8040 -p 8042:8042 -p 49707:49707 -p 50010:50010 -p 50075:50075 -p 50090:50090 --name hp  snceiq/hadoop-docker /etc/bootstrap.sh -bash

测试是否安装成功  
先进入hadoop容器

docker exec -it $\{CONTAINER ID\} /bin/bash  
执行完成docker run 也就是上一步,该步骤可以省略

cd /usr/local/hadoop-2.6.0
    bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output 'dfs[a-z.]+'

如果执行mapreduce程序说明安装成功  
可以打开浏览器查看  
宿主机IP:50070  
docker安装完毕  
为了正常使用还需安装以下东西

vi /etc/profile

在最底部加入下面内容

export HADOOP_HOME="/usr/local/hadoop-2.6.0"
    export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

保存:wq  
使得配置生效

source /etc/profile

查看命令行是否能用

hadoop version

docker命令行安装完毕  
测试jar文件在hadoop中启动  
上传一个jar到宿主机  
我用hadoop-mapreduce-examples-2.6.0.jar(这个可以自己在网上下一个)  
拷贝jar文件到容器  
docker cp 要拷贝的文件路径 容器名：要拷贝到容器里面对应的路径

docker cp /root/hadoop-mapreduce-examples-2.6.0.jar b7d7f88574fb:/usr/local/hadoop-2.6.0

查看是否成功

docker exec -it b7d7f88574fb /bin/bash
    cd /usr/local/hadoop-2.6.0
    ls

上传一个文件到hdfs

vi a.txt

输入下面内容

canglaoshi is mylove
    xiaoze is mylove
    wutenglan is mylove

保存  
hadoop创建文件夹

hadoop fs -mkdir -p /wordcount/input
    hadoop fs -put a.txt /wordcount/input
    hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount /wordcount/input /wordcount/output

查看输出内容

hadoop fs -cat /wordcount/output/part-r-00000

原文链接：https://blog.csdn.net/qq\_16563637/article/details/81702633

[http_hadoop.apache.org_docs_stable_hadoop-project-dist_hadoop-common_SingleCluster.html]: http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
[https_hub.docker.com_r_sequenceiq_hadoop-docker_tags_page_1_ordering_last_updated]: https://hub.docker.com/r/sequenceiq/hadoop-docker/tags?page=1&ordering=last_updated