KiwenLau 基于Docker搭建Hadoop集群之升级版

古城微笑少年丶 2022-11-27 00:38 9阅读 0赞

转载：[https://kiwenlau.com/2016/06/12/160612-hadoop-cluster-docker-update/][https_kiwenlau.com_2016_06_12_160612-hadoop-cluster-docker-update]

注注注注注注 下面的比较全

基于Docker构建的Hadoop开发测试环境，包含Hadoop，Hive，HBase，Spark
    
    https://github.com/ruoyu-chen/hadoop-docker/tree/master

**摘要:** [kiwenlau/hadoop-cluster-docker][kiwenlau_hadoop-cluster-docker]是去年参加[Docker巨好玩][Docker]比赛开发的，得了[二等奖][Link 1]并赢了一块苹果手表，目前这个项目已经在GitHub上获得了236个Star，DockerHub的镜像下载次数2000+。总之，项目还算很受欢迎吧，这篇博客将介绍项目的升级版。

*  作者: [KiwenLau][]
 *  日期: [2016-06-12][]

## 一. 项目介绍 ##

将[Hadoop][]打包到[Docker][Docker 1]镜像中，就可以快速地在单个机器上搭建Hadoop集群，这样可以方便新手测试和学习。

如下图所示，Hadoop的master和slave分别运行在不同的Docker容器中，其中hadoop-master容器中运行NameNode和ResourceManager，hadoop-slave容器中运行DataNode和NodeManager。NameNode和DataNode是Hadoop分布式文件系统HDFS的组件，负责储存输入以及输出数据，而ResourceManager和NodeManager是Hadoop集群资源管理系统YARN的组件，负责CPU和内存资源的调度。

![format_png][]

之前的版本使用serf/dnsmasq为Hadoop集群提供DNS服务，由于Docker网络功能更新，现在并不需要了。更新的版本中，使用以下命令为Hadoop集群创建单独的网络:

sudo docker network create --driver=bridge hadoop

然后在运行Hadoop容器时，使用”–net=hadoop”选项，这时所有容器将运行在hadoop网络中，它们可以通过容器名称进行通信。

**项目更新要点：**

*  去除serf/dnsmasq
 *  合并Master和Slave镜像
 *  使用[kiwenlau/compile-hadoop][kiwenlau_compile-hadoop]项目编译的Hadoo进行安装
 *  优化Hadoop配置

## 二. 3节点Hadoop集群搭建步骤 ##

**1. 下载Docker镜像**

sudo docker pull kiwenlau/hadoop:1.0

**2. 下载GitHub仓库**

git clone https://github.com/kiwenlau/hadoop-cluster-docker

**3. 创建Hadoop网络**

sudo docker network create --driver=bridge hadoop

**4. 运行Docker容器**

cd hadoop-cluster-docker
    ./start-container.sh

**运行结果**

start hadoop-master container...
    start hadoop-slave1 container...
    start hadoop-slave2 container...
    root@hadoop-master:~#

*  启动了3个容器，1个master, 2个slave
 *  运行后就进入了hadoop-master容器的/root目录

**5. 启动hadoop**

*  直接运行 不要sudo

./start-hadoop.sh

*  执行结果

Starting namenodes on [hadoop-master]
    hadoop-master: Warning: Permanently added 'hadoop-master,172.18.0.2' (ECDSA) to the list of known hosts.
    hadoop-master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoop-master.out
    hadoop-slave2: Warning: Permanently added 'hadoop-slave2,172.18.0.4' (ECDSA) to the list of known hosts.
    hadoop-slave1: Warning: Permanently added 'hadoop-slave1,172.18.0.3' (ECDSA) to the list of known hosts.
    hadoop-slave2: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-slave2.out
    hadoop-slave1: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoop-slave1.out
    Starting secondary namenodes [0.0.0.0]
    0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
    0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-root-secondarynamenode-hadoop-master.out
    
    
    starting yarn daemons
    starting resourcemanager, logging to /usr/local/hadoop/logs/yarn--resourcemanager-hadoop-master.out
    hadoop-slave1: Warning: Permanently added 'hadoop-slave1,172.18.0.3' (ECDSA) to the list of known hosts.
    hadoop-slave2: Warning: Permanently added 'hadoop-slave2,172.18.0.4' (ECDSA) to the list of known hosts.
    hadoop-slave1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-slave1.out
    hadoop-slave2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-hadoop-slave2.out

**6. 运行wordcount**

./run-wordcount.sh

**运行结果**

mkdir: cannot create directory 'input': File exists
    20/08/12 08:38:30 INFO client.RMProxy: Connecting to ResourceManager at hadoop-master/172.18.0.2:8032
    20/08/12 08:38:30 INFO input.FileInputFormat: Total input paths to process : 2
    20/08/12 08:38:30 INFO mapreduce.JobSubmitter: number of splits:2
    20/08/12 08:38:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1597220981365_0001
    20/08/12 08:38:31 INFO impl.YarnClientImpl: Submitted application application_1597220981365_0001
    20/08/12 08:38:31 INFO mapreduce.Job: The url to track the job: http://hadoop-master:8088/proxy/application_1597220981365_0001/
    20/08/12 08:38:31 INFO mapreduce.Job: Running job: job_1597220981365_0001
    20/08/12 08:38:39 INFO mapreduce.Job: Job job_1597220981365_0001 running in uber mode : false
    20/08/12 08:38:39 INFO mapreduce.Job:  map 0% reduce 0%
    20/08/12 08:38:45 INFO mapreduce.Job:  map 100% reduce 0%
    20/08/12 08:38:50 INFO mapreduce.Job:  map 100% reduce 100%
    20/08/12 08:38:51 INFO mapreduce.Job: Job job_1597220981365_0001 completed successfully
    20/08/12 08:38:51 INFO mapreduce.Job: Counters: 49
            File System Counters
                    FILE: Number of bytes read=56
                    FILE: Number of bytes written=352398
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=258
                    HDFS: Number of bytes written=26
                    HDFS: Number of read operations=9
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=2
            Job Counters 
                    Launched map tasks=2
                    Launched reduce tasks=1
                    Data-local map tasks=2
                    Total time spent by all maps in occupied slots (ms)=7567
                    Total time spent by all reduces in occupied slots (ms)=2727
                    Total time spent by all map tasks (ms)=7567
                    Total time spent by all reduce tasks (ms)=2727
                    Total vcore-milliseconds taken by all map tasks=7567
                    Total vcore-milliseconds taken by all reduce tasks=2727
                    Total megabyte-milliseconds taken by all map tasks=7748608
                    Total megabyte-milliseconds taken by all reduce tasks=2792448
            Map-Reduce Framework
                    Map input records=2
                    Map output records=4
                    Map output bytes=42
                    Map output materialized bytes=62
                    Input split bytes=232
                    Combine input records=4
                    Combine output records=4
                    Reduce input groups=3
                    Reduce shuffle bytes=62
                    Reduce input records=4
                    Reduce output records=3
                    Spilled Records=8
                    Shuffled Maps =2
                    Failed Shuffles=0
                    Merged Map outputs=2
                    GC time elapsed (ms)=76
                    CPU time spent (ms)=1380
                    Physical memory (bytes) snapshot=832122880
                    Virtual memory (bytes) snapshot=2679701504
                    Total committed heap usage (bytes)=603979776
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters 
                    Bytes Read=26
            File Output Format Counters 
                    Bytes Written=26
    
    input file1.txt:
    Hello Hadoop
    
    input file2.txt:
    Hello Docker
    
    wordcount output:
    Docker  1
    Hadoop  1
    Hello   2

*  NameNode: [http://192.168.59.1:50070/][http_192.168.59.1_50070]Hadoop网页管理地址:
 *  ResourceManager: [http://192.168.59.1:8088/][http_192.168.59.1_8088]

192.168.59.1为运行容器的主机的IP。

## 三. N节点Hadoop集群搭建步骤 ##

### **1. 准备** ###

*  参考第二部分1~3：下载Docker镜像，下载GitHub仓库，以及创建Hadoop网络

### **2. 重新构建Docker镜像** ###

<table> 
 <tbody> 
  <tr> 
   <td> <pre><code>./resize-cluster.sh 5
</code></pre> </td> 
  </tr> 
 </tbody> 
</table>

*  可以指定任意N(N>1)

### **3. 启动Docker容器** ###

<table> 
 <tbody> 
  <tr> 
   <td> <pre><code>./start-container.sh 5
</code></pre> </td> 
  </tr> 
 </tbody> 
</table>

*  与第2步中的N保持一致。

### **4. 运行Hadoop** ###

*  参考第二部分5~6：启动Hadoop，并运行wordcount。

## 参考 ##

1.  [基于Docker搭建多节点Hadoop集群][Docker_Hadoop]
2.  [How to Install Hadoop on Ubuntu 13.10][]

[https_kiwenlau.com_2016_06_12_160612-hadoop-cluster-docker-update]: https://kiwenlau.com/2016/06/12/160612-hadoop-cluster-docker-update/
[kiwenlau_hadoop-cluster-docker]: https://github.com/kiwenlau/hadoop-cluster-docker
[Docker]: http://www.alauda.cn/2015/04/28/docker-great/
[Link 1]: http://www.alauda.cn/2015/06/11/presentation/
[KiwenLau]: http://kiwenlau.com/
[2016-06-12]: http://kiwenlau.com/2016/06/12/160612-hadoop-cluster-docker-update/
[Hadoop]: http://hadoop.apache.org/
[Docker 1]: https://www.docker.com/
[format_png]: /images/20221124/7b554675bf1345f9b915ad5c22063456.png
[kiwenlau_compile-hadoop]: https://github.com/kiwenlau/compile-hadoop
[http_192.168.59.1_50070]: http://192.168.59.1:50070/
[http_192.168.59.1_8088]: http://192.168.59.1:8088/
[Docker_Hadoop]: http://kiwenlau.com/2015/06/08/150608-hadoop-cluster-docker/
[How to Install Hadoop on Ubuntu 13.10]: https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-on-ubuntu-13-10