hadoop部署参考

灰太狼 2022-06-15 00:58 155阅读 0赞

## **一、概要** ##

由于工作需要，最近一段时间开始接触学习hadoop相关的东西，目前公司的实时任务和离线任务都跑在一个hadoop集群，离线任务的特点就是每天定时跑，任务跑完了资源就空闲了，为了合理的利用资源，我们打算在搭一个集群用于跑离线任务，计算节点和储存节点分离，计算节点结合aws的Auto Scaling（自动扩容、缩容服务）以及竞价实例，动态调整，在跑任务的时候拉起一批实例，任务跑完就自动释放掉服务器，本文记录下hadoop集群的搭建过程，方便自己日后查看，也希望能帮到初学者，本文所有软件都是通过yum安装，大家也可以下载相应的二进制文件进行安装，使用哪种方式安装，从属个人习惯。

## **二、环境** ##

**1、角色介绍**

10.10.103.246 namenode   zkfc    journalNode  QuorumaPeerMain  datanode   resourcemanager    nodemanager    WebAppProxyServer  JobHistoryServer

10.10.103.144 namenode   zkfc    journalNode  QuorumaPeerMain  datanode   resourcemanager    nodemanager    WebAppProxyServer

10.10.103.62             zkfc    journalNode  QuorumaPeerMain  datanode                      nodemanager

10.10.20.130 client

**2、基础环境说明**

a、系统版本

我们用的是aws的ec2，用的aws自己定制过的系统，不过和redhat基本相同，内核版本：4.9.20-10.30.amzn1.x86\_64

b、java版本

java version “1.8.0\_121”

c、hadoop版本

hadoop-2.6.0

d、cdh版本

cdh5.11.0

e、关于主机名，因为我这里用的aws的ec2，默认已有主机名，并且内网可以解析，故就不单独做主机名的配置了，如果你的主机名内网不能解析，请一定要配置主机名，集群内部通讯很多组件使用的是主机名

## **三、配置部署** ##

1、设置yum源

vim /etc/yum.repos.d/cloudera.repo

\[cloudera-cdh5-11-0\]

\# Packages for Cloudera’s Distribution for Hadoop, Version 5.11.0, on RedHat or CentOS 6 x86\_64

name=Cloudera’s Distribution for Hadoop, Version 5.11.0

baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86\_64/cdh/5.11.0/

gpgkey=http://archive.cloudera.com/cdh5/redhat/6/x86\_64/cdh/RPM-GPG-KEY-cloudera

gpgcheck=1

\[cloudera-gplextras5b2\]

\# Packages for Cloudera’s GPLExtras, Version 5.11.0, on RedHat or CentOS 6 x86\_64

name=Cloudera’s GPLExtras, Version 5.11.0

baseurl=http://archive.cloudera.com/gplextras5/redhat/6/x86\_64/gplextras/5.11.0/

gpgkey=http://archive.cloudera.com/gplextras5/redhat/6/x86\_64/gplextras/RPM-GPG-KEY-cloudera

gpgcheck=1

PS：我这里安装的5.11.0，如果想安装低版本或者高版本，根据自己的需求修改版本号即可

2、安装配置zookeeper集群

yum -y  install zookeeper zookeeper-server

vi /etc/zookeeper/conf/zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data/zookeeper

clientPort=2181

maxClientCnxns=0

server.1=10.10.103.144:2888:3888

server.2=10.10.103.226:2888:3888

server.3=10.10.103.62:2888:3888

autopurge.snapRetainCount=3

autopurge.purgeInterval=1

mkdir /data/zookeeper              \#创建datadir目录

/etc/init.d/zookeeper-server init  \#所有节点先初始化

echo 1 > /data/zookeeper/myid      \#10.10.103.144上操作

echo 2 > /data/zookeeper/myid      \#10.10.103.226上操作

echo 3 > /data/zookeeper/myid      \#10.10.103.62上操作

/etc/init.d/zookeeper-server       \#启动服务

/usr/lib/zookeeper/bin/zkServer.sh status  \#查看所有节点状态，其中只有一个节点是Mode: leader就正常 了

3、安装

a、10.10.103.246和10.10.103.144安装

yum -y install hadoop hadoop-client hadoop-hdfs hadoop-hdfs-namenode hadoop-hdfs-zkfc hadoop-hdfs-journalnode hadoop-hdfs-datanode hadoop-mapreduce-historyserver hadoop-yarn-nodemanager hadoop-yarn-proxyserver  hadoop-yarn hadoop-mapreduce hadoop-yarn-resourcemanager hadoop-lzo\* impala-lzo

b、10.10.103.62上安装

yum -y install hadoop hadoop-client hadoop-hdfs hadoop-hdfs-journalnode hadoop-hdfs-datanode  hadoop-lzo\* impala-lzo hadoop-yarn hadoop-mapreduce hadoop-yarn-nodemanager

> **PS：**
> 
> 1、一般小公司，计算节点(ResourceManager)和储存节点(NameNode)的主节点部署在两台服务器上做HA，计算节点（NodeManager）和储存节点（DataNode）部署在多台服务器上，每台服务器上都启动NodeManager和DataNode服务。
> 
> 2、如果大集群，可能需要计算资源和储存资源分离，集群的各个角色都有服务器单独部署，个人建议划分如下:
> 
> a、储存节点
> 
> NameNode：
> 
> 需要安装hadoop hadoop-client hadoop-hdfs hadoop-hdfs-namenode hadoop-hdfs-zkfc hadoop-lzo\* impala-lzo
> 
> DataNode：
> 
> 需要安装hadoop hadoop-client hadoop-hdfs hadoop-hdfs-datanode  hadoop-lzo\* impala-lzo
> 
> QJM集群：
> 
> 需要安装hadoop hadoop-hdfs  hadoop-hdfs-journalnode zookeeper zookeeper-server
> 
> b、计算节点
> 
> ResourceManager：
> 
> 需要安装hadoop hadoop-client hadoop-yarn hadoop-mapreduce hadoop-yarn-resourcemanager
> 
> WebAppProxyServer：
> 
> 需要安装 hadoop hadoop-yarn hadoop-mapreduce hadoop-yarn-proxyserver
> 
> JobHistoryServer：
> 
> 需要安装 hadoop hadoop-yarn hadoop-mapreduce hadoop-mapreduce-historyserver
> 
> NodeManager：
> 
> 需要安装hadoop hadoop-client hadoop-yarn hadoop-mapreduce hadoop-yarn-nodemanager

4、配置

a、创建目录并设置权限

mkdir -p /data/hadoop/dfs/nn             \#datanode上操作

chown hdfs:hdfs /data/hadoop/dfs/nn/ -R  \#datanode上操作

mkdir -p /data/hadoop/dfs/dn             \#namenode上操作

chown hdfs:hdfs /data/hadoop/dfs/dn/ -R  \#namenode上操作

mkdir -p /data/hadoop/dfs/jn             \#journalnode上操作

chown hdfs:hdfs /data/hadoop/dfs/jn/ -R  \#journalnode上操作

mkdir /data/hadoop/yarn -p               \#nodemanager上操作

chown yarn:yarn  /data/hadoop/yarn  -R   \#nodemanager上操作

b、撰写配置文件

5、服务启动

a、启动journalnode(三台服务器上都启动)

/etc/init.d/hadoop-hdfs-journalnode start

b、格式化namenode(在其中一台namenode10.10.103.246上操作)

sudo -u hdfs hadoop namenode -format

c、初始化zk中HA的状态(在其中一台namenode10.10.103.246上操作)

sudo -u hdfs hdfs zkfc -formatZK

d、初始化共享Edits文件(在其中一台namenode10.10.103.246上操作)

sudo -u hdfs hdfs namenode -initializeSharedEdits

e、启动10.10.103.246上namenode

/etc/init.d/hadoop-hdfs-namenode start

f、同步源数据并启动10.10.103.144上namenode

sudo -u hdfs hdfs namenode -bootstrapStandby

/etc/init.d/hadoop-hdfs-namenode start

g、在两台namenode上启动zkfc

/etc/init.d/hadoop-hdfs-zkfc start

h、启动datanode(所有机器上操作)

/etc/init.d/hadoop-hdfs-journalnode start

i、在10.10.103.246上启动WebAppProxyServer、JobHistoryServer、httpfs

/etc/init.d/hadoop-yarn-proxyserver start

/etc/init.d/hadoop-mapreduce-historyserver start

/etc/init.d/hadoop-httpfs start

j、在所有机器上启动nodemanager

/etc/init.d/hadoop-yarn-nodemanager restart

## **四、功能验证** ##

1、hadoop功能

a、查看hdfs根目录

\[root@ip-10-10-103-246 ~\]\# hadoop fs -ls /

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Found 3 items

drwxr-xr-x   – hdfs   hdfs          0 2017-05-11 11:40 /tmp

drwxrwx—   – mapred hdfs          0 2017-05-11 11:28 /user

drwxr-xr-x   – yarn   hdfs          0 2017-05-11 11:28 /var

b、上传一个文件到根目录

\[root@ip-10-10-103-246 ~\]\# hadoop fs -put /tmp/test.txt  /

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

\[root@ip-10-10-103-246 ~\]\# hadoop fs -ls /

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Found 4 items

\-rw-r–r–   2 root   hdfs         22 2017-05-11 15:47 /test.txt

drwxr-xr-x   – hdfs   hdfs          0 2017-05-11 11:40 /tmp

drwxrwx—   – mapred hdfs          0 2017-05-11 11:28 /user

drwxr-xr-x   – yarn   hdfs          0 2017-05-11 11:28 /var

c、直接删除文件不放回收站

\[root@ip-10-10-103-246 ~\]\# hadoop fs -rm -skipTrash /test.txt

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Deleted /test.txt

d、跑一个wordcount用例

\[root@ip-10-10-103-246 ~\]\# hadoop fs -put /tmp/test.txt /user/hdfs/rand/

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

\[root@ip-10-10-103-246 conf\]\# sudo -u hdfs  hadoop  jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.11.0.jar wordcount /user/hdfs/rand/ /tmp

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

17/05/11 11:40:08 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to 10.10.103.246

17/05/11 11:40:09 INFO input.FileInputFormat: Total input paths to process : 1

17/05/11 11:40:09 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library

17/05/11 11:40:09 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library \[hadoop-lzo rev 674c65bbf0f779edc3e00a00c953b121f1988fe1\]

17/05/11 11:40:09 INFO mapreduce.JobSubmitter: number of splits:1

17/05/11 11:40:09 INFO mapreduce.JobSubmitter: Submitting tokens for job: job\_1494472050574\_0003

17/05/11 11:40:09 INFO impl.YarnClientImpl: Submitted application application\_1494472050574\_0003

17/05/11 11:40:09 INFO mapreduce.Job: The url to track the job: http://10.10.103.246:8100/proxy/application\_1494472050574\_0003/

17/05/11 11:40:09 INFO mapreduce.Job: Running job: job\_1494472050574\_0003

17/05/11 11:40:15 INFO mapreduce.Job: Job job\_1494472050574\_0003 running in uber mode : false

17/05/11 11:40:15 INFO mapreduce.Job:  map 0% reduce 0%

17/05/11 11:40:20 INFO mapreduce.Job:  map 100% reduce 0%

17/05/11 11:40:25 INFO mapreduce.Job:  map 100% reduce 100%

17/05/11 11:40:25 INFO mapreduce.Job: Job job\_1494472050574\_0003 completed successfully

17/05/11 11:40:25 INFO mapreduce.Job: Counters: 53

File System Counters

FILE: Number of bytes read=1897

FILE: Number of bytes written=262703

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=6431

HDFS: Number of bytes written=6219

HDFS: Number of read operations=6

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Launched reduce tasks=1

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=2592

Total time spent by all reduces in occupied slots (ms)=5360

Total time spent by all map tasks (ms)=2592

Total time spent by all reduce tasks (ms)=2680

Total vcore-milliseconds taken by all map tasks=2592

Total vcore-milliseconds taken by all reduce tasks=2680

Total megabyte-milliseconds taken by all map tasks=3981312

Total megabyte-milliseconds taken by all reduce tasks=8232960

Map-Reduce Framework

Map input records=102

Map output records=96

Map output bytes=6586

Map output materialized bytes=1893

Input split bytes=110

Combine input records=96

Combine output records=82

Reduce input groups=82

Reduce shuffle bytes=1893

Reduce input records=82

Reduce output records=82

Spilled Records=164

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=1

GC time elapsed (ms)=120

CPU time spent (ms)=1570

Physical memory (bytes) snapshot=501379072

Virtual memory (bytes) snapshot=7842639872

Total committed heap usage (bytes)=525860864

Peak Map Physical memory (bytes)=300183552

Peak Map Virtual memory (bytes)=3244224512

Peak Reduce Physical memory (bytes)=201195520

Peak Reduce Virtual memory (bytes)=4598415360

Shuffle Errors

BAD\_ID=0

CONNECTION=0

IO\_ERROR=0

WRONG\_LENGTH=0

WRONG\_MAP=0

WRONG\_REDUCE=0

File Input Format Counters

Bytes Read=6321

File Output Format Counters

Bytes Written=6219

\[root@ip-10-10-103-246 conf\]\#

2、namenode高可用验证

查看http://10.10.103.246:50070

![Hadoop集群部署实战cdh发行版][Hadoop_cdh]

查看http://10.10.103.144:50070

![Hadoop集群部署实战cdh发行版][Hadoop_cdh 1]

停掉10.10.103.246节点的namenode进程，查看10.10.103.144节点是否会提升为active节点

![Hadoop集群部署实战cdh发行版][Hadoop_cdh 2]

3、resourcemanager高可用验证

查看http://10.10.103.246:8088

![Hadoop集群部署实战cdh发行版][Hadoop_cdh 3]

查看http://10.10.103.144:8088

![Hadoop集群部署实战cdh发行版][Hadoop_cdh 4]

在浏览器输入http://10.10.103.144:8088，会跳转到 [http://ip-10-10-103-246.ec2.internal:8088/][http_ip-10-10-103-246.ec2.internal_8088]，ip-10-10-103-246.ec2.internal是10.10.103.246的主机名，说明resourcemanager高可用配置ok，停掉10.10.103.144的

resourcemanager进程，在浏览器输入http://10.10.103.144:8088，就不会在跳转了。

## **五、总结** ##

1、hadoop集群能成本部署完成，这才是开始，后期的维护，业务方问题的解决这些经验需要一点一点积累，多出差多折腾总是好的。

2、对应上面部署的集群后期需要扩容，直接把10.10.103.62这台机器做个镜像，用镜像启动服务器即可，服务会自动启动并且加入到集群

3、云上hadoop集群的成本优化，这里只针对aws而言

a、冷数据存在在s3上，hdfs可以直接支持s3，在hdfs-site.xml里面添加s3的key参数(fs.s3n.awsAccessKeyId和fs.s3n.awsSecretAccessKey)即可，需要注意的是程序上传、下载的逻辑需要多加几个重试机制，s3有时候不稳定会导致上传或者下载不成功

b、使用Auto Scaling服务结合竞价实例，配置扩展策略，比如当cpu大于50%的时候就扩容5台服务器，当cpu小于10%的时候就缩容5台服务器，当然你可以配置更多阶梯级的扩容、缩容策略，Auto Scaling还有一个计划任务的功能，你可以向设置crontab一样设置，让Auto Scaling帮你扩容、缩容服务器。

[Hadoop_cdh]: http://image.codes51.com/Article/image/20170512/20170512002010_5625.png
[Hadoop_cdh 1]: http://image.codes51.com/Article/image/20170512/20170512002010_7031.png
[Hadoop_cdh 2]: http://image.codes51.com/Article/image/20170512/20170512002011_1718.png
[Hadoop_cdh 3]: http://image.codes51.com/Article/image/20170512/20170512002011_3281.png
[Hadoop_cdh 4]: http://image.codes51.com/Article/image/20170512/20170512002011_5468.png
[http_ip-10-10-103-246.ec2.internal_8088]: http://ip-10-10-103-246.ec2.internal:8088/