大数据平台CDH搭建

女爷i 2024-04-17 06:01 73阅读 0赞

# 一、概述 #

Cloudera版本（Cloudera’s Distribution Including Apache Hadoop，简称“CDH”），基于Web的用户界面,支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop,简化了大数据平台的安装、使用难度。

# 二、安装部署 #

| 序号 | IP地址 | 主机名 |系统版本|  
| -------- | -------- | -------- |  
| 1 | 172.20.2.222 | cm-server |centos7.3  
| 2 | 172.20.2.203 | hadoop-1 |centos7.3  
| 3 | 172.20.2.204 | hadoop-2 |centos7.3  
| 4 | 172.20.2.205 | hadoop-3 |centos7.3

### 2.2.1 基础环境部署 ###

a.修改主机名配置hosts

systemctl stop firewalld
    hostnamectl set-hostname  cm-server   #更改个主机名
    sed -i 's/SELINUX=enforcing/SELINUX=disable/g' /etc/selinux/config
    setenforce 0
    cat >>/etc/hosts<<EOF    #添加各个节点hosts解析
    172.20.2.222    cm-server
    172.20.2.203      hadoop-1
    172.20.2.204      hadoop-2
    172.20.2.205      hadoop-3
    EOF

b.配置cm-server免密钥登录其他节点

ssh-keygen -t rsa     #在cm-server生成密钥对
    for num in `seq 1 3`;do ssh-copy-id -i /root/.ssh/id_rsa.pub root@hadoop-$num;done

c.在cm-server安装数据库  
在cm-server上安装mariadb，用于后期数据存储

yum install mariadb*
    systemctl start mariadb
    mysql -uroot password "mysqladmin"
    登录数据库后我们采用root登录

e.java环境配置

如果系统有安装java环境卸载干净使用oracle的jdk，此处使用`jdk-7u80-linux-x64.rpm`,在各节点均配置java环境

rpm -ivh jdk-7u80-linux-x64.rpm
    cat >/etc/profile.d/java.sh<<EOF
    export JAVA_HOME=/usr/java/jdk1.8.0_121
    export CLASSPATH=.:\$JAVA_HOME/jre/lib/rt.jar:\$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar 
    export PATH=\$PATH:\$JAVA_HOME/bin
    EOF
    source /etc/profile.d/java.sh

f.配置各节点服务器需求

sysctl -w vm.swappiness=10
    echo "vm.swappiness=10" >>/etc/sysctl.conf
    echo never > /sys/kernel/mm/transparent_hugepage/defrag
    echo never > /sys/kernel/mm/transparent_hugepage/enabled

### 2.2.2 Cloudera Manager安装 ###

a.下载解压相关软件包

mkdir /software && cd /software
    wget -c https://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz 
    wget -c http://archive.cloudera.com/cdh5/parcels/5.14.2/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel
    wget -c http://archive.cloudera.com/cdh5/parcels/5.14.2/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1 -O CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha
    wget -c http://archive.cloudera.com/cdh5/parcels/5.14.2/manifest.json
    wget -c https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.zip
    tar -zxvf cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz -C /opt/   #解压cm包
    unzip mysql-connector-java-5.1.46.zip  #解压java-mysql连接jar包
    cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /opt/cm-5.14.1/share/cmf/lib/    #将jar包复制到cm的lib目录下
    cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar

b.创建用户及初始化数据库

useradd --system --home=/opt/cm-5.14.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm     #在各个节点均创建用户
    vim /opt/cm-5.14.1/etc/cloudera-scm-agent/config.ini将其中的server_host=cm-server     #指向cm-server
    usage: /opt/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh [options] (postgresql|mysql|oracle) database username [password]    #使用选项
    /opt/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql cmdb -h"cm-server" -uroot -pmysqladmin --scm-host cm-server scm scm scm

c.将文件将cm-server修改完成的文件分发到其他各节点

for i in `seq 1 3`;do scp -r /opt/cm-5.14.1 hadoop-$i:/opt/;done

d.创建本地源

mv CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel* manifest.json /opt/cloudera/parcel-repo/

e.启动服务  
在cm-server启动server和agent服务，在其他节点启动agent服务

/opt/cm-5.14.1/etc/init.d/cloudera-scm-server start
    /opt/cm-5.14.1/etc/init.d/cloudera-scm-agent start

### 2.2.2 Cloudera Manager的web界面配置 ###

服务器均启动后，可以浏览器访问cm-server的7180端口,用户名/密码为admin/admin

![format_png][]

image

![format_png 1][]

image

接受协议继续

![format_png 2][]

image

可以选择适用60天

![format_png 3][]

image

提示一些涉及许可证的信息

![format_png 4][]

image

勾选管理的主机继续操作

![format_png 5][]

image

选择CDH-5.14版本

![format_png 6][]

image

parcel安装

![format_png 7][]

image

主机正确性检查

![format_png 8][]

image

群集设置（选择安装的服务）

![format_png 9][]

image

自定义角色分配，选择安装在那个节点上

![format_png 10][]

image

数据库设置  
需要提前创建数据库及授权其他节点可以正常连接

![format_png 11][]

image

![format_png 12][]

image

审核更改

![format_png 13][]

image

集群安装

![format_png 14][]

image

完成安装

![format_png 15][]

image

![format_png 16][]

image

后期可添加服务

![format_png 17][]

image

![format_png 18][]

[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTAxMTY0NWZlODA1ZmVhNzE_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTNlOGE1Y2MzYjVmZjAzNDU_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzkxNy9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LWM5Mjk5YzRhOWFkNTA3ZGQ_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTVjYzY1YmM1OTdkMWJmMGE_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTFkMTViM2E0MmU4YzJkZTQ_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93Lzg5OC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 5]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTkwZmRlOTgxZjAyNzY4MjA_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 6]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTFmYTU0MWI1YzQzODQzNjI_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 7]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTY2ODM5OGE2M2Q0NjJlMDA_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 8]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LWVhOWZiNTY2MDFmYzAwZjA_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 9]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LWU3YjdmMDk2MTQ3ZTdlOWY_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 10]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTE2ZGJkZGQ5YjRkMjAxZjI_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 11]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTk1MDE4YWFhYjlhZTFiODQ_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 12]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTlhMzIwZmNmZDMyNWFjMmQ_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 13]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LWI1NmNlZDk5Y2QwOTg0OTA_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 14]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTliYjc3ZjZkNzRiNDlhODU_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 15]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTczOTU0N2M4YjIwMWNmMWM_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzcwOC9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 16]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LTE5YWJlMTllZTFmZWZkZTE_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93LzEwMDAvZm9ybWF0L3dlYnA?x-oss-process=image/format,png
[format_png 17]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LWU3ZmVlMWI5MmY2ZjU5MGI_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93Lzg3OS9mb3JtYXQvd2VicA?x-oss-process=image/format,png
[format_png 18]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy85NjE4NjY0LWQwNjBmMTg4M2YzNWQ3ZDA_aW1hZ2VNb2dyMi9hdXRvLW9yaWVudC9zdHJpcCU3Q2ltYWdlVmlldzIvMi93Lzg5Mi9mb3JtYXQvd2VicA?x-oss-process=image/format,png