CDH5搭建

分手后的思念是犯贱 2022-01-22 13:41 230阅读 0赞

> 本文介绍了CDH5的搭建步骤及遇到的问题和解决方案。

参考自：[https://www.cnblogs.com/zhangleisanshi/p/7575579.html][https_www.cnblogs.com_zhangleisanshi_p_7575579.html]及[https://www.cnblogs.com/pojishou/archive/2017/01/12/6267616.html][https_www.cnblogs.com_pojishou_archive_2017_01_12_6267616.html]

## 准备工作 ##

本案例使用公司现成的三台机器，如需自己搭建和配置虚拟机，可参考别的教程。

*  **三节点**的集群，查看机器配置如下：

<table style="margin-left:18pt;"> 
 <tbody> 
 <tr> 
 <td style="vertical-align:top;width:69.8pt;"> IP地址 </td> 
 <td style="vertical-align:top;width:58.1pt;"> 主机别名 </td> 
 <td style="vertical-align:top;width:141.5pt;"> 说明 </td> 
 <td style="vertical-align:top;width:64.15pt;"> 可用内存 </td> 
 <td style="vertical-align:top;width:63.25pt;"> 可用硬盘 </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:69.8pt;"> 10.1.4.18 </td> 
 <td style="vertical-align:top;width:58.1pt;"> b3 </td> 
 <td style="vertical-align:top;width:141.5pt;"> 主节点master(namenode),datanode </td> 
 <td style="vertical-align:top;width:64.15pt;"> 4.3GB </td> 
 <td style="vertical-align:top;width:63.25pt;"> 44GB </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:69.8pt;"> 10.1.4.19 </td> 
 <td style="vertical-align:top;width:58.1pt;"> b4 </td> 
 <td style="vertical-align:top;width:141.5pt;"> datanode </td> 
 <td style="vertical-align:top;width:64.15pt;"> 4.9GB </td> 
 <td style="vertical-align:top;width:63.25pt;"> 44GB </td> 
 </tr> 
 <tr> 
 <td style="vertical-align:top;width:69.8pt;"> 10.1.4.20 </td> 
 <td style="vertical-align:top;width:58.1pt;"> b5 </td> 
 <td style="vertical-align:top;width:141.5pt;"> datanode </td> 
 <td style="vertical-align:top;width:64.15pt;"> 4.8GB </td> 
 <td style="vertical-align:top;width:63.25pt;"> 44GB </td> 
 </tr> 
 </tbody> 
</table>

*  机器系统信息：

[root@bi-zhaopeng03 songzehao]# cat /proc/version 
    Linux version 3.10.0-229.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42 UTC 2015
    [root@bi-zhaopeng03 songzehao]# uname -a
    Linux bi-zhaopeng03 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

## 搭建工作 ##

### JDK安装 ###

因为hadoop是由Java写的，很多启动程序都需要java的环境。JDK安装没什么特别的介绍，安装完查看信息。

[root@bi-zhaopeng03 songzehao]# java -version
    java version "1.8.0_191"
    Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
    Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
    [root@bi-zhaopeng03 songzehao]# echo $JAVA_HOME
    /usr/local/java

### 相关安装介质获取 ###

*  Mysql：

[*https://dev.mysql.com/get/Downloads/MySQL-5.5/**mysql-5.5.62-linux-glibc2.12-x86\_64.tar.gz***][https_dev.mysql.com_get_Downloads_MySQL-5.5_mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz]

*  JDBC驱动：

[*https://dev.mysql.com/get/Downloads/Connector-J/**mysql-connector-java-5.1.47.tar.gz***][https_dev.mysql.com_get_Downloads_Connector-J_mysql-connector-java-5.1.47.tar.gz]

*  cloudera manager包：

[*http://archive.cloudera.com/cm5/cm/5/**cloudera-manager-centos7-cm5.7.2\_x86\_64.tar.gz***][http_archive.cloudera.com_cm5_cm_5_cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz]

*  CDH包：

[*http://archive.cloudera.com/cdh5/parcels/5.7.2/**CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel***][http_archive.cloudera.com_cdh5_parcels_5.7.2_CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel]

[*http://archive.cloudera.com/cdh5/parcels/5.7.2/**CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1***][http_archive.cloudera.com_cdh5_parcels_5.7.2_CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1]

[*http://archive.cloudera.com/cdh5/parcels/5.7.2/**manifest.json***][http_archive.cloudera.com_cdh5_parcels_5.7.2_manifest.json]

### 配置hostname(每台机器) ###

[root@bi-zhaopeng03 songzehao]# vi /etc/sysconfig/network
    NETWORKING=yes
    HOSTNAME=bi-zhaopeng03
    [root@bi-zhaopeng03 songzehao]# vi /etc/hosts
    10.1.4.18  bi-zhaopeng03 b3 （说明：b3也是hostname的别名）
    10.1.4.19  bi-zhaopeng04 b4
    10.1.4.20  bi-zhaopeng05 b5

### 关闭selinux(每台机器) ###

配置selinux需重启后生效

[root@bi-zhaopeng03 songzehao]# vi /etc/sysconfig/selinux
    SELINUX=disabled
    [root@bi-zhaopeng03 songzehao]# sestatus -v
    SELinux status: disabled

###  关闭防火墙(每台机器) ###

[root@bi-zhaopeng03 songzehao]# systemctl stop firewalld
    [root@bi-zhaopeng03 songzehao]# systemctl disable firewalld
    rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
    rm '/etc/systemd/system/basic.target.wants/firewalld.service'
    [root@bi-zhaopeng03 songzehao]# systemctl status firewalld
    firewalld.service - firewalld - dynamic firewall daemon
        Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
        Active: inactive (dead)

### NTP网络时间同步(每台机器) ###

首先安装ntp服务

[root@bi-zhaopeng03 songzehao]# yum -y install ntp
    或
    [root@bi-zhaopeng03 songzehao]# yum install -y ntpdate

然后同步时间，如下是CentOS系统使用NTP来从一个时间服务器同步，把当前时区调整为上海就是+8区，想改其他时区也可以去看看/usr/share/zoneinfo目录

[root@bi-zhaopeng03 songzehao]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
    [root@bi-zhaopeng03 songzehao]# ntpdate us.pool.ntp.org

启动和查看ntp服务

[root@bi-zhaopeng03 songzehao]# systemctl start ntpd  //启动ntp服务
    [root@bi-zhaopeng03 songzehao]# systemctl status ntpd //查看ntp服务状态

### SSH无密码登录配置 (每台机器) ###

检验机器之间是否已配置好SSH免密登录，可ssh hostname测试，我这里的三台机器已被配置好。下面贴出其他小伙伴的操作示范

[root@cdh1 /]# ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key (/root/.ssh/id_rsa):
    /root/.ssh/id_rsa already exists.
    Overwrite (y/n)? y
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in /root/.ssh/id_rsa.
    Your public key has been saved in /root/.ssh/id_rsa.pub.
    The key fingerprint is:
    1d:e9:b4:ed:1d:e5:c6:a7:f3:23:ac:02:2b:8c:fc:ca root@cdh1
    The key's randomart image is:
    +--[ RSA 2048]----+
    |                 |
    |           .     |
    |          +     .|
    |         + +   + |
    |        S + . . =|
    |       .   . . +.|
    |  . o   o   o +  |
    |  .o o . .   o + |
    |   Eo..   ... . o|
    +-----------------+
    [root@cdh1 /]# ssh-copy-id 192.168.42.129
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    root@192.168.42.129's password:
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh '192.168.42.129'"
    and check to make sure that only the key(s) you wanted were added.

### MySQL安装 ###

可参考文首的文章使用rpm方式安装，这里参考百度经验：[https://jingyan.baidu.com/article/a378c9609eb652b3282830fd.html][https_jingyan.baidu.com_article_a378c9609eb652b3282830fd.html]，解压安装（如下仅为解压操作，省略的部分配置步骤安装百度经验即可）

[root@bi-zhaopeng03 songzehao]# tar mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz
    [root@bi-zhaopeng03 songzehao]# cp mysql-5.5.62-linux-glibc2.12-x86_64 /usr/local/mysql -r
    ...

安装完查看信息

[root@bi-zhaopeng03 songzehao]# mysql --help |grep Distrib
    mysql  Ver 14.14 Distrib 5.5.62, for linux-glibc2.12 (x86_64) using readline 5.1

修改密码解压JDBC驱动（需在各节点创建目录***/usr/share/java/***，并将驱动包重命名后放进去）

[root@bi-zhaopeng03 songzehao]# mysql -u root -p          //登录进行去更改密码
    mysql> SET PASSWORD=PASSWORD('123456');
    mysql> update user set host='%' where user='root' and host='localhost';   //允许mysql远程访问
    Query OK, 1 row affected (0.05 sec)
    Rows matched: 1  Changed: 1  Warnings: 0
    mysql> flush privileges;
    Query OK, 0 rows affected (0.00 sec)
    
    [root@bi-zhaopeng03 songzehao]# chkconfig mysql on   //配置开机启动
    [root@bi-zhaopeng03 songzehao]# tar -zcvf mysql-connector-java-5.1.44.tar.gz // 解压mysql-connector-java-5.1.44.tar.gz得到mysql-connector-java-5.1.44-bin.jar
    [root@bi-zhaopeng03 songzehao]# mkdir /usr/share/java // 在各节点创建java文件夹
    [root@bi-zhaopeng03 songzehao]# cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar //将mysql-connector-java-5.1.44-bin.jar拷贝到/usr/share/java路径下并重命名为mysql-connector-java.jar

创建数据库

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
    create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
    create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
     create database monitor DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
    create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
    Query OK, 1 row affected (0.00 sec)
    grant all on *.* to root@"%" Identified by "123456";

### 安装Cloudera-Manager ###

1.解压cm tar包到所有服务器指定目录（或者在主节点解压好，然后通过scp到**各个节点**同一目录下）

[root@bi-zhaopeng03 songzehao]# mkdir /opt/cloudera-manager
    [root@bi-zhaopeng03 songzehao]# tar -axvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager

2.创建cloudera-scm用户**（所有节点）**

[root@bi-zhaopeng03 songzehao]# useradd --system --home=/opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3.在主节点创建cloudera-manager-server的本地元数据保存目录

[root@bi-zhaopeng03 songzehao]# mkdir /var/cloudera-scm-server
    [root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /var/cloudera-scm-server
    [root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera-manager

4.配置从节点cloudera-manger-agent指向主节点服务器

[root@bi-zhaopeng04 ~]# vi /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini
    将server_host改为CMS所在的主机名即b3
    # Hostname of the CM server.
    server_host=b3

5.主节点中创建parcel-repo仓库目录

[root@bi-zhaopeng03 songzehao]# mkdir -p /opt/cloudera/parcel-repo
    [root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
    [root@bi-zhaopeng03 songzehao]# cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo
    注意：其中CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 后缀要把1去掉

6.创建parcels目录（**所有节点**）

[root@bi-zhaopeng03 songzehao]# mkdir -p /opt/cloudera/parcels
    [root@bi-zhaopeng03 songzehao]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
    解释：Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来，分发解压激活到各个节点的/opt/cloudera/parcels目录中

7.初始脚本配置数据库scm\_prepare\_database.sh（在主节点上）

[root@bi-zhaopeng03 songzehao]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hb3 -uroot -p123456 --scm-host b3 scmdbn scmdbu scmdbp
    说明：这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指：
    mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。
    -b3：数据库建立在b3主机上面,也就是主节点上面。
    -uroot：root身份运行mysql。-123456：mysql的root密码是***。
    --scm-host b3：CMS的主机，一般是和mysql安装的主机是在同一个主机上,最后三个参数是：数据库名，数据库用户名，数据库密码。

此处执行可能会报权限不够的错，参考以下或另行处理即可

update user set PASSWORD=PASSWORD('123456') where user='root';
    GRANT ALL PRIVILEGES ON *.* TO 'root'@'b3' IDENTIFIED BY '123456' WITH GRANT OPTION;
    FLUSH PRIVILEGES;

此处执行可能报以下错：

[root@bi-zhaopeng03 ~]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hb3 -uroot -p123456 --scm-host b3 scmdbn scmdbu scmdbp
    JAVA_HOME=/usr/local/java
    Verifying that we can write to /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-server
    2019-06-05 11:21:14,867 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://b3/?useUnicode=true&characterEncoding=UTF-8'
    java.sql.SQLException: Can't create database 'scmdbn'; database exists
            at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
            at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
            at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
            at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
            at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
            at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
            at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
            at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
            at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
            at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:286)
            at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)
            at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)
    2019-06-05 11:21:14,870 [main] ERROR com.cloudera.enterprise.dbutil.DbProvisioner  - Stack Trace:
    java.sql.SQLException: Can't create database 'scmdbn'; database exists
            at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
            at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
            at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
            at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
            at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)
            at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2491)
            at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2449)
            at com.mysql.jdbc.StatementImpl.executeInternal(StatementImpl.java:845)
            at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:745)
            at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:286)
            at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:95)
            at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:110)
    --> Error 1, giving up (use --force if you wish to ignore the error)

可尝试drop掉数据库scmdbn

drop database if exists scmdbn;

也可尝试参考此解决方案：[https://bbs.csdn.net/topics/391836526][https_bbs.csdn.net_topics_391836526]

重新执行初始化CM5数据库命令
    /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -p123456 --scm-host localhost scmdbn scmdbu scmdbp
    再次执行 /opt/cm-5.6.1/etc/init.d/cloudera-scm-server start 启动服务端。
    再次执行 /opt/cm-5.6.1/etc/init.d/cloudera-scm-agent start 启动客户端。

8.启动主节点

[root@bi-zhaopeng03 songzehao]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
    [root@bi-zhaopeng03 songzehao]# chkconfig cloudera-scm-server on
    [root@bi-zhaopeng03 songzehao]# vi /etc/init.d/cloudera-scm-server
    CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default
    [root@bi-zhaopeng03 songzehao]# service cloudera-scm-server start
    //同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-server restart

9.启动cloudera-scm-agent（**所有节点**）

[root@bi-zhaopeng03 songzehao]# mkdir /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-agent
    [root@bi-zhaopeng03 songzehao]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
    [root@bi-zhaopeng03 songzehao]# chkconfig cloudera-scm-agent on
    [root@bi-zhaopeng03 songzehao]# vi /etc/init.d/cloudera-scm-agent
    CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}改为=/opt/cloudera-manager/cm-5.7.2/etc/default
    [root@bi-zhaopeng03 songzehao]# service cloudera-scm-agent start
    //同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent，应该在开机启动脚本/etc/rc.local中加入命令：service cloudera-scm-agent restart

### 在浏览器安装CDH ###

--------------------

面对一个陌生环境的搭建，前面肯定有各种想到想不到查到查不到的问题，要把握住两点，

*  要相信所有问题最终都能被解决掉；
 *  细心查看日志（/opt/cloudera-manager/cm-5.7.2/log/cloudera-scm-server/ cloudera-scm-server.log）和数据库脚本（/opt/cloudera-manager/cm-5.7.2/lib64/cmf/service/common/cloudera-config.sh）内容，分析出错原因。

--------------------

等待主节点完成启动就在浏览器中进行操作了，进入b3:7180 默认使用admin admin登录，在浏览器中使用操作安装，可直接参考[https://www.cnblogs.com/zhangleisanshi/p/7575579.html][https_www.cnblogs.com_zhangleisanshi_p_7575579.html]与[https://www.cnblogs.com/pojishou/archive/2017/01/12/6267616.html][https_www.cnblogs.com_pojishou_archive_2017_01_12_6267616.html]

（1）注意在【**群集设置 - 选择您要再群集上安装的CDH5服务**】步骤里，选择要安装的服务组合，量力而行，我的机器配置较低，所以暂时只选择了“核心Hadoop”组合，之后需要其他组件另行安装即可。

（2）注意在【**群集安装 - 检查主机正确性**】步骤里，若报错“检查器在所有主机上运行失败.”或“该群集的主机上未安装 Java”，查看server日志，应该会报错“JAVA\_HOME is not set and could not be found”。这时候，我们的jdk肯定是正常安装了，所以去查看/opt/cloudera-manager/cm-5.7.2/lib64/cmf/service/common/cloudera-config.sh，可以发现它认定的JAVA8\_HOME中不包含“/usr/local/java”，因此一种方式是软链接到它指定的目录，另一种是在它的集合中加进去“/usr/local/java”，如下：

local JAVA8_HOME_CANDIDATES=(
      '/usr/local/java'
      '/usr/java/jdk1.8'
      '/usr/java/jre1.8'
      '/usr/lib/jvm/j2sdk1.8-oracle'
      '/usr/lib/jvm/j2sdk1.8-oracle/jre'
      '/usr/lib/jvm/java-8-oracle'
    )

最终此步骤成功

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70][]

（3）注意在【**群集设置 - 数据库**】步骤里，若报错“A package was not selected.”，如下：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70 1][]

莫名其妙，日志也没有追加新的信息，返回几步重新再点过来就好了。

安装成功

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70 2][]

1.  hadoop配置文件目录： /etc/hadoop/conf（包括core-site.xml，hdfs-site.xml等）
2.  hdfs文件系统地址：cat /etc/hadoop/conf/core-site.xml，查找fs.defaultFS = hdfs://bi-zhaopeng03:8020
3.  cloudera-manager的web管理地址：主节点ip:7180 
4.  hdfs的web管理地址：主节点ip:50070
5.  yarn的web管理地址：主节点ip:8088
6.  hue的web管理地址：主节点ip:8888

## 测试工作 ##

在集群的一台机器上执行以下模拟圆周率Pi（π）的示例程序：

sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100

通过YARN的Web管理界面（[http://10.1.4.18:8088][http_10.1.4.18_8088]）也可以看到MapReduce的执行状态：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70 3][]

查看MapReduce应用程序的完整日志：

[root@bi-zhaopeng04 ~]# sudo -u hdfs yarn logs -applicationId application_1559721784881_0001

MapReduce执行过程中终端的输出如下：

[root@bi-zhaopeng04 ~]# sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 10 100
    Number of Maps  = 10
    Samples per Map = 100
    Wrote input for Map #0
    Wrote input for Map #1
    Wrote input for Map #2
    Wrote input for Map #3
    Wrote input for Map #4
    Wrote input for Map #5
    Wrote input for Map #6
    Wrote input for Map #7
    Wrote input for Map #8
    Wrote input for Map #9
    Starting Job
    19/06/06 14:08:23 INFO client.RMProxy: Connecting to ResourceManager at bi-zhaopeng03/10.1.4.18:8032
    19/06/06 14:08:23 INFO input.FileInputFormat: Total input paths to process : 10
    19/06/06 14:08:23 INFO mapreduce.JobSubmitter: number of splits:10
    19/06/06 14:08:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1559721784881_0001
    19/06/06 14:08:24 INFO impl.YarnClientImpl: Submitted application application_1559721784881_0001
    19/06/06 14:08:24 INFO mapreduce.Job: The url to track the job: http://bi-zhaopeng03:8088/proxy/application_1559721784881_0001/
    19/06/06 14:08:24 INFO mapreduce.Job: Running job: job_1559721784881_0001
    19/06/06 14:08:29 INFO mapreduce.Job: Job job_1559721784881_0001 running in uber mode : false
    19/06/06 14:08:29 INFO mapreduce.Job:  map 0% reduce 0%
    19/06/06 14:08:34 INFO mapreduce.Job:  map 10% reduce 0%
    19/06/06 14:08:37 INFO mapreduce.Job:  map 20% reduce 0%
    19/06/06 14:08:40 INFO mapreduce.Job:  map 30% reduce 0%
    19/06/06 14:08:43 INFO mapreduce.Job:  map 40% reduce 0%
    19/06/06 14:08:46 INFO mapreduce.Job:  map 50% reduce 0%
    19/06/06 14:08:49 INFO mapreduce.Job:  map 60% reduce 0%
    19/06/06 14:08:52 INFO mapreduce.Job:  map 70% reduce 0%
    19/06/06 14:08:55 INFO mapreduce.Job:  map 80% reduce 0%
    19/06/06 14:08:58 INFO mapreduce.Job:  map 90% reduce 0%
    19/06/06 14:09:06 INFO mapreduce.Job:  map 90% reduce 30%
    19/06/06 14:13:30 INFO mapreduce.Job:  map 90% reduce 0%
    19/06/06 14:13:33 INFO mapreduce.Job:  map 100% reduce 0%
    19/06/06 14:13:38 INFO mapreduce.Job:  map 100% reduce 100%
    19/06/06 14:13:38 INFO mapreduce.Job: Job job_1559721784881_0001 completed successfully
    19/06/06 14:13:38 INFO mapreduce.Job: Counters: 50
            File System Counters
                    FILE: Number of bytes read=91
                    FILE: Number of bytes written=1310718
                    FILE: Number of read operations=0
                    FILE: Number of large read operations=0
                    FILE: Number of write operations=0
                    HDFS: Number of bytes read=2680
                    HDFS: Number of bytes written=215
                    HDFS: Number of read operations=43
                    HDFS: Number of large read operations=0
                    HDFS: Number of write operations=3
            Job Counters 
                    Killed reduce tasks=1
                    Launched map tasks=10
                    Launched reduce tasks=2
                    Data-local map tasks=10
                    Total time spent by all maps in occupied slots (ms)=17328
                    Total time spent by all reduces in occupied slots (ms)=272280
                    Total time spent by all map tasks (ms)=17328
                    Total time spent by all reduce tasks (ms)=272280
                    Total vcore-seconds taken by all map tasks=17328
                    Total vcore-seconds taken by all reduce tasks=272280
                    Total megabyte-seconds taken by all map tasks=17743872
                    Total megabyte-seconds taken by all reduce tasks=278814720
            Map-Reduce Framework
                    Map input records=10
                    Map output records=20
                    Map output bytes=180
                    Map output materialized bytes=340
                    Input split bytes=1500
                    Combine input records=0
                    Combine output records=0
                    Reduce input groups=2
                    Reduce shuffle bytes=340
                    Reduce input records=20
                    Reduce output records=0
                    Spilled Records=40
                    Shuffled Maps =10
                    Failed Shuffles=0
                    Merged Map outputs=10
                    GC time elapsed (ms)=589
                    CPU time spent (ms)=5390
                    Physical memory (bytes) snapshot=4883341312
                    Virtual memory (bytes) snapshot=31219437568
                    Total committed heap usage (bytes)=4787273728
            Shuffle Errors
                    BAD_ID=0
                    CONNECTION=0
                    IO_ERROR=0
                    WRONG_LENGTH=0
                    WRONG_MAP=0
                    WRONG_REDUCE=0
            File Input Format Counters 
                    Bytes Read=1180
            File Output Format Counters 
                    Bytes Written=97
    Job Finished in 315.215 seconds
    Estimated value of Pi is 3.14800000000000000000

机器比较慢，执行了315s，不过结果也是成功计算了出来，说明我们的CDH5已搭建成功。

[https_www.cnblogs.com_zhangleisanshi_p_7575579.html]: https://www.cnblogs.com/zhangleisanshi/p/7575579.html
[https_www.cnblogs.com_pojishou_archive_2017_01_12_6267616.html]: https://www.cnblogs.com/pojishou/archive/2017/01/12/6267616.html
[https_dev.mysql.com_get_Downloads_MySQL-5.5_mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz]: https://dev.mysql.com/get/Downloads/MySQL-5.5/mysql-5.5.62-linux-glibc2.12-x86_64.tar.gz
[https_dev.mysql.com_get_Downloads_Connector-J_mysql-connector-java-5.1.47.tar.gz]: https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.16-1.el6.noarch.rpm
[http_archive.cloudera.com_cm5_cm_5_cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz]: http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz
[http_archive.cloudera.com_cdh5_parcels_5.7.2_CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel]: http://archive.cloudera.com/cdh5/parcels/5.7.2/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel
[http_archive.cloudera.com_cdh5_parcels_5.7.2_CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1]: http://archive.cloudera.com/cdh5/parcels/5.7.2/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha1
[http_archive.cloudera.com_cdh5_parcels_5.7.2_manifest.json]: http://archive.cloudera.com/cdh5/parcels/5.7.2/manifest.json
[https_jingyan.baidu.com_article_a378c9609eb652b3282830fd.html]: https://jingyan.baidu.com/article/a378c9609eb652b3282830fd.html
[https_bbs.csdn.net_topics_391836526]: https://bbs.csdn.net/topics/391836526
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70]: /images/20220122/63d77b17b84d42d883cd19b300dd1b6e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70 1]: /images/20220122/564030606d28435e9c88f3131c0929f2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70 2]: /images/20220122/7e41994a7a6b432697b7bc8672aa668a.png
[http_10.1.4.18_8088]: http://10.1.4.18:8088
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3Nvbmd6ZWhhbw_size_16_color_FFFFFF_t_70 3]: /images/20220122/1a816e038884452bb1ab464774a9e62b.png