CDH搭建

Bertha 。 2022-05-14 00:17 185阅读 0赞

系统环境准备  
1、网络配置

vi /etc/sysconfig/network
    vi /etc/hosts

2、SSH免密钥登录

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    ssh-copy-id

3、防火墙关闭

service iptables stop
    chkconfig iptables off

4、SELINUX关闭

setenforce 0
    vi /etc/selinux/config (SELINUX=disabled)

5、安装JDK配置环境变量

export JAVA_HOME=/usr/java/jdkXXX
    export PATH=$JAVA_HOME/bin:$PATH
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

6、安装NTP  
设置开机启动

chkconfig ntpd on

设置时间同步

ntpdate 202.120.2.101

7、安装配置mysql

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123' WITH GRANT OPTION;
    flush privileges

8、下载第三方依赖包

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb -y

ClouderaManager安装  
1、安装Cloudera Manager Server、Agent

mkdir /opt/cloudera-manager（三个节点都要执行）
    tar xvzf cloudera-manager*.tar.gz -C /opt/cloudera-manager

2、创建用户cloudera-scm（所有节点都执行）

useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3、配置CM Agent  
修改文件  
/opt/cloudera-manager/cm-5.4.3/etc/cloudera-scm-agent/config.ini中server\_host  
4、配置CM Server数据库  
拷贝mysql jar文件到目录 /usr/share/java/ （需要在/usr/share手动创建一个java目录）  
注意jar包名称要修改为mysql-connector-java.jar

grant all on *.* to 'temp'@'%' identified by 'temp' with grant option; （创建用户）

[root@cdh1 ~]# service mysqld start
    Starting mysqld:                                           [  OK  ]
    [root@cdh1 ~]# mysql -uroot -p123;
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 2
    Server version: 5.1.73 Source distribution
    
    Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
    
    Oracle is a registered trademark of Oracle Corporation and/or its
    affiliates. Other names may be trademarks of their respective
    owners.
    
    Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
    
    mysql> grant all on *.* to 'temp'@'%' identified by 'temp' with grant option; 
    Query OK, 0 rows affected (0.01 sec)
    
    mysql> exit
    Bye

[root@cdh1 ~]# cd /opt/cloudera-manager/cm-5.4.3/share/cmf/schema/
    ./scm_prepare_database.sh mysql temp -h cdh1 -utemp -ptemp --scm-host cdh1 scm scm scm

格式：数据库类型、数据库、数据库服务器、用户名、密码、cm server服务器

5、创建Parcel目录  
Server节点

mkdir -p /opt/cloudera/parcel-repo
    chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

Agent节点(所有节点都要执行)

mkdir -p /opt/cloudera/parcels
    chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6、制作CDH本地源  
下载好文件CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel以及manifest.json，将这两个文件放到server节点的/opt/cloudera/parcel-repo下。

[root@cdh1 parcel-repo]# mv ~/install/CDH-5.4.0-1.cdh5.4.0.p0.27-el6.parcel* ./
    [root@cdh1 parcel-repo]# mv ~/install/manifest.json ./

打开manifest.json文件，里面是json格式的配置，找到与下载版本相对应的hash码，新建文件，文件名与你的parel包名一致，并加上.sha后缀，将hash码复制到文件中保存。

7、启动CM Server、Agent

cd /opt/cloudera-manager/cm-5.4.3/etc/init.d/
    ./cloudera-scm-server start

Server首次启动会自动创建表以及数据，不要立即关闭或重启，否则需要删除所有表及数据重新安装

./cloudera-scm-agent start