docker部署ambari-2.7.3（大数据集群快速搭建）大数据服务组件

灰太狼 2022-11-01 13:46 1124阅读 0赞

之前有使用docker搭建了一套Apache Hadoop版本的大数据平台，整个编写脚本和搭建过程花了很多时间，且灵活性不好。Ambari在大数据集群部署方面有得天独厚的优势，但是集群操作系统安装准备工作以及基础包的安装还是需要花费很多的时间。为了节省大数据集群的部署时间接下来我们用Docker容器化的方案部署Ambari。

费话少说，放码出来。让我们开始吧！

# Ambari的架构 #

从Ambari的架构主要有两个组件：Ambari Server和Ambari Agent。

![Ambari架构图][Ambari]

Ambari架构图

# 用docker搭建单机hortonworks/ambari集群环境   参考：[https://hub.docker.com/r/hortonworks/ambari-server/][https_hub.docker.com_r_hortonworks_ambari-server] #

[https://hub.docker.com/r/sequenceiq/hadoop-docker][https_hub.docker.com_r_sequenceiq_hadoop-docker]

一、前期docker环境配置：

1.安装docker服务：

yum -y install docker

2.启动docker服务：

service docker start 或者systemctl start docker

3.查询验证正在运行的docker容器：

docker ps

4,.安装过程中需要重国外docker仓库下载文件，速度太慢，建议配置docker国内镜像仓库：

vi /etc/docker/daemon.json
    
    { "registry-mirrors":["http://c1f0a193.m.daocloud.io"] }

二、安装ambari的docker集群：

提前关闭防火墙和selinux;

1.从GitHub上下载ambari文件到本地虚拟机，[https://github.com/sequenceiq/docker-ambari][https_github.com_sequenceiq_docker-ambari]

虚拟机环境 CentOS Linux release 7.2.1511 (Core)

2.unzip docker-ambari-master .zip 到虚拟机/opt目录下;

3.进入cd /opt/docker-ambari-master，看到有 ambari-functions文件，执行：

.ambari-functions or source ambari-functions

4. 提前执行 amb-settings

可以看到此时IP为空

AMBARI\_SERVER\_IP=

5.执行启动命令，自动下载安装docker集群：

amb-start-cluster 3

注意：该过程时间可能有点长，跟网络有关，请耐心等待。

6. 安装结束后查询docker容器可以看到正在运行的4个集群：

[root@lydia opt]# docker ps
    CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                                                              NAMES
    5fb305ae34f4        hortonworks/ambari-agent:latest    "/usr/sbin/init syste"   19 minutes ago      Up 19 minutes                                                                          amb2
    0bbbbbbaf823        hortonworks/ambari-agent:latest    "/usr/sbin/init syste"   19 minutes ago      Up 19 minutes                                                                          amb1
    7f5aa2f8d48c        hortonworks/ambari-server:latest   "/usr/sbin/init syste"   19 minutes ago      Up 19 minutes       8080/tcp                                                           amb-server
    1fe7970ccce0        sequenceiq/consul:v0.5.0-v6        "/bin/start -server -"   22 minutes ago      Up 22 minutes       53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8500/tcp, 8301-8302/udp   amb-consul

可以看保安装运行成功，其中包含1个server，2个agent和1个consulserver，分别对应不同的容器。

7.目前容器无法连接外网，所以还需要配置：

编辑虚拟机文件：vi /etc/sysctl.conf

增加：net.ipv4.ip\_forward=1

重启服务：systemctl restart network

查看属性是否修改成功：sysctlnet.ipv4.ip\_forward

进入server容器ping外网看能否成功：docker exec -it amb-server /bin/bash

8.再次执行amb-settings，查看server已经获取了 IP：

AMBARI\_SERVER\_IP=172.17.0.3

9.在虚拟机内部用浏览器登陆ambari：http://172.17.0.3:8080，可以看到已经安装成功，接下来可以配置使用集群了；  
  
原文链接：https://blog.csdn.net/feixiangmaomao/article/details/79225643

1.docker部署ambari  
① centos7自身没有yum工具

apt-get install yum -y

② 安装git

yum -y install git

③ 拉取远程仓库

git clone https://gitee.com/horysk/docker-ambari-2.7.3

④ 开始安装

cd docker-ambari-2.7.3
    
    cd ambari-base-local
    
    wget -O local.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo
    
    cd ..
    
    cd ambari-server-local
    
    wget -O local.repo http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo
    
    cd ..

#制作镜像
    ./scripts/build-image.sh
    
    #本地启动
    ./scripts/create-local.sh

⑤ 查看容器

\#查看正在运行的容器

docker ps

\# 获得web ip

docker inspect -f '{
       {range .NetworkSettings.Networks}}{
       {.IPAddress}}{
       {end}}' amb-server

\#访问ambari界面

http://ambari-server（IP）:8080

\# 查看其它节点的ip

docker inspect 容器id | grep -i ip

2.配置ssh免密登录  
我这里的三个容器的ip分别是

172.17.0.2（amb-server)
    172.17.0.3(amb1)
    172.17.0.4(amb2)

① 修改主机名

\# 分别进入每个容器修改hostname

sudo hostnamectl set-hostname node1
    172.17.0.2（amb-server) -> node3
    172.17.0.3(amb1) -> node1
    172.17.0.4(amb2) -> node2

② 修改hosts文件，可以直接ping名字

\#分别进入每个容器

docker exec -it amb-server /bin/bash

vi /etc/hosts
    
    172.17.0.2 node3
    172.17.0.3 node1
    172.17.0.4 node2

\#同时要注释掉：

#172.17.0.3 amb1.service.consul amb1
    #172.17.0.4 amb2.service.consul amb2

③ 生成密钥

首先分别在这3个节点执行指令：ssh-keygen ，然后一直按回车键直到结束。此时你可以发现在 /root/.ssh/ 文件夹下生成了id\_rsa（私钥） 、 id\_rsa.pub （公钥），这是生成的秘钥对

④公钥分享

同步公钥到所有节点需要输入root的密码，3个节点都设置root密码为 ambari123

sudo passwd  
1  
把自己的公钥发送给其它节点，包括自己。指令如下：

ssh-copy-id node1
    
    ssh-copy-id node2
    
    ssh-copy-id node3
    
    ⑤ 配置ambari基础repo（已经配置好，直接进入3）
    
    从如下地址找到hdp3.1.4的yum repo地址
    
    http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
    
    hdp-3.1.4的yum repo地址：
    
    http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-315.xml
    
    打开可以看到hdp, hdp-gpl, hdp-utils的repo
    
    3.辅助repo配置
    #进入每个容器，执行如下操作
    cd /etc/yum.repos.d/
    cp backup_repo/CentOS-Base.repo .
    
    4.创建集群
    点击：LAUNCH INSTALL WIZARD
    取名：idc
    选择HDP3.1
    下一步需要输入ambari-server所在机器的私钥，这样ambari-server才能把组件部署到其他节点
    docker exec -it amb-server /bin/bash
    
    cd /root/.ssh
    
    id_rsa
    
    只能添加node1, node2

接下来按照项目需求进行安装服务  
原文链接：https://blog.csdn.net/qq\_40107474/article/details/111281245

[https://blog.csdn.net/weixin\_41772761/article/details/104536980?utm\_medium=distribute.pc\_relevant.none-task-blog-baidujs\_title-0&spm=1001.2101.3001.4242][https_blog.csdn.net_weixin_41772761_article_details_104536980_utm_medium_distribute.pc_relevant.none-task-blog-baidujs_title-0_spm_1001.2101.3001.4242]

# 编写CentOS7支持Systemd镜像Dockfile #

FROM centos:7
    
    
    MAINTAINER elbert.lau
    
    
    # 2, Setting up systemd
    
    ENV container docker
    
    RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
    
    systemd-tmpfiles-setup.service ] || rm -f $i; done); \
    
    rm -f /lib/systemd/system/multi-user.target.wants/*;\
    
    rm -f /etc/systemd/system/*.wants/*;\
    
    rm -f /lib/systemd/system/local-fs.target.wants/*; \
    
    rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
    
    rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
    
    rm -f /lib/systemd/system/basic.target.wants/*;\
    
    rm -f /lib/systemd/system/anaconda.target.wants/*;
    
    
    VOLUME [ "/sys/fs/cgroup" ]
    
    ENTRYPOINT ["/usr/sbin/init"]
    
    
    RUN yum install -y systemd* && yum clean all

1.  构建ambari-systemd镜像

#!/bin/bash
    
    
    docker rmi local/ambari-systemd:2.6.2.0-155
    
    docker build --rm -t local/ambari-systemd:2.6.2.0-155 .

# 编写Ambari Repos镜像Dockerfile #

FROM local/ambari-systemd:2.6.2.0-155
    
    
    MAINTAINER elbert.lau
    
    
    RUN yum -y install httpd; yum clean all; systemctl enable httpd.service
    
    
    ADD http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/ambari-2.6.2.0-centos7.tar.gz /var/www/html
    
    
    ADD http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz /var/www/html
    
    
    ADD http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz /var/www/html/HDP-UTILS
    
    
    ADD http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/HDP-GPL-2.6.5.0-centos7-gpl.tar.gz /var/www/html
    
    
    EXPOSE 80

# 构建ambari-repos镜像 #

#!/bin/bash
    
    
    docker rmi local/ambari-repos:2.6.2.0-155
    
    docker build --rm -t local/ambari-repos:2.6.2.0-155 .

# 编写Ambari Base镜像Dockerfile #

FROM local/ambari-systemd:2.6.2.0-155
    
    
    MAINTAINER elbert.lau
    
    
    # 1, epel-release install
    
    RUN yum install epel-release -y && yum clean all
    
    RUN yum update -y && yum clean all
    
    
    RUN yum install -y yum-utils yum-plugin-ovl tar git curl bind-utils unzip wget && yum clean all
    
    
    # 2, Setup sshd
    
    RUN yum install -y openssh-server openssh-clients && yum clean all
    
    RUN systemctl enable sshd
    
    
    # 3, kerberos client
    
    RUN yum install -y krb5-workstation && yum clean all
    
    
    # 4, initscripts (service wrapper for servicectl) is need othewise the Ambari is unable to start postgresql
    
    RUN yum install -y initscripts && yum clean all
    
    
    RUN curl -o /usr/bin/jq http://stedolan.github.io/jq/download/linux64/jq && chmod +x /usr/bin/jq
    
    
    ENV JDK_ARTIFACT jdk-8u112-linux-x64.tar.gz
    
    ENV JDK_VERSION jdk1.8.0_112
    
    
    # 5, install Ambari specified 1.8 jdk
    
    RUN mkdir /usr/jdk64 && cd /usr/jdk64 && wget http://public-repo-1.hortonworks.com/ARTIFACTS/$JDK_ARTIFACT \
    
    && tar -xf $JDK_ARTIFACT && rm -f $JDK_ARTIFACT
    
    
    ENV JAVA_HOME /usr/jdk64/$JDK_VERSION
    
    ENV PATH $PATH:$JAVA_HOME/bin
    
    
    # 6, jce
    
    ADD http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip $JAVA_HOME/jre/lib/security/
    
    RUN cd $JAVA_HOME/jre/lib/security && unzip jce_policy-8.zip \
    
    && rm -f jce_policy-8.zip && mv UnlimitedJCEPolicyJDK8/*jar . && rm -rf UnlimitedJCEPolicyJDK8
    
    
    ADD etc/yum.conf /etc/yum.conf
    
    
    # 7, for ERROR: SSLError: Failed to connect. Please check openssl library versions.
    
    RUN sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

# 构建ambari-base镜像 #

#!/bin/bash
    
    
    docker rmi local/ambari-base:2.6.2.0-155
    
    docker build --rm -t local/ambari-base:2.6.2.0-155 .

# 编写Ambari Server镜像Dockerfile #

FROM local/ambari-base:2.6.2.0-155
    
    
    MAINTAINER elbert.lau
    
    
    # 1, add ambari repo
    
    ADD http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/ambari.repo /etc/yum.repos.d/ambari.repo
    
    ADD http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/hdp.repo /etc/yum.repos.d/hdp.repo
    
    ADD http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0/hdp.gpl.repo /etc/yum.repos.d/hdp.gpl.repo
    
    
    # 2, initscripts (service wrapper for servicectl) is need othewise the Ambari is unable to start postgresql
    
    RUN yum -y install ambari-server; yum clean all
    
    
    # 3, add psql connectors to ambari-server so it can be downloaded by services (e.g.: Ranger)
    
    ADD https://jdbc.postgresql.org/download/postgresql-42.2.5.jar /var/lib/ambari-server/resources/postgres-jdbc-driver.jar
    
    
    RUN echo DefaultEnvironment=\"JAVA_HOME=$JAVA_HOME\" >> /etc/systemd/system.conf
    
    
    # 4, ambari server setup
    
    RUN ambari-server setup --silent --java-home $JAVA_HOME; systemctl enable ambari-server.service
    
    
    EXPOSE 8080

# 构建ambari-server镜像 #

#!/bin/bash
    
    
    docker rmi local/ambari-server:2.6.2.0-155
    
    docker build --rm -t local/ambari-server:2.6.2.0-155 .

# 编写Ambari Agent镜像Dockerfile #

# 构建ambari-agent镜像 #

#!/bin/bash
    
    
    docker rmi local/ambari-agent:2.6.2.0-155
    
    docker build --rm -t local/ambari-agent:2.6.2.0-155 .

# 参考资料 #

【1】[https://github.com/horysk/docker-ambari][https_github.com_horysk_docker-ambari]

# 大数据 和 docker #

用docker搭建hadoop集群

用docker搭建zookeeper集群

用docker搭建hbase集群

用docker搭建kafka集群

用docker搭建spark集群

用docker搭建storm集群

请参考 https://github.com/horysk/bigData-in-docker

[Ambari]: /images/20221024/3497905238b746c29c8408f22e95b861.png
[https_hub.docker.com_r_hortonworks_ambari-server]: https://hub.docker.com/r/hortonworks/ambari-server/
[https_hub.docker.com_r_sequenceiq_hadoop-docker]: https://hub.docker.com/r/sequenceiq/hadoop-docker
[https_github.com_sequenceiq_docker-ambari]: https://github.com/sequenceiq/docker-ambari
[https_blog.csdn.net_weixin_41772761_article_details_104536980_utm_medium_distribute.pc_relevant.none-task-blog-baidujs_title-0_spm_1001.2101.3001.4242]: https://blog.csdn.net/weixin_41772761/article/details/104536980?utm_medium=distribute.pc_relevant.none-task-blog-baidujs_title-0&spm=1001.2101.3001.4242
[https_github.com_horysk_docker-ambari]: https://github.com/horysk/docker-ambari