Ambari安装之部署3个节点的HA分布式集群

红太狼 2021-12-12 14:21 460阅读 0赞

转自：   [https://www.cnblogs.com/zlslch/p/6629251.html][https_www.cnblogs.com_zlslch_p_6629251.html]

Flink HA配置 （Standalone Cluster): [https://blog.csdn.net/weixin\_34128534/article/details/86885399][https_blog.csdn.net_weixin_34128534_article_details_86885399]

Ambari各版本兼容的组件查看路径: [https://blog.csdn.net/fengfengchen95/article/details/93326440][https_blog.csdn.net_fengfengchen95_article_details_93326440]

Docker 从零搭建 Hadoop 3.2 集群 (镜像基于Ubuntu 14.04):[https://blog.csdn.net/Y\_C\_S\_M/article/details/90731320][https_blog.csdn.net_Y_C_S_M_article_details_90731320]

hadoop 集群搭建部署: [https://blog.csdn.net/gallop\_liu/article/details/84462744][https_blog.csdn.net_gallop_liu_article_details_84462744]

使用ambari轻松一键式搭建大数据集群: [https://blog.csdn.net/qq\_26442553/article/details/88845920][https_blog.csdn.net_qq_26442553_article_details_88845920]

[Ambari安装之Ambari安装前准备(CentOS6.5）（一）][Ambari_Ambari_CentOS6.5]: [https://www.cnblogs.com/zlslch/p/6629235.html][Ambari_Ambari_CentOS6.5]

003-Ambari一键自动化部署脚本: [https://blog.csdn.net/zhangxiongcolin/article/details/81569710https://blog.csdn.net/zhangxiongcolin/article/details/81569710][https_blog.csdn.net_zhangxiongcolin_article_details_81569710https_blog.csdn.net_zhangxiongcolin_article_details_81569710]

Hadoop建设工具Ambari的安装部署及完整使用（三）——部署安装: [https://blog.csdn.net/balabalayi/article/details/64920580][https_blog.csdn.net_balabalayi_article_details_64920580]

前期博客

# [Ambari安装之部署单节点集群][Ambari] #

其实，按照这个步骤是一样的。只是按照好3个节点后，再做下HA即可。

## 部署3个节点的HA分布式集群 ##

（1）添加机器

和添加服务的操作类似，如下图

![855959-20170401094416789-673519060.png][]

![855959-20170401094429477-1484111006.png][]

之后的添加ambari03、ambari04操作就和部署ambari02类似，可以参照前面的步骤。这里不多赘述。

# [Ambari安装之部署单节点集群][Ambari] #

![855959-20170406143824957-1544134269.png][]

![855959-20170331153738117-996849019.png][]

![复制代码][copycode.gif]

![复制代码][copycode.gif]

![855959-20170406144225378-1032945336.png][]

![855959-20170406144330222-237959548.png][]

![855959-20170406144355878-775689214.png][]

![855959-20170406144955832-503146790.png][]

![855959-20170406151608675-570629403.png][]

（2）添加服务

添加ambari03、ambari04的服务的操作和前面的ambari02类似，具体可以参照前面的步骤。这里不多赘述。

# [Ambari安装之部署单节点集群][Ambari] #

**　当然，我下面这里就是，ambigdata1、ambigdata2、ambigdata3和ambigdata4，组成的4节点的HA大数据集群。**

我这里，就吧所有的服务都加进去。

![855959-20170624185434398-1301979805.png][]

![855959-20170624185446898-1245544413.png][]

（3）配置HA

选择，HDFS   ->  services  ->    service actions    ->  enable namenode HA

YARN  ->   services  ->    service actions    ->enable resourcemanager HA

以后再贴图。

接下来按照命令提示一步一步操作即可

这里，大家，如何控制下呢？

![855959-20170624191102195-535226621.png][]

比如，这里，我参考这篇博客

# [完全分布式Hadoop集群的安装搭建和配置（4节点）][Hadoop_4] #

![855959-20170624191511007-1473439929.png][]

在这里，为了今年实验室环境所需。所以，将计算节点动态设置为4。来迎合论文里的4、3、2和1节点，对比加速比。

则最后得到，是如下

![855959-20170624192546148-596706183.png][]

![855959-20170624192443976-1454147636.png][]

![855959-20170624192623773-166455080.png][]

![855959-20170624210550023-137892858.png][]

![855959-20170624193609101-1153858275.png][]

![855959-20170624193625179-707315895.png][]

![855959-20170624193831991-342559344.png][]

![855959-20170624194056991-1515496111.png][]

![855959-20170624194319413-1156697148.png][]

然后，下面就是，等同于apache版本的hive安装的配置文件hive-site.xml

![855959-20170624194435101-114332248.png][]

![855959-20170624194536632-1149224073.png][]

![855959-20170624194701429-2092236375.png][]

![855959-20170624194737491-89831711.png][]

然后呢，相信大部分博友，跟我一样，在这里，对下面三个现在，都不知道怎么设置。没关系，大家，可以先跳过。

![855959-20170624195322366-2114363748.png][]

![855959-20170624195856913-529751860.png][]

![855959-20170624195356570-1353860046.png][]

以后，再回来设置即可。这个不难。

![855959-20170624195525054-927288788.png][]

![855959-20170624195837210-127415465.png][]

![855959-20170624195934741-835834781.png][]

![855959-20170624200433601-346725777.png][]

保证所有服务都正常启动之后，下面我们就可以上传文件到HDFS，并运行Mapreduce了

![855959-20170624200616351-801134887.png][]

那就改成mysql数据库来吧

![855959-20170624200713382-329410294.png][]

这里啊，我查了下资料

数据库要求

Hive/HCatalog、Oozie和Ambari都需要一个内部数据库。

Hive/HCatalog：默认使用Ambari自带的Mysql5.x。也可以自己配置Oracle 11g r2。

Oozie：默认使用Ambari自带的Derby。也可以自己配置Mysql5.x和Oracle 11g r2。

Ambari：默认使用Ambari自带的PostgreSQL8.x，也可以使用Oracle 11g r2.

![855959-20170624201137913-660326268.png][]

![855959-20170624201200023-141266590.png][]

然后

![855959-20170624201255085-2123191110.png][]

![855959-20170624201317976-773664829.png][]

![855959-20170624201330398-4638739.png][]

http://192.168.86.21:8080/#installer/step8

![855959-20170624201344851-93474785.png][]

比如这不，有个什么Hcat client安装失败。**多刷新，反复这里就是。（因为是网速原因）**

![855959-20170624212759663-318175172.png][]

最后，一定会成功，全部安装完全的！

![855959-20170624223723632-945196563.png][]

**　　如下是我开启全部服务的过程：**

![855959-20170624224137116-1078359445.png][]

![855959-20170624224001616-937069222.png][]

![855959-20170624224057913-793866840.png][]

![855959-20170624224156148-151848785.png][]

![855959-20170624224242007-1023963044.png][]

![855959-20170624231320788-2011850301.png][]

![855959-20170624231402570-1472450737.png][]

![855959-20170624232308398-2144060042.png][]

对于这样的日志报错  
一般是怎么看的，请问

jdbc驱动没有。传个上去

![855959-20170624234302757-863720781.png][]

![855959-20170624234429335-382117762.png][]

手动安装apahce版本的oozie和cdh的oozie，我会下面的这篇博客就是已经解决了

# [Oozie安装时放置Mysql驱动包的总结（网上最全）][Oozie_Mysql] #

现在，这里是ambari里安装oozie，一直不知道怎么放mysql-con\*\*驱动jar包，

以后你有时间解决下！！！

其实，很简单

[root@ambigdata1 ~]# yum install mysql-connector-java
    Is this ok [y/N]: y

即可

心得：一般在公司里，是，开启了服务，就不需再关闭。

只有自己的电脑，比如搭建了4个虚拟机这样的来，关闭再开启。

好啦，写这篇博客是为了帮助入门的博友们，同时，也是，为了最近要在自己实验室，搭建真实物理机器的多节点大数据集群!

（4）运行MapReduce程序

1）上传文件到HDFS

执行命令：cd /usr/hdp/current/hadoop-client/bin这个目录下创建vi wc.txt这个文件,然后输入相应的内容

执行命令：**sudo -u hdfs hadoop fs -put wc.txt /hdpCluster**

文件上传成功之后就可以执行命令运行mapreduce程序了，运行如下命令

yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapeduce-examples-2.7.1.2.4.0-169.jar   wordcount   /hdpCluster/wc.txt   /output

可能会报权限的问题，那么我们只需要修改一下对应的权限即可，例如使用下面的方法。

sudo  -u  hdfs  hadoop fs -ls /

sudo -u  hdfs  hadoop  fs  -chown -R  hadoop:hadoop   /

sudo  -u  hdfs  hadoop  fs  -ls  /

Mapreduce执行成功之后通过如下命令查看程序运行结果

sudo  -u  hdfs  hadoop  fs -cat  /output/part\*

那么到此为止我们的ambari安装部署HA分布式集群就给大家演示完毕了!

作者：[大数据和人工智能躺过的坑][Link 1]  
出处：[http://www.cnblogs.com/zlslch/][Link 1]

[https_www.cnblogs.com_zlslch_p_6629251.html]: https://www.cnblogs.com/zlslch/p/6629251.html
[https_blog.csdn.net_weixin_34128534_article_details_86885399]: https://blog.csdn.net/weixin_34128534/article/details/86885399
[https_blog.csdn.net_fengfengchen95_article_details_93326440]: https://blog.csdn.net/fengfengchen95/article/details/93326440
[https_blog.csdn.net_Y_C_S_M_article_details_90731320]: https://blog.csdn.net/Y_C_S_M/article/details/90731320
[https_blog.csdn.net_gallop_liu_article_details_84462744]: https://blog.csdn.net/gallop_liu/article/details/84462744
[https_blog.csdn.net_qq_26442553_article_details_88845920]: https://blog.csdn.net/qq_26442553/article/details/88845920
[Ambari_Ambari_CentOS6.5]: https://www.cnblogs.com/zlslch/p/6629235.html
[https_blog.csdn.net_zhangxiongcolin_article_details_81569710https_blog.csdn.net_zhangxiongcolin_article_details_81569710]: https://blog.csdn.net/zhangxiongcolin/article/details/81569710
[https_blog.csdn.net_balabalayi_article_details_64920580]: https://blog.csdn.net/balabalayi/article/details/64920580
[Ambari]: http://www.cnblogs.com/zlslch/p/6629249.html
[855959-20170401094416789-673519060.png]: /images/20211211/86548f6a43b140e18ca5a35c8ef0219f.png
[855959-20170401094429477-1484111006.png]: /images/20211211/3adc770b6aa64aa9a985b953969a1e6b.png
[855959-20170406143824957-1544134269.png]: /images/20211211/159ea59243f3490c834c6ef673f6924f.png
[855959-20170331153738117-996849019.png]: /images/20211211/e025cb2a07624af2b644f88bc44fea8c.png
[copycode.gif]: /images/20211211/59d47d85f8f245238089f2ddcfe1eac0.png
[855959-20170406144225378-1032945336.png]: /images/20211211/e20bc7f9419d4c2da2c5f714184ffa0c.png
[855959-20170406144330222-237959548.png]: /images/20211211/e21abd520df84333a0a4ad11a6b84d39.png
[855959-20170406144355878-775689214.png]: /images/20211211/0dd4b017bfc747398911bf23ac21d81a.png
[855959-20170406144955832-503146790.png]: /images/20211211/a24c80bebb8f4338ac50d7ec2fc038c5.png
[855959-20170406151608675-570629403.png]: /images/20211211/a936398f559a4de69dda3da63c954c1d.png
[855959-20170624185434398-1301979805.png]: /images/20211211/10d05671adb14fd99dfbb77b570fec6e.png
[855959-20170624185446898-1245544413.png]: /images/20211211/4463770f2b77476f8086dcbf7dbe4fc1.png
[855959-20170624191102195-535226621.png]: /images/20211211/38418f699b1544cd9a4c151cbcb63f99.png
[Hadoop_4]: http://blog.csdn.net/login_sonata/article/details/52740618
[855959-20170624191511007-1473439929.png]: /images/20211211/bbc47e34b80f4afc92cf9597595b443d.png
[855959-20170624192546148-596706183.png]: /images/20211211/496611846cfc4b41aa1df7add16e3ce2.png
[855959-20170624192443976-1454147636.png]: /images/20211211/ec60fbd1c0894b9aa29d3287cb48dabe.png
[855959-20170624192623773-166455080.png]: /images/20211211/71e081c334cd45989d3710173392b559.png
[855959-20170624210550023-137892858.png]: /images/20211211/3b6fed3c172c423b8929d41047add4f9.png
[855959-20170624193609101-1153858275.png]: /images/20211211/8974ed2e52a940d2945e3132d4f500c9.png
[855959-20170624193625179-707315895.png]: /images/20211211/d86c0f3ba39b4e7e80ccd7af8fb0a116.png
[855959-20170624193831991-342559344.png]: /images/20211211/8ce3cd22b2e5419a80bd3e37ab6c3547.png
[855959-20170624194056991-1515496111.png]: /images/20211211/03fdae983f7c4e5a8f07cd70485a65d3.png
[855959-20170624194319413-1156697148.png]: /images/20211211/ee89edf23cee4c6ba855d11b9511371f.png
[855959-20170624194435101-114332248.png]: /images/20211211/c882b1e0427647e99166f63d703593bd.png
[855959-20170624194536632-1149224073.png]: /images/20211211/d722b1cc5cea4f9da38ab4c6a6f8441d.png
[855959-20170624194701429-2092236375.png]: /images/20211211/081a709b032443459c4ae87773ee1a96.png
[855959-20170624194737491-89831711.png]: /images/20211211/3b043b4ef551435ba437bef156099a75.png
[855959-20170624195322366-2114363748.png]: /images/20211211/3abd92232f9b45288232a1b1e3ad2396.png
[855959-20170624195856913-529751860.png]: /images/20211211/ec444552a625400f8e9d401267fb638a.png
[855959-20170624195356570-1353860046.png]: /images/20211211/100e90bf40f342db9fb1fc8d277d3201.png
[855959-20170624195525054-927288788.png]: /images/20211211/f52181903ae749d3867d5a84282269d2.png
[855959-20170624195837210-127415465.png]: /images/20211211/3288d687bd7246299efc045cc8b0f5ae.png
[855959-20170624195934741-835834781.png]: /images/20211211/79c833fe576e4203be5f5688ca3a5f72.png
[855959-20170624200433601-346725777.png]: /images/20211211/0275edd3cbdf4f8b8911bb82d12fa4b3.png
[855959-20170624200616351-801134887.png]: /images/20211211/076d39e856ed4e5ca5a9fa111dea5bbc.png
[855959-20170624200713382-329410294.png]: /images/20211211/be67c6abecdf4852b12b566c55e3a603.png
[855959-20170624201137913-660326268.png]: /images/20211211/33db8db4235148afa4a784969af1ffc2.png
[855959-20170624201200023-141266590.png]: /images/20211211/ef9eed3ce0f14004998448c59a6bf37b.png
[855959-20170624201255085-2123191110.png]: /images/20211211/9577c869dc664d58999b324047b38dea.png
[855959-20170624201317976-773664829.png]: /images/20211211/7a68f8a62be74b0581a30373e7eca107.png
[855959-20170624201330398-4638739.png]: /images/20211211/f620482da83d48af9df06ebb31c3eece.png
[855959-20170624201344851-93474785.png]: /images/20211211/4a6289d742a44dbf979a6fecc4612828.png
[855959-20170624212759663-318175172.png]: /images/20211211/448c1b66429b4e2bbba2d29a52691032.png
[855959-20170624223723632-945196563.png]: /images/20211211/8a7068e66c244c0aad5d73ef4d4007f0.png
[855959-20170624224137116-1078359445.png]: /images/20211211/86bc3988f9e143b889f9c6cb2cf0a1ea.png
[855959-20170624224001616-937069222.png]: /images/20211211/30f1ee3486104e9f9999865721c38dc8.png
[855959-20170624224057913-793866840.png]: /images/20211211/175d9407e7fd457982a58fadf08ab9da.png
[855959-20170624224156148-151848785.png]: /images/20211211/ff8988c4192a459bb119131e4baa2d36.png
[855959-20170624224242007-1023963044.png]: /images/20211211/297860ee1e1840338e72ca4f8a558f2e.png
[855959-20170624231320788-2011850301.png]: /images/20211211/55d9d56ee2974dd4afc8abba30882c3d.png
[855959-20170624231402570-1472450737.png]: /images/20211211/13654d39c6d6499b8926f7a524bad103.png
[855959-20170624232308398-2144060042.png]: /images/20211211/ba20a688285945078bab96925003f4e4.png
[855959-20170624234302757-863720781.png]: /images/20211211/9b63042632b44c7cbd765c2416661eec.png
[855959-20170624234429335-382117762.png]: /images/20211211/ee6496a44c10479ca26545435d279440.png
[Oozie_Mysql]: http://www.cnblogs.com/zlslch/p/6831024.html
[Link 1]: http://www.cnblogs.com/zlslch/