hadoop伪分布式集群搭建（超详细）

╰+攻爆jí腚メ 2024-04-03 09:45 11阅读 0赞

需要java安装包和hadoop安装包的可以选择这里下载

链接：https://pan.baidu.com/s/1sQQ-uiwmJvYFPgpVsftL6Q?

提取码：aubt

**第一步，安装centos7并检查网络环境**

为虚拟机导入centos7环境，centos7的安装这里不做赘述，直接上干货

注意VLAN设置： 虚拟机设置为NAT模式

![1ce9c20324354a74b390d5c5a59937d3.png][]

查看linux网络ip地址；

![977301fa2a014c9abf14f0f3a87839e2.png][]

查看网络连接 ping [www.baidu.com][]

![d4795e854736405688d8f1aaf324774e.png][]

如果Ping不通

打开网卡编辑

![167f2f14e3fd47b494c7df4d858a9dc2.png][]

打开后 修改为

![2ec3e47cb9714d23b133c2df1ec2d0c3.png][]![2e5c55abc4ae418280405ed76d7374a3.png][]

编辑完成后保存退出。

然后重启网络服务 service network restart

![4fcc3aae30e343a1af68c0a95cef2714.png][]

修改名字hostname 修改 vim /etc/hostname

![0f2ef9ed11b74f938264092f5766f465.png][]

重启计算机 reboot

![03e4d9485e16449da862af779c2d373b.png][]

**第三步,上传java,hadoop安装包并完成环境配置**

安装java的jdk

上传对应的安装包到虚拟机中,并解压

在虚拟机node1中新建两个目录software(安装包文件)和server(解压文件)

mkdir -p /develop/software

mkdir -p /develop/server

![bae22037479c4722b1b273999ed18789.png][]

进入并导入安装包（我这里使用的xsheel传输本地文件）

![fe350557273e40ec92372e57e584f0a0.png][]

查看java版本并完成卸载

![6d9e8925f86546a7bb357e38c8609d22.png][]

检索java环境 rpm -qa | grep jdk 或者 rpm -qa | grep java （一样的）

![84140708fd874e10a4c69d6f3f933660.png][]

![6892747921f749db9260d97293c32042.png][]

卸载java环境 yum remove java.jdk

卸载完成后剩余

![e276519e38fc4183bcd824cc24723344.png][]

安装新的java

解压jdk的tar包到/develop/server目录

tar -zxvf jdk-8u241-linux-x64.tar.gz -C /develop/server/（后面的为新建的文件夹）

![07702bb9acdd4123b9da6a4e90c0eddb.png][]

解压hadoop安装包到/develop/server

tar -zxvf hadoop-2.7.5.tar.gz -C /develop/server/

![d921e05cb3b14428ae6110604b44c72d.png][]

查看java安装文件

![6d5a8fbbb0644e4eaf79074c7631559e.png][]

修改文件夹root权限 chown -R root (rwx)(读写访问) 地址

![ffb7c6920d83436dbf0355ebdcc5c63f.png][]

修改文件名 mv 原文件名 修改文件名 (便于环境的配置)

![e5b51c49f6c24bb6a87806803927e0c2.png][]

进入jdk目录，复制jdk的地址

![fd57c5a6844442c599cfa7c6ca6bf227.png][]

配置环境变量

vim /etc/profile

![34caf35976dc4e8fa0ac365e651c5eda.png][]

![caedcbbfad074f1590626c7c5c1495e0.png][]

刷新配置文件

source /etc/profile

![8f9c870d2437420aa486adb4069002f2.png][]

**第四步**

修改hadoop配置文件

设置namenode作为java程序的入口

进入文件 core-site.xml

![eb16cca7bd024900add3860077dc96aa.png][]

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

<name>hadoop.temp.dir</name>

<value>/develop/server/hadoop/temp</value>

</property>

</configuration>

![63fdeca1e80c4a16a5ede8240a2788b1.png][]

配置hadoop的java路径

进入文件 vim Hadoop-env.sh

![9ef0cc48801d47cdbe77dbe1c855fddd.png][]

修改java路径

![80f5f742247d47e58a1487e4d8c13f30.png][]

修改hdfs-site.xml文件 这里只有一个节点 replication设置为1

![2e7b131379df4a4f86272db592c99ac0.png][]

<name>dfs.replication</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>file:/develop/server/hadoop/temp/name</value>

</property>

<name>dfs.namenode.data.dir</name>

<value>file:/develop/server/hadoop/temp/data</value>

</property>

![be770ca033b94497bcd817bce240430f.png][]

修改 vim mapred-site.xml.template 文件

![704a9b64a572428788024e5219dde1ab.png][]

![8fe98b817c134067bd79d26dc88ce461.png][]

修改vim yarn-site.xml 文件

![62bb21a8c5504f8ab398393cc02dbdd3.png][]

<name>yarn.resourcemanager.hostname</name>

<value>master</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce\_shuffle</value>

</property>

<name>yarn.resourcemanager.webapp.address</name>

<value>localhost:9000</value>

</property>

![63fac26f5a1745138d9144c4ea555e4f.png][]

切换到hadoop的etc/hadopp目录，修改hadoop-env.sh

![b1387393e11d477886e5474bfc2299c9.png][]

**最后一步**

启动和停止

进入hadoop的bin文件下 cd bin

输入：： hdfs namenode -format

![ebc904e5eeba449b8091cca30e780e6d.png][]

得到

![ecd916c7b28c4bdbaebe30e439bf9c45.png][]

代表格式化成功

**启动hdfs和yarn**

进入 sbin 文件夹 cd sbin

启动start-all.sh

![052721e2558740639d7f8a714545928b.png][]

得到

![81497208b7ce46ccb09a9ace9ee0b5fe.png][]

使用jsp查看

![7ced7d67ef3440a69a121800bdc0945e.png][]

停止

![62e343b57684413283b699d385ddfcdd.png][]

得到

![429a895df3e440efa5b728fc2881ff6c.png][]

使用jps查看

![4c6d98d5f80742278422a2f57c444039.png][]

在启动的情况下访问hadoop

浏览器中输入[http://localhost:50070][http_localhost_50070] 访问hdfs 访问成功

![feaafbf67b1d4607a3355724a02b485e.png][]

![df325dfc280b463ab07898a760b0fd8d.png][]

[1ce9c20324354a74b390d5c5a59937d3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/a062af1c0d1f4f02917b38d8f26a4a08.png
[977301fa2a014c9abf14f0f3a87839e2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/613bd3c15dcc44578314e3bc1dbebc44.png
[www.baidu.com]: http://www.baidu.com
[d4795e854736405688d8f1aaf324774e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/4b4971e69bf4415ba6484b889fc110ae.png
[167f2f14e3fd47b494c7df4d858a9dc2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/4654017673e249619150c1e0a8ec3acd.png
[2ec3e47cb9714d23b133c2df1ec2d0c3.png]: https://img-blog.csdnimg.cn/2ec3e47cb9714d23b133c2df1ec2d0c3.png
[2e5c55abc4ae418280405ed76d7374a3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/c915fd512a7a45928f7b7538e6de924f.png
[4fcc3aae30e343a1af68c0a95cef2714.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8766150b04a54f61a2d0ea239f56252a.png
[0f2ef9ed11b74f938264092f5766f465.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8461879d6af048548597025b25b79f5f.png
[03e4d9485e16449da862af779c2d373b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/6f38f324880b49fe9fb7821f16073ed7.png
[bae22037479c4722b1b273999ed18789.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/5591fa7c02ab44afb41c2f893b6d55cf.png
[fe350557273e40ec92372e57e584f0a0.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/7544cc48ef75418a93d5f33497cf0661.png
[6d9e8925f86546a7bb357e38c8609d22.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/13b8615b41ff499fbf7077a02aa55d10.png
[84140708fd874e10a4c69d6f3f933660.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/a6ae89f6e015463bb2ebf9b032023867.png
[6892747921f749db9260d97293c32042.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/84094d74d6804bd8b8000805edad4ea1.png
[e276519e38fc4183bcd824cc24723344.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/57f8ddab21c64b2693c51dd424154b33.png
[07702bb9acdd4123b9da6a4e90c0eddb.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/b8368a67530c4c97bac87b65a0f184de.png
[d921e05cb3b14428ae6110604b44c72d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/0065c6916c5847569975f3ba15577fe0.png
[6d5a8fbbb0644e4eaf79074c7631559e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/b8fea7e8f6df41a186437794d72fcdf2.png
[ffb7c6920d83436dbf0355ebdcc5c63f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/02b6bbc5c8364aa488f63072a95b315b.png
[e5b51c49f6c24bb6a87806803927e0c2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/73d99072a3c347e59fb46ea6a7158f29.png
[fd57c5a6844442c599cfa7c6ca6bf227.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/453ae1fe98ff4873ab004ff32d85043a.png
[34caf35976dc4e8fa0ac365e651c5eda.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/48ca101f11564eaba610479da0330ad5.png
[caedcbbfad074f1590626c7c5c1495e0.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/3a4c77208dbd46e5a08dce3002c8d0bd.png
[8f9c870d2437420aa486adb4069002f2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/b7d614925683413db3b3844267741972.png
[eb16cca7bd024900add3860077dc96aa.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/0b240e632ee34f99b8892816b307f396.png
[63fdeca1e80c4a16a5ede8240a2788b1.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/37977419dae24478932eaf69a303a915.png
[9ef0cc48801d47cdbe77dbe1c855fddd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/c600c192622a48a1b1ae6f6d6568d629.png
[80f5f742247d47e58a1487e4d8c13f30.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8941e50d601b41eaaa7073b05e1d266d.png
[2e7b131379df4a4f86272db592c99ac0.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8facdb1c506344f19555c1c29635feb4.png
[be770ca033b94497bcd817bce240430f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2fb735f0c5ce404d80a65d768097249e.png
[704a9b64a572428788024e5219dde1ab.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/233aa77b71784992a0df0a6461c20bad.png
[8fe98b817c134067bd79d26dc88ce461.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/d788e567e2544077b1aa1f7354f76d8d.png
[62bb21a8c5504f8ab398393cc02dbdd3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8e8f386fb38e4db085b44283dc346614.png
[63fac26f5a1745138d9144c4ea555e4f.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/7b37db3c3cc742b3965013c9341ce1f5.png
[b1387393e11d477886e5474bfc2299c9.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/8f0658215ace4687ab774394cda5a4d5.png
[ebc904e5eeba449b8091cca30e780e6d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/01bf319b2ff542e3b62efee0ab7940ee.png
[ecd916c7b28c4bdbaebe30e439bf9c45.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/ca9ee9a6e42d413089f62dd530287867.png
[052721e2558740639d7f8a714545928b.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/5e61272b314e43ab84aafc3402f1e435.png
[81497208b7ce46ccb09a9ace9ee0b5fe.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/7800dbb9b8ca417cb59295fe92d905e5.png
[7ced7d67ef3440a69a121800bdc0945e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/717e310eb667449bb5cf64f3e21ae923.png
[62e343b57684413283b699d385ddfcdd.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/dfb120cd82164128974581d02095f4b5.png
[429a895df3e440efa5b728fc2881ff6c.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/2155e44983294db0b349eca9d20225ba.png
[4c6d98d5f80742278422a2f57c444039.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/afe25202f4db482fbf5784b9fdcd3637.png
[http_localhost_50070]: http://localhost:50070
[feaafbf67b1d4607a3355724a02b485e.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/069f3957cdc648709b70877f04e43040.png
[df325dfc280b463ab07898a760b0fd8d.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/03/d3b161d95a4a47aea58bfe9fac187387.png