VirtualBox + CentOS7 搭建 Hadoop 伪分布式模式

àì夳堔傛蜴生んèń 2023-01-13 01:42 143阅读 0赞

**目录**

一、准备好 CentOS7 的ISO 文件，JDK1.8 安装文件，hadoop-2.7 安装文件

二、安装CentOS7系统

三、CentOS7环境准备

1、关闭防火墙

2、安装 ifconfig 命令用于查看IP

3、将所jdk,hadoop文件传入linux系统

4、解压缩 jdk-8u281-linux-x64.tar.gz

5、配置jdk环境变量 vi /etc/profile 在最末未增加

6、检查jdk安装是否成功

7、免密码ssh设置

四、配置Hadoop

1、解压缩 hadoop-2.7.5.tar.gz

2、配置  hadoop-env.sh

3、配置   core-site.xml

4、配置 hdfs-site.xml

5、将Hadoop加入环境变量

6、格式化一个新的分布式文件系统

7、启动NameNode和DataNode守护进程

8、浏览Web界面以查找NameNode

9、关闭 Hadoop

五、在单节点上的YARN

1、配置 mapred-site.xml

2、配置 yarn-site.xml

3、启动ResourceManager守护程序和NodeManager守护程序

4、浏览Web界面以找到ResourceManager

5、停止守护进程

--------------------

# 一、准备好 CentOS7 的ISO 文件，JDK1.8 安装文件，hadoop-2.7 安装文件 #

链接：https://pan.baidu.com/s/1lmA0Ai9ivPftJgC6DbpN7g   
提取码：iw3g

![20210415111448799.png][]

# 二、安装CentOS7系统 #

没有截图的页面默认配置直接下一步

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 1][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 3][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 4][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 5][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 7][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 8][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 9][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 10][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 11][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 12][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 13][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 14][]

# 三、CentOS7环境准备 #

root 登录系统

## 1、关闭防火墙 ##

查看当前防火墙状态  Active: active (running) 表示开启
    [root@centos7-hadoop sbin]# systemctl status firewalld.service
    关闭防火墙
    [root@centos7-hadoop sbin]# systemctl stop firewalld.service
    查看当前防火墙状态  Active: inactive (dead) 表示关闭
    [root@centos7-hadoop sbin]# systemctl status firewalld.service
    永久关闭防火墙
    [root@centos7-hadoop sbin]# systemctl disable firewalld.service

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 15][]

## 2、安装 ifconfig 命令用于查看IP ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 16][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 17][]![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 18][]

## 3、将所jdk,hadoop文件传入linux系统 ##

jdk目录：/usr/java

hadoop目录：/opt/hadoop

## 4、解压缩 jdk-8u281-linux-x64.tar.gz  ##

tar -zxvf jdk-8u281-linux-x64.tar.gz  到 /usr/java

## 5、配置jdk环境变量 vi /etc/profile 在最末未增加  ##

#java
    export JAVA_HOME=/usr/java/jdk1.8.0_281
    export PATH=$JAVA_HOME/bin:$PATH
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib

## 6、检查jdk安装是否成功 ##

![20210415114812506.png][]

## 7、免密码ssh设置 ##

现在确认能否不输入口令就用ssh登录localhost:  
$ ssh localhost

如果不输入口令就无法用ssh登陆localhost，执行下面的命令：

$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
      $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
      $ chmod 0600 ~/.ssh/authorized_keys

# 四、配置Hadoop #

## 1、解压缩 hadoop-2.7.5.tar.gz  ##

tar -zxvf hadoop-2.7.5.tar.gz   到 /opt/hadoop

进入目录 /opt/hadoop/hadoop-2.7.5/

## 2、配置  hadoop-env.sh ##

vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/hadoop-env.sh

将 export JAVA\_HOME=$\{JAVA\_HOME\}  
改成 export JAVA\_HOME=/usr/java/jdk1.8.0\_281

## 3、配置   core-site.xml ##

vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/core-site.xml

<configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
        </property>
    </configuration>

## 4、配置 hdfs-site.xml ##

vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/hdfs-site.xml

<configuration>
        <property>
            <name>dfs.replication</name>
            <value>1</value>
        </property>
    </configuration>

## 5、将Hadoop加入环境变量 ##

vi ~/.bash\_profile 在文件最后加入

#HADOOP
    export HADOOP_HOME=/opt/hadoop/hadoop-2.7.5
    export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

执行 source ~/.bash\_profile 使环境变量生效

## 6、格式化一个新的分布式文件系统 ##

$ bin/hdfs namenode -format

## 7、启动NameNode和DataNode守护进程 ##

$ sbin/start-dfs.sh

## 8、浏览Web界面以查找NameNode ##

*  NameNode- http://localhost:50070/

## 9、关闭 Hadoop ##

$ sbin/stop-dfs.sh

# 五、在单节点上的YARN #

进入目录 /opt/hadoop/hadoop-2.7.5/

## 1、配置 mapred-site.xml ##

复制模板文件

cp /opt/hadoop/hadoop-2.7.5/etc/hadoop/mapred-site.xml.template /opt/hadoop/hadoop-2.7.5/etc/hadoop/mapred-site.xml

编辑文件 vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/mapred-site.xml

<configuration>
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
    </configuration>

## 2、配置 yarn-site.xml ##

编辑文件 vi /opt/hadoop/hadoop-2.7.5/etc/hadoop/yarn-site.xml

<configuration>
        <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
        </property>
    </configuration>

## 3、启动ResourceManager守护程序和NodeManager守护程序 ##

$ sbin/start-yarn.sh

## 4、浏览Web界面以找到ResourceManager ##

ResourceManager - http://localhost:8088/

## 5、停止守护进程 ##

$ sbin/stop-yarn.sh

[20210415111448799.png]: /images/20221022/2042f7f7c3734141b3df67ed2b6df50b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20210415112308732.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20210415112326438.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 2]: /images/20221022/873d13a7bed64cc6b4f15163d4b76109.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20210415112423400.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20210415112439845.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20210415112457903.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 6]: /images/20221022/0403c92a8c3a46acb663c0c58ffe20a3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20210415112655497.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20210415112708976.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/2021041511272355.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20210415112731339.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 11]: /images/20221022/1d2717e127bf4ce396b40ff1157a2497.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/2021041511281936.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/2021041511282910.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 14]: /images/20221022/9bd8b4451cc54d9a9a94b4fc4067acc2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 15]: /images/20221022/2c89567604024a7cb9f1341c585e8328.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 16]: https://img-blog.csdnimg.cn/20210415113315241.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 17]: https://img-blog.csdnimg.cn/2021041511335137.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d1Y2hlbmxoeQ_size_16_color_FFFFFF_t_70 18]: /images/20221022/6ab74a46f9e54b5ea0d1e4f7e28ab779.png
[20210415114812506.png]: /images/20221022/ef3789e692af4bf3b199bdf434a9b268.png