2.1.6 hadoop体系之离线计算-hdfs分布式文件系统-全分布式安装hadoop2.7.7（3）

系统管理员 2022-12-18 00:45 206阅读 0赞

**目录**

1.下载并解压

2.配置环境变量

3.修改Hadoop配置文件

3.1 hadoop-env.sh

3.2 core-site.xml

3.3 hdfs-site.xml

3.4 mapred-site.xml

3.5 yarn-site.xml

3.6 master

3.7 slaves

4.分发程序

5.node01初始化

6.启动集群

7.查看集群

8.问题

--------------------

# 1.下载并解压 #

下载 Hadoop 安装包，这里我下载的是hadoop-2.7.7.tar.gz

[xiaokang@hadoop ~]$ sudo tar -zxvf hadoop-2.7.7.tar.gz -C /opt/software/

# 2.配置环境变量 #

[xiaokang@hadoop ~]$ sudo vim /etc/profile

在原来jdk基础上更新配置环境变量：

# java
    export JAVA_HOME=/home/java/jdk1.8.0_131
    export JRE_HOME=/home/java/jdk1.8.0_131
    export CLASSPATH=$JRE_HOME/lib/rt.jar:$JRE_HOME/lib/ext
    export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
    
    # hadoop
    export HADOOP_HOME=/opt/software/hadoop-2.7.7

执行 `source` 命令，使得配置的环境变量立即生效：

[xiaokang@hadoop ~]$ source /etc/profile

#  3.修改Hadoop配置文件 #

先创建好所需目录

[xiaokang@hadoop01 hadoop-2.7.7]$ mkdir /opt/software/hadoop-2.7.7/tmp
    [xiaokang@hadoop01 hadoop-2.7.7]$ mkdir -p /opt/software/hadoop-2.7.7/dfs/namenode_data
    [xiaokang@hadoop01 hadoop-2.7.7]$ mkdir -p /opt/software/hadoop-2.7.7/dfs/datanode_data
    [xiaokang@hadoop01 hadoop-2.7.7]$ mkdir -p /opt/software/hadoop-2.7.7/checkpoint/dfs/cname

## 3.1 hadoop-env.sh ##

#25行 export JAVA_HOME
    export JAVA_HOME=/opt/moudle/jdk1.8.0_191
    #33行 export HADOOP_CONF_DIR
    export HADOOP_CONF_DIR=/opt/software/hadoop-2.7.7/etc/hadoop

## 3.2 core-site.xml ##

<configuration>
            <property>
                
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop01:9000</value>
            </property>  
            <property>
                   
                <name>hadoop.tmp.dir</name>
                <value>file:///opt/software/hadoop-2.7.7/tmp</value>
            </property>
            <property>
                
                <name>io.file.buffer.size</name>
                <value>4096</value>
            </property>
    </configuration>

## 3.3 hdfs-site.xml ##

<configuration>
            <property>
               
               <name>dfs.block.size</name>
               <value>134217728</value>
            </property>
            <property>
                
                <name>dfs.replication</name> 
                <value>3</value>
            </property>
            <property>
                 
                <name>fs.checkpoint.dir</name>
                <value>file:///opt/software/hadoop-2.7.7/checkpoint/dfs/cname</value>
            </property>
            <property>
                
                <name>dfs.name.dir</name> 
                <value>file:///opt/software/hadoop-2.7.7/dfs/namenode_data</value>
            </property>
            <property>
                
                <name>dfs.data.dir</name> 
                <value>file:///opt/software/hadoop-2.7.7/dfs/datanode_data</value>
            </property>
            <property>
                
                <name>dfs.namenode.secondary.http-address</name> 
                <value>hadoop02:50090</value>
            </property>
            <property>
                
                <name>dfs.permissions</name> 
                <value>false</value>
            </property>
    </configuration>

## 3.4 mapred-site.xml ##

<configuration>
            <property>  
                
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
            </property>
            <property>
                
                <name>mapreduce.jobhistory.address</name>
                <value>hadoop01:10020</value>
            </property>
            <property>
                
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>hadoop01:19888</value>
            </property>
    </configuration>

## 3.5 yarn-site.xml ##

<configuration>
            <property>
                
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop01</value>
            </property>
            <property>
                <name>yarn.resourcemanager.address</name>
                <value>hadoop01:8032</value>
            </property>
            <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>hadoop01:8088</value>
            </property>
            <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>hadoop01:8030</value>
            </property>
            <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>hadoop01:8031</value>
            </property>
            <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>hadoop01:8033</value>
            </property>
            <property>
                
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
            </property>
            <property>
                
                <name>yarn.log-aggregation-enable</name>
                <value>true</value>
            </property>
            <property>
                
                <name>yarn.log-aggregation.retain-seconds</name>
                <value>604800</value>
            </property>
    </configuration>

## 3.6 master ##

在当前配置文件目录内是不存在master文件的，我们使用**vim**写入内容到master内保存即可

[xiaokang@hadoop01 hadoop]$ vim master
    hadoop01

## 3.7 slaves ##

配置所有从属节点的主机名或 IP 地址，每行一个。所有从属节点上的 `DataNode` 服务和 `NodeManager` 服务都会被启动。

hadoop01
    hadoop02
    hadoop03

# 4.分发程序 #

将 Hadoop 安装包分发到其他两台服务器，分发后建议在这两台服务器上也配置一下 Hadoop 的环境变量。

# 将安装包分发到hadoop002
    [xiaokang@hadoop01 hadoop]$ sudo scp -r /opt/software/hadoop-2.7.7/ xiaokang@hadoop02:/opt/software/
    # 将安装包分发到hadoop003
    [xiaokang@hadoop01 hadoop]$ sudo scp -r /opt/software/hadoop-2.7.7/ xiaokang@hadoop03:/opt/soft

#  5.node01初始化 #

\[xiaokang@hadoop01 ~\]$ hdfs namenode -format

# 6.启动集群 #

在 `hadoop01` 上启动 Hadoop集群。此时 `hadoop02` 和 `hadoop03` 上的相关服务也会被启动：

# 启动dfs服务
    [xiaokang@hadoop01 ~]$ start-dfs.sh
    # 启动yarn服务
    [xiaokang@hadoop01 ~]$ start-yarn.sh
    # 启动任务历史服务器
    [xiaokang@hadoop01 ~]$ mr-jobhistory-daemon.sh start historyserver

相应的关闭：stop和kill -9

# 7.查看集群 #

在每台服务器上使用 `jps` 命令查看服务进程，或直接进入 Web-UI 界面进行查看，端口为 `50070`。可以看到此时有三个可用的 `Datanode`：

[xiaokang@hadoop01 hadoop-2.7.7]$ jps
    13664 ResourceManager
    13250 NameNode
    14101 JobHistoryServer
    14135 Jps
    13387 DataNode
    13773 NodeManager
    
    [xiaokang@hadoop02 hadoop-2.7.7]$ jps
    9793 SecondaryNameNode
    9685 DataNode
    9898 NodeManager
    9930 Jps
    
    [xiaokang@hadoop03 hadoop-2.7.7]$ jps
    9880 DataNode
    10024 NodeManager
    10056 Jps

YARN:[http://192.168.0.111:8088/cluster][http_192.168.0.111_8088_cluster]

HDFS:[http://192.168.0.111:50070/dfshealth.html\#tab-overview][http_192.168.0.111_50070_dfshealth.html_tab-overview]

# 8.问题 #

如果jps显示正确，但是hdfs客户端只显示一个node，注意查看自己的/etc/hosts文件，特别是localhost位置，特别是node01

![20201024160803639.png][]

[http_192.168.0.111_8088_cluster]: http://192.168.0.111:8088/cluster
[http_192.168.0.111_50070_dfshealth.html_tab-overview]: http://192.168.0.111:50070/dfshealth.html#tab-overview
[20201024160803639.png]: /images/20221122/93cbcf79234d4240abb08249d3cc7b61.png