HDFS NameNode DataNode 多目录配置

浅浅的花香味﹌ 2022-02-24 06:57 198阅读 0赞

现在目录结构  
![format_png][]

## NameNode ##

NameNode 的本地目录可以配置成多个，且每个目录存放内容相同，增加了可靠性。

### 配置 hdfs-site.xml ###

<property>
        <name>dfs.namenode.name.dir</name> 
        <value>file:///${hadoop.tmp.dir}/dfs/name1,file:///${hadoop.tmp.dir}/dfs/name2</value> 
    </property>

分发到其他节点。

### 停止集群 ###

[root@hadoop103 hadoop]# stop-yarn.sh 
    stopping yarn daemons
    stopping resourcemanager
    hadoop102: stopping nodemanager
    hadoop104: stopping nodemanager
    hadoop103: stopping nodemanager
    no proxyserver to stop
    [root@hadoop102 dfs]# stop-dfs.sh 
    Stopping namenodes on [hadoop102]
    hadoop102: stopping namenode
    hadoop103: stopping datanode
    hadoop104: stopping datanode
    hadoop102: stopping datanode
    hadoop105: ssh: connect to host hadoop105 port 22: No route to host
    Stopping secondary namenodes [hadoop104]
    hadoop104: stopping secondarynamenode

### 删除 data 和 logs 中所有数据。 ###

[root@hadoop102 hadoop-2.7.2]# rm -rf data/ logs/ [root@hadoop103 hadoop-2.7.2]# rm -rf data/ logs/ [root@hadoop104 hadoop-2.7.2]# rm -rf data/ logs/

### 格式化集群并启动 ###

格式化

[root@hadoop102 hadoop-2.7.2]# hdfs namenode -format

启动

[root@hadoop102 hadoop-2.7.2]# start-dfs.sh 
    Starting namenodes on [hadoop102]
    hadoop102: starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-namenode-hadoop102.out
    hadoop102: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop102.out
    hadoop103: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop103.out
    hadoop104: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop104.out
    Starting secondary namenodes [hadoop104]
    hadoop104: starting secondarynamenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-hadoop104.out

### 查看 ###

[root@hadoop102 dfs]# pwd
    /opt/module/hadoop-2.7.2/data/tmp/dfs
    [root@hadoop102 dfs]# ll
    总用量 12
    drwx------. 3 root root 4096 4月   3 13:21 data
    drwxr-xr-x. 3 root root 4096 4月   3 13:21 name1
    drwxr-xr-x. 3 root root 4096 4月   3 13:21 name2
    [root@hadoop102 dfs]#

![15542689862527][]

## Datanode ##

datanode 配置成多个目录，每个目录存储的数据不一样。即:数据不是副本。

### 配置 hdfs-site.xml ###

<property> 
        <name>dfs.datanode.data.dir</name>
        <value>file:///${hadoop.tmp.dir}/dfs/data1,file:///${hadoop.tmp.dir}/dfs/data2</value>
    </property>

分发到各节点。

### 停止集群 ###

[root@hadoop102 dfs]# stop-dfs.sh
    Stopping namenodes on [hadoop102]
    hadoop102: stopping namenode
    hadoop102: stopping datanode
    hadoop103: stopping datanode
    hadoop104: stopping datanode
    Stopping secondary namenodes [hadoop104]
    hadoop104: stopping secondarynamenode

### 删除 data 和 logs 中所有数据。 ###

[root@hadoop102 hadoop-2.7.2]# rm -rf data/ logs/ [root@hadoop103 hadoop-2.7.2]# rm -rf data/ logs/ [root@hadoop104 hadoop-2.7.2]# rm -rf data/ logs/

### 格式化集群并启动 ###

格式化

[root@hadoop102 hadoop-2.7.2]# hdfs namenode -format

启动

[root@hadoop102 hadoop-2.7.2]# start-dfs.sh 
    Starting namenodes on [hadoop102]
    hadoop102: starting namenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-namenode-hadoop102.out
    hadoop103: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop103.out
    hadoop102: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop102.out
    hadoop104: starting datanode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-datanode-hadoop104.out
    Starting secondary namenodes [hadoop104]
    hadoop104: starting secondarynamenode, logging to /opt/module/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-hadoop104.out

### 查看 ###

[root@hadoop102 dfs]# pwd
    /opt/module/hadoop-2.7.2/data/tmp/dfs
    [root@hadoop102 dfs]# ll
    总用量 16
    drwx------. 3 root root 4096 4月   3 13:31 data1
    drwx------. 3 root root 4096 4月   3 13:31 data2
    drwxr-xr-x. 3 root root 4096 4月   3 13:31 name1
    drwxr-xr-x. 3 root root 4096 4月   3 13:31 name2

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-C2tJlOS0-1595569036879)(https://mweb.yljphp.com/2019-04-03-15542695530213.jpg)\]

![format_png 1][]

## Web UI ##

\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-erB21CXb-1595569036894)(https://mweb.yljphp.com/2019-04-03-15542697700758.jpg)\]

[format_png]: /images/20220224/52d9cf1a52244dea8aea661a59d0b346.png
[15542689862527]: /images/20220224/837936e882154b08aedb835ac7cacb1f.png
[format_png 1]: /images/20220224/d72bf42a3d7845a0b36db1611790372f.png