hdfs一次一直处于安全模式检查与修复

桃扇骨 2022-12-27 09:00 213阅读 0赞

### hdfs一次一直处于安全模式检查与修复 ###

*  查找问题
 *  思考
 *  排查解决
 *  解决方案1
 *  解决方案2
 *  反思

进入cm界面发现两个服务出现了异常

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70]

# 查找问题 #

1.  二话不说，重启  
    本地的服务哈，如果是线上的请跳过…  
    反馈结果：问题依旧
2.  查看相关信息吧  
    我们知道hive依赖hdfs，所以hive的问题有可能是hdfs导致  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 1]  
    两条信息  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 2]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 3]  
    HDFS Canary这个要读写hdfs验证hdfs的可用性。  
    NameNode 当前处于 safe mode。所以是第二个问题导致的前面的验证不通过。所以我们需要排查一下namenode的sofemode

命令行查询一下

hdfs dfsadmin -safemode get

![在这里插入图片描述][20201209220140777.png]

# 思考 #

想必大家都知道hdfs的启动过程  
1.启动服务，datenode向namenode汇报自己的block信息，最初处于安全模式，如果每个最小的block满足设置的阈值，namenode离开安全模式，补充block副本数不足的，进行补充。  
既然我们一直处于安全模式，说明我的block数不对，难道我集群有问题？  
我去看了一下主机状态  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 4]  
这些节点都是正常的，我们还有一个命令,可以验证block是不是正常的

hadoop fsck /

# 排查解决 #

[gugu@slave1 ~]$ hadoop fsck /
    WARNING: Use of this script to execute fsck is deprecated.
    WARNING: Attempting to execute replacement "hdfs fsck" instead.
    Connecting to namenode via http://master.com:9870/fsck?ugi=gugu&path=%2F
    FSCK started by gugu (auth:SIMPLE) from /192.168.2.101 for path / at Wed Dec 09 22:07:50 CST 2020
    /tmp/logs/gugu/logs/application_1607435161095_0001/slave1.com_8041: MISSING 1 blocks of total size 3                                                                                                            7242 B.
    /tmp/logs/gugu/logs/application_1607435161095_0001/slave2.com_8041: CORRUPT blockpool BP-572917739-1                                                                                                            92.168.2.100-1601750079728 block blk_1073745376
    /tmp/logs/gugu/logs/application_1607435161095_0001/slave2.com_8041: CORRUPT 1 blocks of total size 1                                                                                                            2697 B.
    Status: CORRUPT
     Number of data-nodes:  3
     Number of racks:               1
     Total dirs:                    325
     Total symlinks:                0
    
    Replicated Blocks:
     Total size:    4994301 B
     Total files:   76 (Files currently being written: 3)
     Total blocks (validated):      71 (avg. block size 70342 B) (Total open file blocks (not validated)                                                                                                            : 3)
      ********************************
      UNDER MIN REPL'D BLOCKS:      2 (2.8169014 %)
      dfs.namenode.replication.min: 1
      CORRUPT FILES:        2
      MISSING BLOCKS:       1
      MISSING SIZE:         37242 B
      CORRUPT BLOCKS:       1
      CORRUPT SIZE:         12697 B
      ********************************
     Minimally replicated blocks:   69 (97.1831 %)
     Over-replicated blocks:        0 (0.0 %)
     Under-replicated blocks:       0 (0.0 %)
     Mis-replicated blocks:         0 (0.0 %)
     Default replication factor:    1
     Average block replication:     0.97183096
     Missing blocks:                1
     Corrupt blocks:                1
     Missing replicas:              0 (0.0 %)
     Blocks queued for replication: 0
    
    Erasure Coded Block Groups:
     Total size:    0 B
     Total files:   0
     Total block groups (validated):        0
     Minimally erasure-coded block groups:  0
     Over-erasure-coded block groups:       0
     Under-erasure-coded block groups:      0
     Unsatisfactory placement block groups: 0
     Average block group size:      0.0
     Missing block groups:          0
     Corrupt block groups:          0
     Missing internal blocks:       0
     Blocks queued for replication: 0
    FSCK ended at Wed Dec 09 22:07:50 CST 2020 in 16 milliseconds
    The filesystem under path '/' is CORRUPT

我们可以看到

The filesystem under path '/' is CORRUPT

说明我们的根路径是没有通过检查的，我们在上面可以看到健康检查的蛛丝马迹

FSCK started by gugu (auth:SIMPLE) from /192.168.2.101 for path / at Wed Dec 09 22:07:50 CST 2020
    /tmp/logs/gugu/logs/application_1607435161095_0001/slave1.com_8041: MISSING 1 blocks of total size 3                                                                                                            7242 B.
    /tmp/logs/gugu/logs/application_1607435161095_0001/slave2.com_8041: CORRUPT blockpool BP-572917739-1                                                                                                            92.168.2.100-1601750079728 block blk_1073745376
    /tmp/logs/gugu/logs/application_1607435161095_0001/slave2.com_8041: CORRUPT 1 blocks of total size 1                                                                                                            2697 B.

是这几个文件的问题  
一看是我spark日志的文件  
不如，嘿嘿嘿删掉可好（仅供参考）

# 解决方案1 #

文件修复

1.  离开安全模式

hdfs dfsadmin -safemode leave

![在这里插入图片描述][20201209221515877.png]  
2. 尝试修复文件

hdfs debug recoverLease -path /hbase/oldWALs/slave1.com%2C16020%2C1608213311522.meta.1608216920810.meta -retries 2

说明：-path 修复文件路径  
\-retries 尝试次数  
![在这里插入图片描述][20201219094519377.png]  
所有文件修复后再次检查状态并且重新启动

hdfs fsck /

![hdfs检查正常][hdfs]

![集群所有角色都正常][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 5]

# 解决方案2 #

首先我们要离开安全模式，否则我们是无法上传,更新，删除文件的

hdfs dfsadmin -safemode leave

![在这里插入图片描述][20201209221515877.png]  
所以我就去愉快地删除我的文件了

hdfs dfs -rmr /tmp/logs/gugu/logs/application_1607435161095_0001

![在这里插入图片描述][20201209221758111.png]  
重启服务  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 6]  
啊哈怎么还是有异常  
点开让我康康  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 7]  
我去hdfsui康康  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 8]  
ok i know  
删除文件进入了垃圾箱  
so 我们清理下垃圾箱（但是先要离开安全模式），或者我们删除的时候跳过垃圾箱即可

hdfs dfsadmin -safemode leave
    hdfs dfs -expunge

![在这里插入图片描述][20201209223544182.png]  
这次我记住了，清理完重新检查一下  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 9]  
然后重启服务，可以了哈哈哈  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 10]

删除的时候跳过垃圾箱

hdfs dfs -rmr -f -skipTrash /tmp/logs/gugu/logs/application_1607435161095_0001

# 反思 #

hdfs安全模式一直没有正常离开说明我们的文件有损坏，后来我想到我的hdfs副本之前为了节约资源设置成了1，还是恢复成默认值3吧，这样也可以减少出现问题的概率，虽说我的虚拟机已经备份了镜像，但是所有的备份都是最后的保障，我们一般不会动用这个手段

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70]: /images/20221120/ab22989677ef4a57b0dc9f09ed9c0fa3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 1]: /images/20221120/c9d0ea7797cd4312983bf1b769e52c03.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 2]: /images/20221120/962bb89f8263449688cc3b623ac1c401.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 3]: /images/20221120/bb68461c7eec4372ad2367cfde32ced2.png
[20201209220140777.png]: /images/20221120/50b65aed15f449d6950a0b4dab24b534.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 4]: /images/20221120/335a7a87cad54ffa8acc919ab8504557.png
[20201209221515877.png]: /images/20221120/f9cc5854446d4dc3bf8f6e68c2077075.png
[20201219094519377.png]: /images/20221120/ba45ffb7b511413d8617041c67e57d98.png
[hdfs]: /images/20221120/93cbcf5c0c454d55b190799ef86a9440.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 5]: /images/20221120/78a9f9091963487c93bcb7389ce18901.png
[20201209221758111.png]: /images/20221120/d8bf462abaec4d10b97f1f5428cb0436.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 6]: /images/20221120/327fdd8082ae4fe098d4f5beefe5b7cf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 7]: /images/20221120/4b1505fda399444db7a531cd48970de4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 8]: /images/20221120/083c0f23c01b4096a5351dc5005ff9a9.png
[20201209223544182.png]: /images/20221120/f519c3094b0e4cf7946511102b2c053c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 9]: /images/20221120/505d96093bbd42c08a0c922ed4ac8812.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYXpoYWd1_size_16_color_FFFFFF_t_70 10]: /images/20221120/80b8c8f9a88e4cd3a4d091d94191ee28.png