hadoop升级之fsck命令迎战miss block警告

落日映苍穹つ 2022-04-13 11:16 561阅读 0赞

\[size=large\]继上次升级hadoop完毕后，集群启动正常，但是在访问Namenode的50070的界面上，发现了如下截图的警告信息： \[/size\]  
  
\[img\]http://dl2.iteye.com/upload/attachment/0099/3811/ef2a4650-2939-3697-a7b3-c300c9a9364d.jpg\[/img\]  
\[size=large\]如上异常，是什么意思呢？看了下官方的FAQ,大致意思就是，有几个块的数据，在现有的DataNode节点上，没有一个存储的，但是在NameNode的元数据里却存在。  
  
怎么解决这个问题呢，下面介绍一个hadoop的健康监测命令fsck，  
fsck工具来检验HDFS中的文件是否正常可用。这个工具可以检测文件块是否在DataNode中丢失，是否低于或高于文件副本。 fack命令用法如下：  
注：此处必须是启动hadoop hdfs的账号才有权查看  
Usage: DFSck <path> \[-list-corruptfileblocks | \[-move | -delete | -openforwrite\] \[-files \[-blocks \[-locations | -racks\]\]\]\]  
<path> 检查的起始目录  
\-move 将损坏的文件移到到/lost+found  
\-delete 删除损坏的文件  
\-files 打印出所有被检查的文件  
\-openforwrite 打印出正在写的文件  
\-list-corruptfileblocks print out list of missing blocks and files they belong to  
\-blocks 打印出block报告  
\-locations 打印出每个block的位置  
\-racks 打印出data-node的网络拓扑结构  
默认情况下，fsck会忽略正在写的文件，使用-openforwrite可以汇报这种文件  
\[/size\]  
\[b\]\[color=green\]\[size=large\]下面是是一个运行的命令例子：\[/size\]\[/color\]\[/b\]

[search@fse01 hadoop]$ bin/hadoop fsck / -files
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    
    Connecting to namenode via http://hadoop1:50070
    FSCK started by search (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 21 15:10:06 CST 2014
    / <dir>
    /tmp <dir>
    /tmp/hadoop-search <dir>
    /tmp/hadoop-search/mapred <dir>
    /tmp/hadoop-search/mapred/staging <dir>
    /tmp/hadoop-search/mapred/staging/search <dir>
    /tmp/hadoop-search/mapred/staging/search/.staging <dir>
    /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0001 <dir>
    /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006 <dir>
    /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar 13393150 bytes, 1 block(s): 
    /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-2026656432152987415
     MISSING 1 blocks of total size 13393150 B
    /tmp/hadoop-search/mapred/staging/zhouming <dir>
    /tmp/hadoop-search/mapred/staging/zhouming/.staging <dir>
    /tmp/hadoop-search/mapred/staging/zhouming/.staging/job_201403111456_0028 <dir>
    /tmp/hadoop-search/mapred/system <dir>
    /tmp/hadoop-search/mapred/system/jobtracker.info 4 bytes, 1 block(s): 
    /tmp/hadoop-search/mapred/system/jobtracker.info: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-7234718182174847600
     MISSING 1 blocks of total size 4 B
    /tmp/hadoop-yarn <dir>
    /tmp/hadoop-yarn/staging <dir>
    /tmp/hadoop-yarn/staging/history <dir>
    /tmp/hadoop-yarn/staging/history/done_intermediate <dir>
    /tmp/hadoop-yarn/staging/history/done_intermediate/search <dir>
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001-1405911635075-search-random%2Dwriter-1405912101075-40-0-SUCCEEDED-default.jhist 340102 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001.summary 342 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0001_conf.xml 79756 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003-1405921787298-search-com.dhgate.search.fse.proinfo.dataload.NewLoadData-1405921809468-1-1-SUCCEEDED-default.jhist 32989 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003.summary 390 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0003_conf.xml 83089 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004-1405922486108-search-Rebuild+Index%3Afsesearch%2D%3Ecollection1%2D%3Eshard1-1405922501203-1-0-SUCCEEDED-default.jhist 20173 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004.summary 369 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/history/done_intermediate/search/job_1405911270768_0004_conf.xml 86592 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/search <dir>
    /tmp/hadoop-yarn/staging/search/.staging <dir>
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002 <dir>
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.jar 30472539 bytes, 1 block(s):  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742242_1099511720037. Target Replicas is 10 but found 4 replica(s).
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.split 116 bytes, 1 block(s):  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742243_1099511720038. Target Replicas is 10 but found 4 replica(s).
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.splitmetainfo 31 bytes, 1 block(s):  OK
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.xml 72772 bytes, 1 block(s):  OK
    /user <dir>
    /user/search <dir>
    /user/search/fse <dir>
    /user/search/fse/kms <dir>
    /user/search/fse/kms/in <dir>
    /user/search/fse/kms/in/0 3 bytes, 1 block(s):  OK
    /user/search/fse/kms/index <dir>
    /user/search/fse/kms/index/index_shard_00_5e62b650 <dir>
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0.fdt 11069 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0.fdx 47 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0.fnm 992 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0.nvd 683 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0.nvm 68 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0.si 376 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.doc 50029 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.pos 101757 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.tim 62843 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/_0_Lucene41_0.tip 2319 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/segments.gen 20 bytes, 1 block(s):  OK
    /user/search/fse/kms/index/index_shard_00_5e62b650/segments_1 69 bytes, 1 block(s):  OK
    /user/search/fse/kms/out <dir>
    /user/search/fse/kms/out/0 <dir>
    /user/search/fse/kms/out/0/00 425334 bytes, 1 block(s):  OK
    /user/search/fse/proinfo <dir>
    /user/search/fse/proinfo/schema.xml 32194 bytes, 1 block(s):  OK
    /user/search/rand <dir>
    /user/search/rand/_SUCCESS 0 bytes, 0 block(s):  OK
    /user/search/rand/part-m-00000 1077280794 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00001 1077294346 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00002 1077276960 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00003 1077280905 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00004 1077288199 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00005 1077290933 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00006 1077270249 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00007 1077290630 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00008 1077279315 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00009 1077294441 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00010 1077288193 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00011 1077299978 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00012 1077274992 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00013 1077295278 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00014 1077280338 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00015 1077285350 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00016 1077276847 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00017 1077285962 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00018 1077280969 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00019 1077285691 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00020 1077292174 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00021 1077292141 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00022 1077285419 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00023 1077285751 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00024 1077282762 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00025 1077282468 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00026 1077283184 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00027 1077281056 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00028 1077292387 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00029 1077283341 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00030 1077288015 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00031 1077276923 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00032 1077296469 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00033 1077288224 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00034 1077278284 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00035 1077297217 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00036 1077289846 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00037 1077284545 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00038 1077290084 bytes, 9 block(s):  OK
    /user/search/rand/part-m-00039 1077282121 bytes, 9 block(s):  OK
    Status: CORRUPT
     Total size:    43136702998 B
     Total dirs:    32
     Total files:   71
     Total symlinks:                0
     Total blocks (validated):      390 (avg. block size 110606930 B)
      ********************************
      CORRUPT FILES:        2
      MISSING BLOCKS:       2
      MISSING SIZE:         13393154 B
      CORRUPT BLOCKS:       2
      ********************************
     Minimally replicated blocks:   388 (99.48718 %)
     Over-replicated blocks:        0 (0.0 %)
     Under-replicated blocks:       2 (0.51282054 %)
     Mis-replicated blocks:         0 (0.0 %)
     Default replication factor:    3
     Average block replication:     2.9897435
     Corrupt blocks:                2
     Missing replicas:              12 (1.0075567 %)
     Number of data-nodes:          4
     Number of racks:               1
    FSCK ended at Mon Jul 21 15:10:06 CST 2014 in 48 milliseconds
    
    
    The filesystem under path '/' is CORRUPT

\[color=olive\]\[size=large\]通过上面这个命令我们就可以监测出文件的存储状况，监测出有无效的或者缺失的block，下面我们就可以执行-delete命令，来清楚无效的信息块。  
\[/size\]\[/color\]

[search@fse01 hadoop]$ bin/hadoop  fsck /  -delete 
    DEPRECATED: Use of this script to execute hdfs command is deprecated.
    Instead use the hdfs command for it.
    
    Connecting to namenode via http://hadoop1:50070
    FSCK started by search (auth:SIMPLE) from /127.0.0.1 for path / at Mon Jul 21 15:11:39 CST 2014
    .
    /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-2026656432152987415
    
    /tmp/hadoop-search/mapred/staging/search/.staging/job_201312191115_0006/job.jar: MISSING 1 blocks of total size 13393150 B..
    /tmp/hadoop-search/mapred/system/jobtracker.info: CORRUPT blockpool BP-1264683061-172.16.70.21-1405911170103 block blk_-7234718182174847600
    
    /tmp/hadoop-search/mapred/system/jobtracker.info: MISSING 1 blocks of total size 4 B...........
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.jar:  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742242_1099511720037. Target Replicas is 10 but found 4 replica(s).
    .
    /tmp/hadoop-yarn/staging/search/.staging/job_1405911270768_0002/job.split:  Under replicated BP-1264683061-172.16.70.21-1405911170103:blk_1073742243_1099511720038. Target Replicas is 10 but found 4 replica(s).
    ..........................................................Status: CORRUPT
     Total size:    43136702998 B
     Total dirs:    32
     Total files:   71
     Total symlinks:                0
     Total blocks (validated):      390 (avg. block size 110606930 B)
      ********************************
      CORRUPT FILES:        2
      MISSING BLOCKS:       2
      MISSING SIZE:         13393154 B
      CORRUPT BLOCKS:       2
      ********************************
     Minimally replicated blocks:   388 (99.48718 %)
     Over-replicated blocks:        0 (0.0 %)
     Under-replicated blocks:       2 (0.51282054 %)
     Mis-replicated blocks:         0 (0.0 %)
     Default replication factor:    3
     Average block replication:     2.9897435
     Corrupt blocks:                2
     Missing replicas:              12 (1.0075567 %)
     Number of data-nodes:          4
     Number of racks:               1
    FSCK ended at Mon Jul 21 15:11:39 CST 2014 in 45 milliseconds
    
    
    The filesystem under path '/' is CORRUPT

\[size=large\]最后一点，需要注意的是，这个命令在namenode文件信息较大的时候，会比较影响hadoop性能，所以应该慎用，通常可以在集群空闲的时间段，执行一次，查看整体的HDFS副本健康状况！\[/size\]