hadoop2.7执行离线备份hbase根目录时报错Mismatch in length

古城微笑少年丶 2022-06-03 08:12 98阅读 0赞

\[hdfs@hmaster-1 root\]$ hadoop distcp hdfs://hdfs-ha/apps/hbase/data hdfs://hdfs-ha/backup/hbase\_bk\_20171222

**出现的错误摘要：**

Caused by: java.io.IOException:Mismatch in length of source:hdfs://hdfs-ha/apps/hbase/data/WALs/core-06/backup/hbase\_bk\_20171222/.distcp.tmp.attempt\_1506497058251\_3878\_m\_000000\_1

at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)

at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)

最大可能的原因是由于文件未关闭而影响了distcp作业执行，验证一下是不是，查看这个文件状态：

\[hdfs@hmaster-1 ~\]$ hdfs fsck hdfs://hdfs-ha/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353

Connecting to namenode via http://hmaster-1.ksc.com:50070/fsck?ugi=hdfs&path=%2Fapps%2Fhbase%2Fdata%2FWALs%2Fcore-06%2C16020%2C1513713156134-splitting%2Fcore-06%252C16020%252C1513713156134.default.1513946044353

FSCK started by hdfs (auth:SIMPLE) from /172.31.0.10 for path /apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353 at Fri Dec 22 22:18:06 CST 2017

.Status: HEALTHY

Total size: 16004696 B

Total dirs: 0

Total files: 1

Total symlinks: 0

Total blocks (validated): 1 (avg. block size 16004696 B)

Minimally replicated blocks: 1 (100.0 %)

Over-replicated blocks: 0 (0.0 %)

Under-replicated blocks: 0 (0.0 %)

Mis-replicated blocks: 0 (0.0 %)

Default replication factor: 3

Average block replication: 3.0

Corrupt blocks: 0

Missing replicas: 0 (0.0 %)

Number of data-nodes: 8

Number of racks: 1

FSCK ended at Fri Dec 22 22:18:06 CST 2017 in 21 milliseconds

The filesystem under path '/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353 ' is HEALTHY

\[hdfs@hmaster-1 ~\]$

发现此文件真的是没关闭状态。那么，解决办法是只要关闭该文件。怎样实现该文件的关闭呢？

Hadoop DFS自带了恢复机制，由于Lease Recovery会触发Block Recovery，当Data Node完成Block Recovery后文件会被关闭。所以这里手动执行recoverLease即可。那么把上面报出来的文件(上文红色字体部分)关闭即可，操作流程：

第一步，关闭此文件之前先拷贝此文件：

\[hdfs@hmaster-1 root\]$ hdfs dfs -cp hdfs://hdfs-ha/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353hdfs://hdfs-ha/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353\_backup

\[hdfs@hmaster-1 root\]$ hdfs debug  recoverLease \-path hdfs://hdfs-ha/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353

recoverLease returned false.

Giving up on recoverLease for hdfs://hdfs-ha/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353 after 1 try.

第二步，执行recoverLease操作：

\[hdfs@hmaster-1 ~\]$ hdfs debug recoverLease -path hdfs://hdfs-ha/apps/hbase/data/WALs/core-06,16020,1513713156134-splitting/core-06%2C16020%2C1513713156134.default.1513946044353

第三步，再次执行hadoop distcp，成功！

\[hdfs@hmaster-1 ~\]$ hadoop distcp hdfs://hdfs-ha/apps/hbase/data hdfs://hdfs-ha/backup/hbase\_bk\_20171222

17/12/22 22:31:12 INFO mapreduce.Job: map 98% reduce 0%

17/12/22 22:32:57 INFO mapreduce.Job: map 99% reduce 0%

17/12/22 22:33:03 INFO mapreduce.Job: map 100% reduce 0%

17/12/22 22:34:39 INFO mapreduce.Job: Job job\_1506497058251\_3881 completed successfully

17/12/22 22:34:40 INFO mapreduce.Job: Counters: 35

File System Counters

FILE: Number of bytes read=0

FILE: Number of bytes written=3363460

FILE: Number of read operations=0

FILE: Number of large read operations=0