Incompatible clusterIDs in dfs/dn: namenode clusterID=cluster19; datanode clusterID=hacl-pepstack

忘是亡心i 2022-03-16 08:54 96阅读 0赞

## ** cdh中hdfs的namenode的datenode的cluster Id不一致导致hdfs启动失败** ##

2019-02-26 17:46:40,663 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
    org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /data10/dfs/nn is in an inconsistent state: cluster Id is incompatible with others.
    	at org.apache.hadoop.hdfs.server.common.StorageInfo.setClusterId(StorageInfo.java:166)
    	at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:134)
    	at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:626)
    	at org.apache.hadoop.hdfs.server.namenode.NNStorage.readProperties(NNStorage.java:655)
    	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:336)
    	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1096)
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:780)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:614)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:676)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)

手动修改namenode中的VERSION文件，我的这个文件存在两个，路径分别为/data1/dfs/nn/current和/data10/dfs/nn/current，分别按照以下步骤修改：

cd /data10/dfs/nn/current

vim VERSION

#Tue Feb 26 17:26:34 CST 2019
    namespaceID=689934995
    clusterID=hacl-pepstack-com
    cTime=0
    storageType=NAME_NODE
    blockpoolID=BP-165196685-10.142.149.248-1551173194257
    layoutVersion=-60

将VERSION文件中namespaceID=689934995改为DATANODE日志中报的datanode clusterID = hacl-pepstack-com，然后重新启动namenode就好了。

我的datenode的日志文件路径是/var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-A5-401-NF5280M4-2017-247.BIGDATA.CHINATELECOM.CN.log.out，datenode的报错为：

java.io.IOException: Incompatible clusterIDs in /data11/dfs/dn: namenode clusterID = cluster7; datanode clusterID = hacl-pepstack-com
    	at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:779)
    	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadStorageDirectory(DataStorage.java:302)
    	at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadDataStorage(DataStorage.java:418)
    	at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:397)
    	at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575)
    	at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1523)
    	at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1483)
    	at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
    	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
    	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
    	at java.lang.Thread.run(Thread.java:745)