Hadoop,Hbase,nutch搭建过程遇到的问题

小鱼儿 2021-09-22 09:40 276阅读 0赞

问题如下：在使用Nutch进行抓取的时候，程序卡在Injection urlDir下面不向下走了，查看nutch下面的hadoop.log没有报任何错误。

![SouthEast][]

查看 Hbase 文件logs日志下的Hmaster日志有如下错误

DEBUG org.apache.hadoop.hbase.util.FSUtils: Created version file at hdfs://192.168.14.151:9000/home/alan/hbase/tmp set its version at:7
    2017-11-08 18:20:55,751 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/alan/hbase/tmp/hbase.version could only be replicated to 0 nodes, instead of 1
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1556)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
            at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
            at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:422)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
            at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
    
            at org.apache.hadoop.ipc.Client.call(Client.java:1070)
            at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
            at com.sun.proxy.$Proxy8.addBlock(Unknown Source)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)

网上说这个错误跟Hadoop的namenode节点相关，有打开hadoop下面的logs目录，发现报错如下，虽然使用JPS查看namenode进程是存在的：

2017-11-09 09:23:07,080 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root cause:java.io.IOException: File /home/alan/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    2017-11-09 09:23:07,080 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call addBlock(/home/alan/hadoop/tmp/mapred/system/jobtracker.info, DFSClient_-1833596442, null) from 192.168.14.151:51911: error: java.io.IOException: File /home/alan/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    java.io.IOException: File /home/alan/hadoop/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
    	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1556)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:696)
    	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
    	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
    	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:422)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
    	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)

试过了所有网上的方式，重新格式化namenode，修改host，最后发现是自己的系统硬盘空间不足，所以无法向namenode写入数据。

查看硬盘命令：df -h

[SouthEast]: /images/20210920/026cfd90ea7b4200ad567ae96ed0853b.png