启动hadoop步骤和遇到的坑

雨点打透心脏的1/2处 2021-07-26 23:31 382阅读 0赞

1.  格式化HDFS分布式文件系统

hadoop namenode –format

启动Hadoop

start-all.sh

停止Hadoop

stop-all.sh

jps命令可以看到Hadoop的所有守护进程

![2020081418412374.png][]

用hdfs dfsadmin -report 命令来检查，能看到DataNode状态才是正常

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70][]

可以通过Hadoop NameNode和JobTracker的Web接口来查看集群是否启动成功，其访问地址如下

http://192.168.96.128:8088/cluster

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 1][]

NameNode为http://192.168.96.128:50070/dfshealth.html\#tab-overview

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 2][]

JobTracker为[http://localhost:50030/][http_localhost_50030]

到此仅仅是在linux上布置成功

# **在eclipse上运行hdfs** #

很多博客说要下载插件hadoop-eclipse-plugin-2.6.0.jar，我下载后也放在eclipse安装文件夹的插件plugins下了，那头小象始终不见出来，发现不装插件也可以运行wordCount，与版本有关，但是不是必须对应的，

1、下载hadoop.dll和winutils.exe

[https://github.com/steveloughran/winutils/blob/master/hadoop-2.8.3/bin/hadoop.dll][https_github.com_steveloughran_winutils_blob_master_hadoop-2.8.3_bin_hadoop.dll]

winutils.exe放在hadoop-2.8.4\\bin

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 3][]

hadoop.dll放在C:\\Windows\\System32

![20200814184221312.png][]

1.  环境变量

访问hdfs时报权限错误

Permission denied: user=administrator, access=WRITE,

配置环境变量

![20200814184237466.png][]

配置服务器上的hadoop用户，因为执行login.login的时候调用了hadoop里面的HadoopLoginModule方法，会先读取HADOOP\_USER\_NAME系统环境变量，然后是java环境变量，如果再没有就从NTUserPrincipal里面取

![20200814184359698.png][]

配置好环境，重启Eclipse

1.  Maven中配置

<?xml version="1.0"?>
    
    -<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0">
    
    <modelVersion>4.0.0</modelVersion>
    
    <groupId>HadoopJar</groupId>
    
    <artifactId>Hadoop</artifactId>
    
    <version>0.0.1-SNAPSHOT</version>
    
    <packaging>jar</packaging>
    
    <name>Hadoop</name>
    
    <url>http://maven.apache.org</url>
    
    
    -<properties>
    
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    
    <hadoop.version>2.8.4</hadoop.version>
    
    </properties>
    
    
    -<dependencies>
    
    
    -<dependency>
    
    <groupId>org.apache.hadoop</groupId>
    
    <artifactId>hadoop-common</artifactId>
    
    <version>${hadoop.version}</version>
    
    </dependency>
    
    
    
    
    
    -<dependency>
    
    <groupId>org.apache.hadoop</groupId>
    
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    
    <version>${hadoop.version}</version>
    
    </dependency>
    
    
    
    
    
    -<dependency>
    
    <groupId>org.apache.hadoop</groupId>
    
    <artifactId>hadoop-hdfs</artifactId>
    
    <version>${hadoop.version}</version>
    
    </dependency>
    
    
    
    
    
    -<dependency>
    
    <groupId>org.apache.hadoop</groupId>
    
    <artifactId>hadoop-mapreduce-client-common</artifactId>
    
    <version>${hadoop.version}</version>
    
    </dependency>
    
    
    
    
    
    -<dependency>
    
    <groupId>org.apache.hadoop</groupId>
    
    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
    
    <version>${hadoop.version}</version>
    
    </dependency>
    
    
    -<dependency>
    
    <groupId>junit</groupId>
    
    <artifactId>junit</artifactId>
    
    <version>3.8.1</version>
    
    <scope>test</scope>
    
    </dependency>
    
    
    -<dependency>
    
    <groupId>jdk.tools</groupId>
    
    <artifactId>jdk.tools</artifactId>
    
    <version>1.8</version>
    
    <scope>system</scope>
    
    <systemPath>D:\Java\jdk1.8.0_101\lib/tools.jar</systemPath>
    
    </dependency>
    
    </dependencies>
    
    
    -<build>
    
    <finalName>Hadoop</finalName>
    
    
    -<plugins>
    
    
    -<plugin>
    
    <artifactId>maven-compiler-plugin</artifactId>
    
    
    -<configuration>
    
    <source>1.8</source>
    
    <target>1.8</target>
    
    <encoding>UTF-8</encoding>
    
    </configuration>
    
    </plugin>
    
    
    -<plugin>
    
    <groupId>org.apache.maven.plugins</groupId>
    
    <artifactId>maven-resources-plugin</artifactId>
    
    
    -<configuration>
    
    <encoding>UTF-8</encoding>
    
    </configuration>
    
    </plugin>
    
    </plugins>
    
    </build>
    
    </project>

1.  wordCount Demo

1.  配置文件

主机名要替换成IP

hadoop.tmp.dir /usr/local/hadoop-2.8.4/tmp Abase for other temporary directories.  fs.defaultFS hdfs://192.168.96.128:9000  io.file.buffer.size 4096

1.  运行结果

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 4][]

Hdfs dfs –mkdir /hi

1.   
2.   
3.

[2020081418412374.png]: /images/20210726/20af25d8127b43c791fc969f67c08ae6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70]: /images/20210726/136598e1556048edb2d47e8bcc7d7ae9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 1]: /images/20210726/570b22e7ab6942ca8d5f0d24479f1929.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 2]: /images/20210726/6db3ffce60c64718a06a143fd4121df4.png
[http_localhost_50030]: http://localhost:50030/
[https_github.com_steveloughran_winutils_blob_master_hadoop-2.8.3_bin_hadoop.dll]: https://github.com/steveloughran/winutils/blob/master/hadoop-2.8.3/bin/hadoop.dll
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 3]: /images/20210726/99c51556ec7740ba8bdf88da5c29f725.png
[20200814184221312.png]: /images/20210726/06097e8f0f0e4a29930ecf33634649d3.png
[20200814184237466.png]: /images/20210726/472dbdc1f90a48e48001f2b47c3aa4f0.png
[20200814184359698.png]: /images/20210726/ffad1a5e1e504411852edd461e9a7b96.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2x1YW5zag_size_16_color_FFFFFF_t_70 4]: /images/20210726/bf4b4432bb4a474381ee33203fd8dfc5.png