eclipse配置连接Hadoop

小灰灰 2022-11-09 06:07 162阅读 0赞

### HDFS的Java API编程配置 ###

*   *  一、远程开发环境搭建
     *   *  1.1 解压Hadoop压缩包，配置环境
         *  1.2 配置winutils.exe 和 hadoop.dll ；hadoop-eclipse-plugin-2.6.0.jar
         *  1.3 用Eclipse远程连接Hadoop集群

--------------------

[Hadoop集群搭建及配置③ —— 基础环境搭建][Hadoop_ _]

[Hadoop集群搭建及配置④ —— JDK简介及其安装][Hadoop_ _ JDK]

[Hadoop集群搭建及配置⑤ —— Zookeeper 讲解及安装][Hadoop_ _ Zookeeper]

[Hadoop集群搭建及配置⑥ —— Hadoop组件安装及配置][Hadoop_ _ Hadoop]

[Hadoop集群搭建及配置⑦—— Spark&Scala安装配置][Hadoop_ Spark_Scala]

[Hadoop集群搭建及配置⑧——Hbase的安装配置][Hadoop_Hbase]

[eclipse 实现 Hdfs java API][eclipse _ Hdfs java API]

--------------------

HDFS不仅可以通过Shell操作Hadoop集群中的文件系统，还提供了Java API接口完成相应的功能。

**文件资料：**

链接：[https://pan.baidu.com/s/1WveL4l5Wt8A1XJXX40IZFQ][https_pan.baidu.com_s_1WveL4l5Wt8A1XJXX40IZFQ]  
提取码：**xlxq**

--------------------

## 一、远程开发环境搭建 ##

### 1.1 解压Hadoop压缩包，配置环境 ###

将**hadoop-2.7.3.tar.gz**(按照自己在虚拟机安装的版本) 解压到Windows下，在解压过程中可能会有错误提示，这是由于Linux文件格式和Windows不一样造成的，可以忽略。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]  
右击我的电脑–>属性–>高级系统设置里面环境变量要配置三个属性：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]  
（1）添加变量名：HADOOP\_HOME； 变量值：hadoop解压后的路径`D:\bigdate\hadoop-2.7.3`。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]

（2）path属性里添加一个：`%HADOOP_HOME%\bin`  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]

(3）添加变量名：HADOOP\_USER\_NAME 变量值：root  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]

### 1.2 配置winutils.exe 和 hadoop.dll ；hadoop-eclipse-plugin-2.6.0.jar ###

1.  将解压出的hadoop.dIIl和winutils.exe两个文件,分别放到 `D:\bigdate\hadoop-2.7.3\bin` 目录和 `C:\Windows\System32` 目录。  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 6]  
    .
2.  把 `hadoop-eclipse-plugin-2.6.0.jar` 放到 `D:\Program Files (x86)\eclipse\dropins` 文件夹里  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 7]

### 1.3 用Eclipse远程连接Hadoop集群 ###

打开eclipse，新建项目Map/Reduce——Hadoop1。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 8]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 9]  
点击 Map/Reduce Locations,然后在其下面空白处右键,选择New Hadoop Location,

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 10]

**将值修改为下图所示，其中：**  
①Location name: 命名新建的Hadoop连接；

②Map/Reduce(V2) Master:填写Hadoop集群的ResourceManager的IP地址和端口；  
通过配置文件`yarn-site.xml`，属性`yarn.resourcemanager.address`的值，查看端口号。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 11]

③DFS Master：填写Hadoop集群的NameNode的IP地址和端口；  
通过配置文件`core-site.xml`，属性fs.default.name的值，查看端口号。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 12]

④User name：用户名  
设置完毕后点击finish。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 13]  
连接Hadoop成功后，左上角DFS刷新一下会显示出HDFS文件系统根目录下的文件。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 14]  
可以看到文件是一样的！！！

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 15]  
注意 连接Hadoop时，虚拟机的Hadoop集群一定要开启，不然连接不上！！

[Hadoop_ _]: https://blog.csdn.net/weixin_44775255/article/details/113755997
[Hadoop_ _ JDK]: https://blog.csdn.net/weixin_44775255/article/details/113771546
[Hadoop_ _ Zookeeper]: https://blog.csdn.net/weixin_44775255/article/details/113772666
[Hadoop_ _ Hadoop]: https://blog.csdn.net/weixin_44775255/article/details/113775449
[Hadoop_ Spark_Scala]: https://blog.csdn.net/weixin_44775255/article/details/114518207
[Hadoop_Hbase]: https://blog.csdn.net/weixin_44775255/article/details/114676669
[eclipse _ Hdfs java API]: https://blog.csdn.net/weixin_44775255/article/details/114809298
[https_pan.baidu.com_s_1WveL4l5Wt8A1XJXX40IZFQ]: https://pan.baidu.com/s/1WveL4l5Wt8A1XJXX40IZFQ
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70]: /images/20221023/ffe29b480d944f1380e9b03adab388a3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 1]: /images/20221023/153bcb0b17104540a0021898d82ed34b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 2]: /images/20221023/4a1e071a50ee4be3b96112f5d940b006.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 3]: /images/20221023/10d9ff20a9534e4eb14c3967aa909bda.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 4]: /images/20221023/f4f263bf6424486bb47edac1c131b14b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 5]: /images/20221023/7b4199cccb6f4fbda6323884f7156ac8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 6]: /images/20221023/968872feb4304f53b732a68d006a00a6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 7]: /images/20221023/7426193ba0d44b90ac76b798f42398cc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 8]: /images/20221023/a68fff1b6ab24ff9a0e326bc396a0424.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 9]: /images/20221023/d528115aba374866baef71de5cdae991.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 10]: /images/20221023/8f9cee43b97f47e3b191a5413aaf5954.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 11]: /images/20221023/cad3a05da2b84f9483b90152410102c5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 12]: /images/20221023/753b3e654a5646e0a13005c5b8137497.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 13]: /images/20221023/4ce1c6236cee4c7988799d580af656df.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 14]: /images/20221023/b3077ef9897d40a697723533a58c6efa.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80NDc3NTI1NQ_size_16_color_FFFFFF_t_70 15]: /images/20221023/2d0da300a99b48699df49882eb1d7f88.png