Windows下的搭建Hadoop、Spark和Scala编程环境

浅浅的花香味﹌ 2021-10-01 03:04 360阅读 0赞

# Windows下的搭建Hadoop、Spark和Scala编程环境 #

本文是基于Windows 10系统环境，搭建Hadoop、Spark和Scala编程环境

*  **Windows 10**
 *  **IntelliJ**
 *  **scala-2.11.12**
 *  **spark-2.4.3-bin-hadoop2.7**
 *  **hadoop-2.7.1**

--------------------

## 一、JDK安装 ##

### (1) 下载jdk ###

首先需要安装JDK，[jdk-8u221-windows-x64.exe][]  
然后双击安装到指定的目录，本文安装到 **C:\\Program Files\\Java\\jdk1.8.0\_181**，无论读者安装到哪个目录下，请记住该目录，在配置环境变量时，会用到该目录  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70]

### (2) 配置环境变量 ###

*  **新建一个系统变量 JAVA\_HOME**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 1]
 *  **然后编辑系统变量 PATH**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 2]
 *  **新建一个系统变量 CLASSPATH**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 3]

### (3) 验证java安装是否成功 ###

*  **打开的cmd命令行窗口下运行下面命令，查看是否设置成功。**

java -version

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 4]

## 二、scala安装 ##

### (1) 下载scala ###

*  **首先下载scala，[scala-2.11.12.msi][]**  
    然后双击安装到指定的目录，本文安装到 **D:\\Program\\scala**，无论读者安装到哪个目录下，请记住该目录，在配置环境变量时，会用到该目录  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 5]

### (2) 配置环境变量 ###

*  **新建一个系统变量 SCALA\_HOME**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 6]
 *  **然后编辑系统变量 PATH**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 7]
 *  **修改系统变量 CLASSPATH**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 8]

### (3) 验证scala安装是否成功 ###

*  **打开的cmd命令行窗口下运行下面命令，查看是否设置成功。**

scala

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 9]

## 三、Hadoop安装 ##

### (1) 下载Hadoop ###

[hadoop-2.7.1.tar.gz][]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 10]

*  **使用管理员身份，去解压 hadoop-2.7.1.tar.gz，不使用管理员身份会出错**
 *  **本文解压到了 D:\\Program\\hadoop\\hadoop-2.7.1**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 11]

### (2) 配置环境变量 ###

*  **新建一个系统变量 HADOOP\_HOME**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 12]
 *  **然后编辑系统变量 PATH**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 13]

### (3) 下载winutils.exe文件 ###

*  **下载 [winutils.exe][]**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 14]
 *  **将下载好winutils.exe后，将这个文件放入到Hadoop的bin目录下，我这里是D:\\Program\\hadoop\\hadoop-2.7.1**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 15]

## 四、spark安装 ##

### (1) 下载spark ###

*  **首先下载带有Hadoop版本的spark，spark-2.4.3-bin-hadoop2.7**
 *  **下载后得到了大约200M的文件： spark-2.4.3-bin-hadoop2.7.tgz**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 16]

### (2) 解压spark安装包 ###

*  **使用管理员身份，去解压 spark-2.4.3-bin-hadoop2.7.tgz，不使用管理员身份会出错**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 17]
 *  **并且需要注意的是，在Spark的文件目录路径名中，不要出现空格，类似于“Program Files”这样的文件夹名是不被允许的。**
 *  **本文将Spark安装在以下的目录 D:\\Program\\spark**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 18]

### (3) 配置环境变量 ###

*  **然后编辑系统变量 PATH**  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 19]

### (4) 验证spark安装是否成功 ###

*  **打开的cmd命令行窗口下运行下面命令，查看是否设置成功。**

spark-shell

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 20]

## 五、安装scala的IDEA插件 ##

### (1) 下载scala插件 ###

### (2) 配置scala环境 ###

1.  点击【项目】
2.  选择【Add Framework Support…】  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70_pic_center]
3.  选择【scala】
4.  点击【create】
5.  选择scala的安装目录

--------------------

[jdk-8u221-windows-x64.exe]: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70]: /images/20210920/ee5958fe2c0b4e6180aedd3d9111531a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 1]: /images/20210920/1a3cfd74c14a46de964bc5c6529e2f93.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 2]: /images/20210920/6dbc4f78ecb147f0866d603f1ee2d84b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 3]: /images/20210920/8a8c312e577a4bb08fb60f9a40d9acdf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 4]: /images/20210920/e505a43d692b4e579e72810e6558cfa2.png
[scala-2.11.12.msi]: https://www.scala-lang.org/download/2.11.12.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 5]: /images/20210920/50150c175c0748b08891636aa20528fc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 6]: /images/20210920/3c8b086d0e95460cabbee227d02b105a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 7]: /images/20210920/1fb6a3cda95841b4bcc203c9dca3bf17.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 8]: /images/20210920/05e8ef7c746b4887a4c316d67fe63623.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 9]: /images/20210920/b41e7f8acafb4575a57b1d525d4fda09.png
[hadoop-2.7.1.tar.gz]: https://archive.apache.org/dist/hadoop/common/hadoop-2.7.1/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 10]: /images/20210920/c880583872f2464cbfb7e8c3a9283f9a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 11]: /images/20210920/b84ccefc94fa411eb149138fa7fe4176.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 12]: /images/20210920/09cf8456ca37464d93a314ccf4c3e3a6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 13]: /images/20210920/3cf17634a2f240beabfe9c7b33186f49.png
[winutils.exe]: https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 14]: /images/20210920/4e49b561b4db4641ab1d6a4a9fccf662.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 15]: /images/20210920/292853c34ba44497b234f6da39fdda29.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 16]: /images/20210920/c3779535803140c590683e668a063dc4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 17]: /images/20210920/ab83cf39b9964cc3844017b7019fa3f2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 18]: /images/20210920/4970dc54211b4c2bbafee4030e9b2997.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 19]: /images/20210920/7c743903df5448bc96c6a9ef668c3607.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70 20]: /images/20210920/960d2a713405489c8e255660470a9433.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMyNTk5NDc5_size_16_color_FFFFFF_t_70_pic_center]: /images/20210920/b2e334a9e8e34e4985ccf7e2102757cf.png