Spark--安装与配置

柔光的暖阳◎ 2022-10-31 13:48 157阅读 0赞

原文网址：[Spark--安装与配置\_IT利刃出鞘的博客-CSDN博客][Spark--_IT_-CSDN]

# 简介 #

需要安装的：JDK、scala、Spark（注意与hadoop的版本对应）。

可选的安装：hadoop

# 安装 #

## windows ##

**其他网址**

> [Spark在Windows下的环境搭建\_大气人生-CSDN博客][Spark_Windows_-CSDN]  
> [Windows10安装spark（包括hadoop安装）\_小白白的博客-CSDN博客][Windows10_spark_hadoop_-CSDN]

**1.安装JDK**

> 略

**2.安装Hadoop**

> 见：[Hadoop系列--安装与设置\_feiying0canglang的博客-CSDN博客][Hadoop_--_feiying0canglang_-CSDN]

**3.安装Spark**

> 下载地址：[Downloads | Apache Spark][Downloads _ Apache Spark] （Pre-built：已经编译好了，下载来直接用就好）
> 
> 本处下载：spark-3.0.2-bin-hadoop3.2.tgz

**4.安装Scala**

> scala是为了运行spark-shell而安装的。

> 下载地址：[All Available Versions | The Scala Programming Language][All Available Versions _ The Scala Programming Language] //本处下载：scala-2.12.13.msi
> 
> 傻瓜式安装。安装后cmd输入scala若有输出则安装成功。
> 
> 注意版本： Spark 2.x is pre-built with Scala 2.11 except version 2.4.2, which is pre-built with Scala 2.12. Spark 3.0+ is pre-built with Scala 2.12.
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaXlpbmcwY2FuZ2xhbmc_size_16_color_FFFFFF_t_70][]

**5.运行spark**

> 到解压目录\\bin，运行：spark-shell
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaXlpbmcwY2FuZ2xhbmc_size_16_color_FFFFFF_t_70 1][]

**6.测试**

> 访问：[http://localhost:4040/][http_localhost_4040]
> 
> 结果：
> 
> ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaXlpbmcwY2FuZ2xhbmc_size_16_color_FFFFFF_t_70 2][]

## Docker ##

# 使用 #

## Python下的PySpark ##

针对Python下的Spark，有PySpark，和Scala下的spark-shell类似可以对Spark进行一些简单的调试和测试。

[Spark--_IT_-CSDN]: https://knife.blog.csdn.net/article/details/113964761
[Spark_Windows_-CSDN]: https://blog.csdn.net/u011513853/article/details/52865076
[Windows10_spark_hadoop_-CSDN]: https://blog.csdn.net/weixin_39750084/article/details/84453711
[Hadoop_--_feiying0canglang_-CSDN]: https://blog.csdn.net/feiying0canglang/article/details/113923682
[Downloads _ Apache Spark]: http://spark.apache.org/downloads.html
[All Available Versions _ The Scala Programming Language]: https://www.scala-lang.org/download/all.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaXlpbmcwY2FuZ2xhbmc_size_16_color_FFFFFF_t_70]: /images/20221024/a50a087d07c54dff9b33cc7785592ce7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaXlpbmcwY2FuZ2xhbmc_size_16_color_FFFFFF_t_70 1]: /images/20221024/fb283271cf6b45d3abab3aaaf37d053d.png
[http_localhost_4040]: http://localhost:4040/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaXlpbmcwY2FuZ2xhbmc_size_16_color_FFFFFF_t_70 2]: /images/20221024/3cb2f4ee63fb44f5b24cdb2dfe08454a.png