Spark安装和配置

男娘i 2022-07-29 08:41 308阅读 0赞

http://[blog.csdn.net/pipisorry/article/details/50924395][blog.csdn.net_pipisorry_article_details_50924395]

## 安装Hadoop ##

Spark默认使用HDFS充当持久化层,所以需要安装Hadoop,当然也可以不安装。

最好下载和spark对应的版本。

# spark单机环境安装 #

\{linux下的安装，windows上也类似\}

### 安装jdk依赖 ###

\[[java环境配置：安装jdk][java_jdk]\]

### **下载spark相应hadoop2.\*版本** ###

选择Spark最新发布版，一个预构建的Hadoop 2.\*包，直接下载。

[Download Apache Spark™][Download Apache Spark]

选择mirror下载，第二个下载链接较快

~/ENV$ wget http://mirror.bit.edu.cn/apache/spark/spark-2.0.0/spark-2.0.0-bin-hadoop2.7.tgz

解压到要解压的目录，如/opt

$ **sudo tar -xvf /media/pika/softwares/Downloads/LinuxInstalls/spark-1.6.1-bin-hadoop2.6.tgz -C /opt**

这样就可以在/opt/spark-1.6.1-bin-hadoop2.6目录下通过$./bin/spark-shell 或者$./bin/pyspark使用spark了。不过可以配置一下更好用：

### spark配置 ###

创建指向该Spark版本的符号链接到spark目录。这样你可以简单地下载新/旧版本的Spark，然后修改链接来管理Spark版本，而不用更改路径或环境变量。  
**pika:~$sudo ln -s /opt/spark-1.6.1-bin-hadoop2.6 /opt/spark**  
修改BASH配置，将Spark添加到PATH中，设置SPARK\_HOME环境变量，这样就可以直接在命令行上执行spark命令了。  
Ubuntu上，编辑~/.profile 或 ~/.bash\_profile文件   Note: ~/.profile is not read by bash(1), if ~/.bash\_profile or ~/.bash\_login exists.

pika:~$vi .profile  
pika:~$sudo vi /etc/environment \#修改这个也可以，不过是全局的不好

~$ vi ~/.bashrc    \#自己用的话就个人设置吧

将以下语句添加到文件中PATH的上面一行中：

**export SPARK\_HOME=/opt/spark**

将pyspark的python版本指向python3

**export PYSPARK\_PYTHON=python3**

将下面的语句加入到PATH中  
\# added by Anaconda3 4.1.1 installer  
export PATH="/home/piting/ENV/anaconda3/bin:$PATH"  
export PATH="**$SPARK\_HOME/bin**:$PATH"

ps: if you want to run in in IPython Notebook, write:  
PYSPARK\_PYTHON=python3 PYSPARK\_DRIVER\_PYTHON=ipython3 PYSPARK\_DRIVER\_PYTHON\_OPTS="notebook" ./bin/pyspark    \#当然这是要安装的

\[[Apache Spark: How to use pyspark with Python 3][Apache Spark_ How to use pyspark with Python 3]\]

出错处理：

Exception: Python in worker has different version 2.7 than that in driver 3.5, PySpark cannot run with different minor versions

Exception: Python in worker has different version 3.4 than that in driver 3.5, PySpark cannot run with different minor versions

解决

PYSPARK_PYTHON = /home/piting/ENV/anaconda3/bin/python

os.environ["PYSPARK_PYTHON"] = PYSPARK_PYTHON

driver对应的python版本就是SPARK\_PYTHONPATH设置的

worker对应的python版本就是PYSPARK\_PYTHON设置的

有时如果你使用的是虚拟环境或者anaconda，不能直接设置export PYSPARK\_PYTHON=python3，这样可能直接指向的是系统全局的设置，而不是当前用户的。

修改.bashrc只要. .bashrc就立刻生效。

修改生效pika:~$. /etc/environment，只是在root用户下当前命令窗口生效，总是生效要重启。

### 启动spark ###

$spark-shell   \#启动scala spark

**$pyspark**  \#启动python spark

PS：本博客是根据《spark机器学习》学习后记录的笔记。书代码及勘误\[[CODE DOWNLOADS & ERRATA][CODE DOWNLOADS _ ERRATA]\]，填写好邮箱等就会发送给你了，lz：其实就是这个地址http://www.packtpub.com/code\_download/17400，另\[[《Machine Learning with Spark》书评与作者访谈][Machine Learning with Spark]\]。

## 简略Spark输出 ##

Spark（和PySpark）的执行可以特别详细，很多INFO日志消息都会打印到屏幕。开发过程中，这些非常恼人，因为可能丢失Python栈跟踪或者print的输出。为了减少Spark输出 – 你可以设置$SPARK\_HOME/conf下的log4j。

首先，拷贝一份$SPARK\_HOME/conf/log4j.properties.template文件，去掉“.template”扩展名。  
$**sudo cp $SPARK\_HOME/conf/log4j.properties.template $SPARK\_HOME/conf/log4j.properties**

编辑新文件，用WARN替换代码中出现的INFO。你的log4j.properties文件类似：  
\# Set everything to be logged to the console  
 19 log4j.rootCategory=**WARN**, console  
 20 log4j.appender.console=org.apache.log4j.ConsoleAppender  
 21 log4j.appender.console.target=System.err  
 22 log4j.appender.console.layout=org.apache.log4j.PatternLayout  
 23 log4j.appender.console.layout.ConversionPattern=%d\{yy/MM/dd HH:mm:ss\} %p %c\{1\}: %m%n  
 24   
 25 \# Settings to quiet third party logs that are too verbose  
 26 log4j.logger.org.spark-project.jetty=**WARN**  
 27 log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR  
 28 log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=**WARN**  
 29 log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=**WARN**  
 30 log4j.logger.org.apache.parquet=ERROR  
 31 log4j.logger.parquet=ERROR  
 32   
 33 \# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in       SparkSQL with Hive support  
 34 log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL  
 35 log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR  
现在运行PySpark，输出消息将会更简略！

\[[spark快速入门之最简配置 spark 1.5.2 hadoop 2.7 配置][spark_ spark 1.5.2 hadoop 2.7]\]

[皮皮blog][blog]

# Spark分布式集群环境搭建    #

\[[Spark分布式集群环境搭建][Spark]\]

[皮皮blog][blog]

# pycharm中配置pyspark #

## pycharm中使用pyspark    ##

### python安装py4j ###

**pip install py4j**

### pycharm中配置python开发环境 ###

其实只需要在代码中加入下面语句就可以了，这样就不用配置pycharm了，免得换一个IDE还是会出错（主要原因还是因为系统的spark环境没配置好，如使用ssh远程登录时可能没执行spark配置的环境文件）。如果报错：ImportError: No module named 'SocialNetworks'通过加入下面的代码就可以解决。

SPARK_HOME = /home/piting/ENV/spark
    PYSPARK_PYTHON = /home/piting/ENV/anaconda3/bin/python
    SPARK_LOCAL_IP = 192.168.0.172
    # spark environment settingsos.environ['SPARK_HOME'] = conf.get(SECTION, 'SPARK_HOME')
     # Path for pyspark and py4jsys.path.append(os.path.join(conf.get(SECTION, 'SPARK_HOME'), 'python')) # sys.path.append("/home/piting/ENV/spark/python/lib/py4j-0.10.1-src.zip") os.environ["PYSPARK_PYTHON"] = conf.get(SECTION, 'PYSPARK_PYTHON')
     # You might need to enter your local IP os.environ['SPARK_LOCAL_IP'] = conf.get(SECTION, 'SPARK_LOCAL_IP') from pyspark import SparkContext

\[[如何使用PyCharm编写Spark程序(pyspark) ][PyCharm_Spark_pyspark_]\]

Note: 如果配置了下面的PyCharm里面配置第三方库的代码自动提示，包含了语句sys.path.append("/home/piting/ENV/spark/python")的效果。

或者也可以这样（局部设置）

在项目Project的要运行的python代码运行配置中选择“Run” ->“Edit Configurations” ->“Environment variables”“Configurations” ->“Environment variables”增加SPARK\_HOME目录与PYTHONPATH目录。

SPARK\_HOME:Spark安装目录; PYTHONPATH:Spark安装目录下的Python目录

![Center][]

测试一下是否成功，DataAnalysis.py:

from pyspark import SparkContext
    sc = SparkContext("local[4]", "tianchi data analysis")
    songs_data = sc.textFile(r"/media/pika/files/machine_learning/RecSys/Tianchi/mars_tianchi_songs.csv")
    print(songs_data.first())

**Note**: 也可以使用spark-submit命令来运行py代码: $ **spark-submit DataAnalysis.py**

输出类似下面内容就成功了（print输出正确就OK了）：

![Center 1][]

\[[使用PyCharm配置Spark的Python开发环境（基础）][PyCharm_Spark_Python]\]

\[[importing pyspark in python shell][]\]

## PyCharm里面配置第三方库的代码自动提示 ##

按照上面配置好pycharm后，在pycharm中*from* pyspark *import* SparkContext会出现红色波浪线，这样就不能在输入代码时候自动提示。

或者py 文件可以运行，无 Bug，不过写了import pyspark之后或者类似下面这句话导入，对应的还是有红色波浪线，其实是可以正常运行的，只是 PyCharm提示错误：*from* pyspark *import* SparkContext   Unresolved reference pyspark, SparkContext . Updated generator3.py script function isCallable to handle failure to retrieve '\_\_call\_\_' attribute.\[[Unresolved reference in PyQt for QtGui module][]\]

### 需要如下配置第三方库代码的自动提示 ###

点击最右边下面的interpreter paths按钮添加SPARK\_PYTHON路径如/opt/spark/python

![Center 2][]

\[[To enable IDE (PyCharm) syntax support for Apache Spark][To enable IDE _PyCharm_ syntax support for Apache Spark]\]

from: [http://blog.csdn.net/pipisorry/article/details/50924395][blog.csdn.net_pipisorry_article_details_50924395]

ref: \[[安装Hadoop及Spark(Ubuntu 16.04)][Hadoop_Spark_Ubuntu 16.04]\]

\[ 《Spark 官方文档》 [Spark配置][Spark 1]\]

[blog.csdn.net_pipisorry_article_details_50924395]: http://blog.csdn.net/pipisorry/article/details/50924395
[java_jdk]: http://blog.csdn.net/pipisorry/article/details/42918761
[Download Apache Spark]: http://spark.apache.org/downloads.html
[Apache Spark_ How to use pyspark with Python 3]: http://stackoverflow.com/questions/30279783/apache-spark-how-to-use-pyspark-with-python-3
[CODE DOWNLOADS _ ERRATA]: https://www.packtpub.com/books/content/support/17400
[Machine Learning with Spark]: http://www.infoq.com/cn/articles/machine-learning-with-spark-book-review
[spark_ spark 1.5.2 hadoop 2.7]: http://www.cnblogs.com/tina-smile/p/5045927.html
[blog]: http://blog.csdn.net/pipisorry
[Spark]: http://my.oschina.net/jackieyeah/blog/659741
[PyCharm_Spark_pyspark_]: http://blog.csdn.net/cds86333774/article/details/51264073
[Center]: /images/20220728/47ff85cfb1c948c28cabbdaee6d20733.png
[Center 1]: /images/20220728/91448b1e7edb4463b2f6ac7d44804b11.png
[PyCharm_Spark_Python]: http://www.tuicool.com/articles/iAbInuj
[importing pyspark in python shell]: http://stackoverflow.com/questions/23256536/importing-pyspark-in-python-shell
[Unresolved reference in PyQt for QtGui module]: https://youtrack.jetbrains.com/issue/PY-5131?query=%22
[Center 2]: /images/20220728/cc3fd5908fee44b385573eabee4638ad.png
[To enable IDE _PyCharm_ syntax support for Apache Spark]: https://gist.github.com/bigaidream/40fe0f8267a80e7c9cf8
[Hadoop_Spark_Ubuntu 16.04]: http://www.cnblogs.com/bovenson/p/5760856.html
[Spark 1]: http://ifeve.com/spark-config/