spark 单机模式下安装

蔚落 2022-06-06 06:51 185阅读 0赞

解压复制到/usr/local/目录下 并改名为spark

在spark/conf下  
编辑 vi spark-env.sh 在第一行增加

export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)

这样spark就可读写hdfs数据

进入spark/bin目录下

run-example SparkPi 2>&1 | grep "Pi is"

输入结果 : Pi is roughly 3.1406957034785172

进入spark  
spark shell 可以看到比较酷的logo

Welcome to
          ____              __
         / __/__  ___ _____/ /__
        _\ \/ _ \/ _ `/ __/  '_/
       /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
          /_/

### 安装sbt ###

下载地址[https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.tgz][https_downloads.lightbend.com_scala_2.12.4_scala-2.12.4.tgz]  
解压到/usr/local/sbt 目录下 ，解压后里面就是一个jar文件  
在/usr/local/sbt下新建文件 “sbt” 并编辑内容：

#!/bin/bash
    SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
    java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"

在/usr/local/sbt下新建文件 “repositories” 并编辑内容：

[repositories]
    #本地源
    local
    #国内源，osChina
    osc: http://maven.oschina.net/content/groups/public/
    #兼容 Ivy 路径布局
    oschina-ivy:http://maven.oschina.net/content/groups/public/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
    #添加国外源备用
    typesafe: http://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly
    sonatype-oss-releases
    maven-central
    sonatype-oss-snapshots!/bin/bash

此时/usr/local/sbt下的目录结构

-rw-rw-r--. 1 hadoop hadoop     587 Nov  2 01:21 repositories
    -rwxrw-r--. 1 hadoop hadoop     155 Nov  2 01:31 sbt
    -rw-rw-r--. 1 hadoop hadoop 1210231 Nov  1 14:19 sbt-launch.jar

保存后，回到终端，在/usr/local/sbt目录下输入命令:  
sbt sbt-version  
等待下载，完成后如下提示：

[info] Set current project to sbt (in build file:/usr/local/sbt/) [info] 0.13.11

在家目录下创建scala工程目录结构  
mkdir -p ~/sparkapp/src/main/scala  
在scala目录下编写代码文件  
vi ~/sparkapp/src/main/scala/SimpleApp.scala  
内容为统计文本中a字母 ,b字母的行数：

import org.apache.spark.SparkContext
    import org.apache.spark.SparkContext._
    import org.apache.spark.SparkConf
    
    object SimpleApp{
            def main(args: Array[String]){
                    val logFile = "file:///usr/local/spark/README.md"
                    val conf = new SparkConf().setAppName("Simple Application")
                    val sc = new SparkContext(conf);
                    val logData = sc.textFile(logFile,2).cache()
                    val numAs = logData.filter(line => line.contains("a")).count()
                    val numBs = logData.filter(line => line.contains("b")).count()
                    println("Lines with a: %s, Lines with b: %s".format(numAs,numBs))
            }
    }

接着在sparkapp目录下创建文件编写内容  
vim ~/sparkapp/simple.sbt  
内容如下 (其中2.11.8是scala版本， 2.1.0是spark)

name := "Simple Project"
    version := "1.0"
    scalaVersion := "2.11.8"
    libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"

接着使用sbt将程序打包  
/usr/local/sbt/sbt package (一个很漫长的过程，会下载很多jar)

下载成功之后进入  
cd ~/sparkapp/target/scala-2.11  
会有一个simple-project\_2.11-1.0.jar包

运行程序,在终端输入：

/usr/local/spark/bin/spark-submit --class "SimpleApp" ~/sparkapp/target/scala-2.11/simple-project_2.11-1.0.jar 2>&1 | grep "Lines with a:"

得到如下结果：

Lines with a: 62, Lines with b: 30

[https_downloads.lightbend.com_scala_2.12.4_scala-2.12.4.tgz]: https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.tgz