idea运行spark的wordcount与eclipse运行spark的wordcount示例，及本地运行sparkpi

ゝ一世哀愁。 2022-08-07 04:58 240阅读 0赞

**一、idea运行wordcount**

1、下载idea的社区版本（免费的）

[http://www.jetbrains.com/idea/download/][http_www.jetbrains.com_idea_download]

2、安装scala插件

File-->Settings...----->Plugins

![Center][]

点击红色按钮，在搜索scala点击右面的初始化按钮安装scala插件

3、创建scala项目

File--->New Project....----->scala

![Center 1][]

选择好对应的SDK，完成创建

4、创建包类结构如下

![Center 2][]

/\*\*  
 \* 统计字符出现次数  
 \*/  
object WordCount \{  
  def main(args: Array\[String\]) \{  
    if (args.length < 2) \{  
      System.err.println("Usage: <file>")  
      System.exit(1)  
    \}  
  
    val conf = new SparkConf()  
    val sc = new SparkContext(conf)  
    val line = sc.textFile(args(0))  
    //输出到文件  
    line.flatMap(\_.split(" ")).map((\_, 1)).reduceByKey(\_+\_).saveAsTextFile(args(1))  
    //输出到屏幕  
    line.flatMap(\_.split(" ")).map((\_, 1)).reduceByKey(\_+\_).collect().foreach(println)  
    sc.stop()  
  \}  
\}

引入spark包

File---->Project Structure...---->Libraries

点击 “+” 选择java

![Center 3][]

5、打包

File-->Project Structure...--->Artifacts

![Center 4][]

接下来build

Build--->Build Artifaces...--->build

地一次选择build第二次选择rebuild

7、执行

将文件上传至linux上

运行spark环境

命令如下：

\[hadoop@master bin\]$ ./spark-submit --master spark://192.168.189.136:7077 --class main.scala.com.spark.firstapp.WordCount --executor-memory 1g /opt/testspark/FirstSparkApp2.jar hdfs://master:9000/user/hadoop/input/README.txt hdfs://master:9000/user/hadoop/output

8、结果

(if,1)  
(Commerce,,1)  
(or,2)  
(another,1)  
(software.,2)  
(laws,,1)  
(BEFORE,1)  
(source,1)  
(Hadoop,,1)  
(to,2)  
(written,1)  
(code,1)  
(software,,2)  
(Regulations,,1)  
(more,2)  
(regulations,1)  
(see,1)  
(of,5)  
(libraries,1)  
(by,1)  
(exception,1)  
(Control,1)  
(Government,1)  
(code.,1)  
(eligible,1)  
(both,1)  
(License,1)  
(Foundation,1)  
(functions,1)  
(and,6)  
(software:,1)  
(5D002.C.1,,1)  
((TSU),1)  
(Hadoop,1)  
15/03/13 14:34:17 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler\{/metrics/json,null\}  
15/03/13 14:34:17 INFO handler.ContextHandler: stopped o.e.j.s.ServletContextHandler\{/stages/stage/kill,null\}

输出在屏幕上的结果如上，也可以查看hdfs上的结果文件

**二、eclipse下创建scala运行wordcount**

1、下载scala ide for eclipse

[http://scala-ide.org/download/sdk.html][http_scala-ide.org_download_sdk.html]

2、创建scala项目

File-->New-->Scala Project

![Center 5][]

项目结构如下

![Center 6][]

与上述代码一致

引入spark的编译包

![Center 7][]

3、打包

File--->Export--->java-->JAR file

![Center 8][]

4、运行

上传到linux上

命令一样，结果页一样参考上面的

**三、本地运行SparkPi，idea及eclipse都可**

代码如下：

object SparkPi {
      def main(args: Array[String]) {
        //val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://192.168.189.136:7077").setJars(List("D:\\scala\\sparkjar\\sparktest.jar"))
        //val spark = new SparkContext("spark://master:7070", "Spark Pi", "F:\\soft\\spark\\spark-1.1.0-bin-hadoop2.4", List("out\\artifacts\\sparkTest_jar\\sparkTest.jar"))
        val conf = new SparkConf().setAppName("Spark Pi").setMaster("local")//主要是这句
        val spark = new SparkContext(conf)
        val slices = if (args.length > 0) args(0).toInt else 2
        val n = 100000 * slices
        val count = spark.parallelize(1 to n, slices).map { i =>
          val x = random * 2 - 1
          val y = random * 2 - 1
          if (x * x + y * y < 1) 1 else 0
        }.reduce(_ + _)
        println("Pi is roughly " + 4.0 * count / n)
        spark.stop()
      }
    }

[http_www.jetbrains.com_idea_download]: http://www.jetbrains.com/idea/download/
[Center]: /images/20220805/0386bf155e114c24b30f5559b5be16ec.png
[Center 1]: /images/20220805/89ef802ef3e04ccc9f85794684154dba.png
[Center 2]: /images/20220805/113b07eeb07c418b94fb9b96b06d08fc.png
[Center 3]: /images/20220805/12ef9e31aada4187818c44e62164a645.png
[Center 4]: /images/20220805/27c1726888af4f4d993a6debcd24de93.png
[http_scala-ide.org_download_sdk.html]: http://scala-ide.org/download/sdk.html
[Center 5]: /images/20220805/f94487d63b6b4af58e1f5d31aa48180e.png
[Center 6]: /images/20220805/a01fedcb46724fe0a0e2e7b4cfae5f11.png
[Center 7]: /images/20220805/6bdab391c477404db8395680d2c3b56c.png
[Center 8]: /images/20220805/bf10facd380949c4be6534619321da96.png