spark wordcount

浅浅的花香味﹌ 2022-06-06 07:22 182阅读 0赞

### 访问本地数据 ###

在/usr/local/spark/mycode/创建一个文件  
vi /usr/local/spark/mycode/word1.txt 输入一些单词，作为需要统计的数据

进入spark-shell

val textFile = sc.textFile("file:///usr/local/spark/mycode/word.txt")
    
    textFile.first(); //显示文本的第一行
    
    textFile.saveAsTextFile("file:///usr/local/spark/mycode/wordcopy") //把文件数据写到wordcopy目录下， 目录下会有part-00000为数据内容

### 访问hdfs上数据 ###

把word.txt上传到hdfs  
创建文件夹

> hadoop fs -mkdir /input  
> 上传  
> hadoop fs -put word.txt /input

进入spark-shell

> val textFile = sc.textFile(“hdfs://localhost:9000/input/word.txt”)  
> 上面路径也可以简写成:  
> val textFile = sc.textFile(“/input/word.txt”)

### 单词统计 ###

两种方式可以进行单词统计操作

#### 在spar-shell下方式 ####

> val wordCount = textFile.flatMap(line => line.split(” “)).map(word => (word,1)).reduceByKey((a,b) => a+b)
> 
> wordCount.collect()

输出内容

Array[(String, Int)] = Array((python,1), (hello,3), (java,1), (spark,1))

### 通过submit方式 ###

创建单词文件  
mkdir -p /usr/local/spark/mycode  
vi /usr/local/spark/mycode/word.txt

hello python
    hello java
    hello spark

创建scala工程的多级目录, 这里的wordcount就相当于程序的主目录  
mkdir -p /usr/local/spark/mycode/wordcount/src/main/scala

在scala目录下编写test.scala  
vi ~/src/main/scala/test.scala  
内容如下：

import org.apache.spark.SparkContext
    import org.apache.spark.SparkContext._
    import org.apache.spark.SparkConf
    
    object WordCount {
            def main(args: Array[String]){
                    val inputFile = "file:///usr/local/spark/mycode/word.txt"
                    val conf = new SparkConf().setAppName("WordCount").setMaster("local[2]")
                    val sc = new SparkContext(conf)
                    val textFile = sc.textFile(inputFile)
                    val wordCount = textFile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a, b) => a+b)
                    wordCount.foreach(println)
            }
    }

在工程主目录wordcount下创建simple.sbt  
内容为：

name := "Simple Project"
    version := "1.0"
    scalaVersion := "2.11.8"
    libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"

在工程主目录wordcount下执行命令,进行打包  
/usr/local/sbt/sbt package

执行结束后会有如下提示

[info] Done packaging.
    [success] Total time: 101 s, completed Nov 2, 2017 9:21:56 AM

进入工程目录下查看生成的jar文件(simple-project\_2.11-1.0.jar)  
cd /usr/local/spark/mycode/wordcount/target/scala-2.11

接着通过spark-submit 执行任务  
/usr/local/spark/bin/spark-submit –class “WordCount” /usr/local/spark/mycode/wordcount/target/scala-2.11/simple-project\_2.11-1.0.jar

在输出的信息中可以看到如下结果:

(spark,1)
    (python,1)
    (hello,3)
    (java,1)