IDEA 创建scala spark的Mvn项目

青旅半醒 2022-04-13 11:18 234阅读 0赞

# IDEA 创建scala spark的Mvn项目 #

对人工智能感兴趣的同学，可以点击以下链接：

[现在人工智能非常火爆，很多朋友都想学，但是一般的教程都是为博硕生准备的，太难看懂了。最近发现了一个非常适合小白入门的教程，不仅通俗易懂而且还很风趣幽默。所以忍不住分享一下给大家。点这里可以跳转到教程。][Link 1]

[https://www.cbedai.net/u014646662][Link 1]

## 1.创建一个简单的Mvn项目 ##

1.1新建项目File -> new -> project -> Maven

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70][]

1.2  点击下一步，填写gid和aid

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 1][]

1.3 点击下一步，点击完成

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 2][]

mvn项目就建好了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 3][]

1.4 不要忘记你的Mvn配置（File -> Setting 搜索Maven）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 4][]

## 2 添加依赖（配置要和你的环境一致） ##

<properties>
            <scala.version>2.10.5</scala.version>
            <hadoop.version>2.6.5</hadoop.version>
        </properties>
    
        <repositories>
            <repository>
                <id>scala-tools.org</id>
                <name>Scala-Tools Maven2 Repository</name>
                <url>http://scala-tools.org/repo-releases</url>
            </repository>
        </repositories>
    
        <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>1.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 5][]

## 3.添加Scala libraries ##

File -> Project Structure

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 6][]

选择对应的版本（第一次选择应该下载，不过下载很慢，建议去官网下载Scala，然后选择目录）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 7][]

点击 OK

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 8][]

项目下就引入scala

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 9][]

## 4. 配置hadoop，配置hadoop-common-2.6.0-bin，以免出现null/winutils ##

hadoop-common-2.6.0-bin[下载][Link 2]：[https://download.csdn.net/download/u014646662/10816318][Link 2]

配置环境变量：

![20181129140428402.png][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 10][]

## 5.可以写scala程序了 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 11][]

## 6 编写Scala类 ##

package cn.com.tengen.test
    
    import java.util.Scanner
    
    import org.apache.spark.SparkContext
    import org.apache.spark.SparkConf
    
    
    class ScalaSparkContext(appName:String="Spark"){
      val sc:SparkContext = getSparkContext()
      def getSparkContext() = {
        val conf = new SparkConf().setAppName(appName)
        conf.setMaster("local")
        val sc = new SparkContext(conf)
        sc
      }
    
      def stop() = {
        println("按回车键结束：")
        val in = new Scanner(System.in)
        in.nextLine()
        sc.stop()
      }
    }
    
    
    object ScalaSparkContext {
      def main(args: Array[String]) {
        val sparkContext = new ScalaSparkContext()
        val sc = sparkContext.sc
        val input = sc.textFile("F:\\stoke_data\\2018-10-08-09-32-44.txt").map(_.toLowerCase)
        input
          .flatMap(line => line.split("[,]"))
          .map(word => (word, 1))
          .reduceByKey((count1, count2) => count1 + count2)
          .saveAsTextFile("aaaa")
        sparkContext.stop()
      }
    }

运行后，在控制台中可以看到：

18/11/29 14:46:01 INFO Utils: Successfully started service 'SparkUI' on port 4040.
    18/11/29 14:46:01 INFO SparkUI: Started SparkUI at http://172.16.66.1:4040

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 12][]

就进入web页面

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 13][]

注意：在sc.stop前可以访问

[Link 1]: https://www.cbedai.net/u014646662
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70]: /images/20220413/a33343231a734fb69605a3c2fed32198.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 1]: /images/20220413/7790b875611e41a79f6f763819de63a5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 2]: /images/20220413/19fca5f2267d4335b1de5889ec9ea0bc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 3]: /images/20220413/806d8e8b492140139e0a23055ea94ac9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 4]: /images/20220413/74450724492d4751871637ef210b26ac.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 5]: /images/20220413/8c44a8af24e949f69e9bff48bd61b29f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 6]: /images/20220413/80bb0c5fc301427aa4d5f41f52f42107.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 7]: /images/20220413/eacbb37a8b9b4cb99143497936fbc358.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 8]: /images/20220413/eb40dd1d64b146f0b3ffbe7e4c96460e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 9]: /images/20220413/ee6c4cd4967f46fdb963403f0d5e7ccc.png
[Link 2]: https://download.csdn.net/download/u014646662/10816318
[20181129140428402.png]: /images/20220413/a6d71c95c1b6406e8b04cbf012423e1b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 10]: /images/20220413/8025c687393a451e869de686344da1a8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 11]: /images/20220413/dfb86c8838a04146911298b29ab4aaea.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 12]: /images/20220413/143d747677bd44a4bf8dda10ec7c190d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTQ2NDY2NjI_size_16_color_FFFFFF_t_70 13]: /images/20220413/b942f6ff1e27442c92088a78d97b3ede.png