IDEA中编写Spark的WordCount程序 06

谁借莪１个温暖的怀抱￠ 2022-01-10 12:21 272阅读 0赞

# 1. 创建一个项目 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70]

# 2. 选择Maven项目,然后点击next #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 1]

# 3. 填写Maven的坐标,然后点击next #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 2]

# 4. 填写项目名称,然后点击finish #

![在这里插入图片描述][20190620090321397.png]

# 5. 创建好Maven项目后,点击Enable Auto-Import #

![在这里插入图片描述][20190620090345969.png]

# 6. 配置Maven的pom.xml #

<properties>
            <scala.version>2.11.8</scala.version>
            <spark.version>2.1.3</spark.version>
        </properties>
        <dependencies>
            <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-library</artifactId>
                <version>${scala.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>${spark.version}</version>
            </dependency>
        </dependencies>
        <build>
            <sourceDirectory>src/main/scala</sourceDirectory>
            <testSourceDirectory>src/test/scala</testSourceDirectory>
            <plugins>
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                            <configuration>
                                <args>
                                    <arg>-dependencyfile</arg>
                                    <arg>${project.build.directory}/.scala_dependencies</arg>
                                </args>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-shade-plugin</artifactId>
                    <version>2.4.3</version>
                    <executions>
                        <execution>
                            <phase>package</phase>
                            <goals>
                                <goal>shade</goal>
                            </goals>
                            <configuration>
                                <filters>
                                    <filter>
                                        <artifact>*:*</artifact>
                                        <excludes>
                                            <exclude>META-INF/*.SF</exclude>
                                            <exclude>META-INF/*.DSA</exclude>
                                            <exclude>META-INF/*.RSA</exclude>
                                        </excludes>
                                    </filter>
                                </filters>
                                <transformers>
                                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                        <mainClass></mainClass>
                                    </transformer>
                                </transformers>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
    </build>

# 7. 添加src/main/scala和src/test/scala,与pom.xml中的配置保持一致 #

![在这里插入图片描述][20190620090500696.png]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 3]

# 8. 新建一个scala class,类型为Object #

![在这里插入图片描述][20190620090535217.png]

# 9. 编写Spark程序 #

import org.apache.spark.{ SparkConf, SparkContext}
    import org.apache.spark.rdd.RDD
    
    object WordCount{ 
    	def main(args:Array[String]):Unit = { 
    		//设置spark的配置文件信息
    		val sparkConf:SparkConf = new SparkConf().setAppName("WordCount")
    		//构建SparkContext上下文对象,它是程序的入口,所有计算的源头
    		val sc:SparkContext = new SparkContext(sparkConf)
    		//读取文件
    		val file:RDD[String] = sc.textFile(args(0))
    
    		//对文件中每一行单词进行压平切分
    		val words:RDD[String] = file.flatMap(_.split(" "))
    		//对每一个单词计数1, 转化为(单词,1)
    		val wordAndOne:RDD[(String,Int)] = words.map(x=>(x,1))
    		//相同的单词进行汇总,前一个下划线表示累加数据,后一个下划线表示新数据
    		val result:RDD[(String,Int)] = wordAndOne.reduceByKey(_+_)
    		//保存数据到hdfs
    		result.saveAsTextFile(args(1))
    		sc.stop()
    	}
    }

# 10. 使用Maven打包 #

点击idea右侧的Maven Project选项  
![在这里插入图片描述][2019062009203014.png]  
点击Lifecycle,选择package，然后点击**Run Maven Build**  
![在这里插入图片描述][20190620092050727.png]

# 11. 选择编译成功的jar包,并将该jar上传到Spark集群的某个节点上 #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 4]

# 12. 首先启动hdfs和Spark集群 #

*  启动hdfs  
    /export/servers/hadoop/sbin/start-dfs.sh
 *  启动spark  
    /export/servers/spark/sbin/start-all.sh

# 13. 使用spark-submit命令提交Spark应用(注意参数的顺序) #

spark-submit \
    --class cn.test.spark.WordCount \
    --master spark://node1:7077 \
    --executor-memory 1g \
    --total-executor-cores 2 \
    /root/spark-1.0-SNAPSHOT.jar \
    /words.txt \
    /spark_out

这里通过spark-submit提交任务到集群上。用的是spark的Standalone模式  
Standalone模式是Spark内部默认实现的一种集群管理模式，这种模式是通过集群中的Master来统一管理资源。

1.  查看Spark的web管理界面  
    地址： 192.168.200.160:8080  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 5]
2.  查看hdfs上的结果文件  
    ![在这里插入图片描述][2019062009240494.png]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70]: /images/20220110/11442f1f645e4a8db61a91a901f9153d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 1]: /images/20220110/70fe460697ff4e5bb3b3f10d8a6ea785.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 2]: /images/20220110/a068a185cf894071a734561f0c85c03e.png
[20190620090321397.png]: /images/20220110/457f58b5b8564a25a8f052752f63237e.png
[20190620090345969.png]: /images/20220110/1c7f8874b1ab4be0b85d4a0ca7b756ee.png
[20190620090500696.png]: /images/20220110/deb41de3e55d4c33b8c41557a0e932a6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 3]: /images/20220110/d8390fccd20a4323bf68a77587b9bb6a.png
[20190620090535217.png]: /images/20220110/3feca281c17c4d2aa6bb023d70ccda0a.png
[2019062009203014.png]: /images/20220110/d2ec26303c0e4161a35e874184ef12b1.png
[20190620092050727.png]: /images/20220110/0c63409f5db34de3a6a215f7117987c3.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 4]: /images/20220110/862595354dbe496eb911f82eb1d9baf8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjY0MTkwOQ_size_16_color_FFFFFF_t_70 5]: /images/20220110/95a8fd28985a4fb2bcdfc23a09806d22.png
[2019062009240494.png]: /images/20220110/65f52e0396404eba8cabb2aac346de9d.png