spark java eclipse_windows下 eclipse搭建spark java编译环境

亦凉 2022-11-02 13:13 163阅读 0赞

环境：

win10

jdk1.8

之前有在虚拟机或者集群上安装spark安装包的，解压到你想要放spark的本地目录下，比如我的目录就是D:\\Hadoop\\spark-1.6.0-bin-hadoop2.6

/\*\*

\*注意：

之前在linux环境下安装的spark的版本是spark-2.2.0-bin-hadoop2.6，但后来搭建eclipse的spark开发环境时发现spark-2.2.0-bin-hadoop2.6解压后没有lib文件，也就没有关键的spark-assembly-1.6.0-hadoop2.6.0.jar这个jar包，不知道spark-2.2.0以后怎么支持eclipse的开发，所以我换了spark-1.6.0，如果有知道的大神，谢谢在下边留言指导一下。

\*\*/

下边就简单了，先配置spark的环境变量，先添加一个SPARK\_HOME,如下：

![cae231b8c9f3fbadbc1dac5aea664858.png][]

然后把SPARK\_HOME配置到path，如下：

![3eeecacd6f7dab95d83f6ac37d1ee3c6.png][]

这样环境就搭好了，然后就是在eclipse上创建一个普通的java项目，然后把spark-assembly-1.6.0-hadoop2.6.0.jar这个包复制进工程并且导入，如下图

![035e8745de3949f593336336c1fd5827.png][]

就可以开发spark程序了，下边附上一段小的测试代码：

importjava.util.Arrays;importorg.apache.spark.SparkConf;importorg.apache.spark.api.java.JavaPairRDD;importorg.apache.spark.api.java.JavaRDD;importorg.apache.spark.api.java.JavaSparkContext;importorg.apache.spark.api.java.function.FlatMapFunction;importorg.apache.spark.api.java.function.Function2;importorg.apache.spark.api.java.function.PairFunction;importorg.apache.spark.api.java.function.VoidFunction;importscala.Tuple2;public classWordCount \{public static voidmain(String\[\] args) \{

SparkConf conf= new SparkConf().setMaster("local").setAppName("wc");

JavaSparkContext sc= newJavaSparkContext(conf);

JavaRDD text = sc.textFile("hdfs://master:9000/user/hadoop/input/test");

JavaRDD words = text.flatMap(new FlatMapFunction() \{private static final long serialVersionUID = 1L;

@Overridepublic Iterable call(String line) throwsException \{return Arrays.asList(line.split(" "));//把字符串转化成list

\});

JavaPairRDD pairs = words.mapToPair(new PairFunction() \{private static final long serialVersionUID = 1L;

@Overridepublic Tuple2 call(String word) throwsException \{//TODO Auto-generated method stub

return new Tuple2(word, 1);

\});

JavaPairRDD results = pairs.reduceByKey(new Function2() \{private static final long serialVersionUID = 1L;

@Overridepublic Integer call(Integer value1, Integer value2) throwsException \{//TODO Auto-generated method stub

return value1 +value2;

\});

JavaPairRDD temp = results.mapToPair(new PairFunction, Integer, String>() \{private static final long serialVersionUID = 1L;

@Overridepublic Tuple2 call(Tuple2tuple)throwsException \{return new Tuple2(tuple.\_2, tuple.\_1);

\});

JavaPairRDD sorted = temp.sortByKey(false).mapToPair(new PairFunction, String, Integer>() \{private static final long serialVersionUID = 1L;

@Overridepublic Tuple2 call(Tuple2tuple)throwsException \{//TODO Auto-generated method stub

return new Tuple2(tuple.\_2,tuple.\_1);

\});

sorted.foreach(new VoidFunction>() \{private static final long serialVersionUID = 1L;

@Overridepublic void call(Tuple2 tuple) throwsException \{

System.out.println("word:" + tuple.\_1 + " count:" +tuple.\_2);

\});

sc.close();

统计的文件是下边的内容：

Look!at the window there leans an old maid. She plucks the

withered leaf from the balsam, and looks at the grass-covered rampart,

on which many children are playing. What is the old maid thinking

of?A whole life drama is unfolding itself before her inward gaze."The poor little children, how happy they are- how merrily they

play and romp together! What red cheeks and what angels' eyes! but

they have no shoes nor stockings. They dance on the green rampart,

just on the place where, according to the old story, the ground always

sank in, and where a sportive, frolicsome child had been lured by

means of flowers, toys and sweetmeats into an open grave ready dugforit, and which was afterwards closed over the child; and from that

moment, the old story says, the ground gave way no longer, the mound

remained firm and fast, and was quickly covered with the green turf.

The little people who now play on that spot know nothing of the old

tale,elsewould they fancy they heard a child crying deep below the

earth, and the dewdrops on each blade of grass would be to them

tears of woe. Nordothey know anything of the Danish King who here,

in the face of the coming foe, took an oath before all his trembling

courtiers that he would hold out with the citizens of his capital, and

die here in his nest; they know nothing of the men who have fought

here, or of the women who from here have drenched with boiling water

the enemy, clad in white, and'biding in the snow to surprise the

city.

当然也可以把这个工程打包成jar包，放在spark集群上运行，比如我打成jar包的名称是WordCount.jar

运行命令：/usr/local/spark/bin/spark-submit --master local --class cn.spark.test.WordCount /home/hadoop/Desktop/WordCount.jar

[cae231b8c9f3fbadbc1dac5aea664858.png]: /images/20221024/953ffa6868bd4af78144e2b08eca4ec7.png
[3eeecacd6f7dab95d83f6ac37d1ee3c6.png]: /images/20221024/41d98bf2087f43859a3f7ac8cb0b6b97.png
[035e8745de3949f593336336c1fd5827.png]: /images/20221024/8c58dcc7373b457093b2c3b1e89deff9.png