Hadoop系列-MapReduce编程入门案例（八）

ゞ浴缸里的玫瑰 2022-06-13 04:29 196阅读 0赞

Github代码下载地址：

1，[JAVA工程代码][JAVA]

2，[Maven工程代码][Maven]

代码如下：

**\[java\]**  [view plain][]   [copy][view plain]

[print][view plain] [?][view plain]

1.     
2.  **import** org.apache.hadoop.conf.Configuration;  
3.  **import** org.apache.hadoop.fs.Path;  
4.  **import** org.apache.hadoop.io.IntWritable;  
5.  **import** org.apache.hadoop.io.Text;  
6.  **import** org.apache.hadoop.mapreduce.Job;  
7.  **import** org.apache.hadoop.mapreduce.Mapper;  
8.  **import** org.apache.hadoop.mapreduce.Reducer;  
9.  **import** org.apache.hadoop.mapreduce.lib.input.FileInputFormat;  
10. **import** org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;  
11. **import** org.apache.hadoop.util.GenericOptionsParser;  
12.    
13. **import** java.io.IOException;  
14.    
15. **public** **class** EventCount \{  
16.    
17.     **public** **static** **class** MyMapper **extends** Mapper<Object, Text, Text, IntWritable>\{  
18.         **private** **final** **static** IntWritable one = **new** IntWritable(1);  
19.         **private** Text event = **new** Text();  
20.    
21.         **public** **void** map(Object key, Text value, Context context) **throws** IOException, InterruptedException \{  
22.             **int** idx = value.toString().indexOf(" ");  
23.             **if** (idx > 0) \{  
24.                 String e = value.toString().substring(0, idx);  
25.                 event.set(e);  
26.                 context.write(event, one);  
27.             \}  
28.         \}  
29.     \}  
30.    
31.     **public** **static** **class** MyReducer **extends** Reducer<Text,IntWritable,Text,IntWritable> \{  
32.         **private** IntWritable result = **new** IntWritable();  
33.    
34.         **public** **void** reduce(Text key, Iterable<IntWritable> values, Context context) **throws** IOException, InterruptedException \{  
35.             **int** sum = 0;  
36.             **for** (IntWritable val : values) \{  
37.                 sum += val.get();  
38.             \}  
39.             result.set(sum);  
40.             context.write(key, result);  
41.         \}  
42.     \}  
43.    
44.     **public** **static** **void** main(String\[\] args) **throws** Exception \{  
45.         Configuration conf = **new** Configuration();  
46.         String\[\] otherArgs = **new** GenericOptionsParser(conf, args).getRemainingArgs();  
47.         **if** (otherArgs.length < 2) \{  
48.             System.err.println("Usage: EventCount <in> <out>");  
49.             System.exit(2);  
50.         \}  
51.         Job job = Job.getInstance(conf, "event count");  
52.         job.setJarByClass(EventCount.**class**);  
53.         job.setMapperClass(MyMapper.**class**);  
54.         job.setCombinerClass(MyReducer.**class**);  
55.         job.setReducerClass(MyReducer.**class**);  
56.         job.setOutputKeyClass(Text.**class**);  
57.         job.setOutputValueClass(IntWritable.**class**);  
58.         FileInputFormat.addInputPath(job, **new** Path(otherArgs\[0\]));  
59.         FileOutputFormat.setOutputPath(job, **new** Path(otherArgs\[1\]));  
60.         System.exit(job.waitForCompletion(**true**) ? 0 : 1);  
61.     \}  
62. \}

运行“mvn package”命令产生jar包hadoopstudy-1.0-SNAPSHOT.jar，并将jar文件复制到[hadoop][]安装目录下

这里假定我们需要分析几个日志文件中的Event信息来统计各种Event个数，所以创建一下目录和文件

**\[plain\]**  [view plain][]   [copy][view plain]

[print][view plain] [?][view plain]

1.  /tmp/input/event.log.1  
2.  /tmp/input/event.log.2  
3.  /tmp/input/event.log.3

因为这里只是要做一个列子，所以每个文件内容可以都一样，假如内容如下

**\[plain\]**  [view plain][]   [copy][view plain]

[print][view plain] [?][view plain]

1.  JOB\_NEW ...  
2.  JOB\_NEW ...  
3.  JOB\_FINISH ...  
4.  JOB\_NEW ...  
5.  JOB\_FINISH ...

然后把这些文件复制到HDFS上

**\[plain\]**  [view plain][]   [copy][view plain]

[print][view plain] [?][view plain]

1.  $ bin/hdfs dfs -put /tmp/input /user/fkong/input

运行mapreduce作业

**\[plain\]**  [view plain][]   [copy][view plain]

[print][view plain] [?][view plain]

1.  $ bin/hadoop jar hadoopstudy-1.0-SNAPSHOT.jar my.hadoopstudy.mapreduce.EventCount /user/minbo/input /user/minbo/output

查看执行结果

**\[plain\]**  [view plain][]   [copy][view plain]

[print][view plain] [?][view plain]

1.  $ bin/hdfs dfs -cat /user/minbo/output/part-r-00000

注意：

如果运行mapreduce job时提示不能分配内存，表示Jvm默认内存配置不够，则需要配置HADOOP\_HOME/etc/hadoop/core-site.xml文件，最终配置如下图：

![Center][]

参考资料：

[http://hadoop.apache.org/docs/r2.7.3/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html][http_hadoop.apache.org_docs_r2.7.3_hadoop-mapreduce-client_hadoop-mapreduce-client-core_MapReduceTutorial.html]

[http://blog.csdn.net/kongxx/article/details/42339581][http_blog.csdn.net_kongxx_article_details_42339581]

[http://blog.csdn.net/ruidongliu/article/details/10006905][http_blog.csdn.net_ruidongliu_article_details_10006905]

[JAVA]: https://github.com/hemin1003/hadoop-study/tree/master/src/com/hadoop/minbo/mapreduce
[Maven]: https://github.com/hemin1003/java-study/tree/master/src/main/java/com/minbo/javademo/hadoop
[view plain]: http://blog.csdn.net/kongxx/article/details/42339581#
[hadoop]: http://lib.csdn.net/base/hadoop
[Center]: /images/20220613/37c6d6dbc3ef4e52991467df018b5806.png
[http_hadoop.apache.org_docs_r2.7.3_hadoop-mapreduce-client_hadoop-mapreduce-client-core_MapReduceTutorial.html]: http://hadoop.apache.org/docs/r2.7.3/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
[http_blog.csdn.net_kongxx_article_details_42339581]: http://blog.csdn.net/kongxx/article/details/42339581
[http_blog.csdn.net_ruidongliu_article_details_10006905]: http://blog.csdn.net/ruidongliu/article/details/10006905