轻松入门大数据第4章入门到进阶：系统玩转分布式计算框架MapReduce 4.1 Hadoop之MapReduce 4.1.50 场景题之groupby功能开发及测试

电玩女神 2024-03-17 09:14 7阅读 0赞

### 轻松入门大数据 ###

#### 文章目录 ####

*   *  轻松入门大数据
     *   *  第4章 入门到进阶：系统玩转分布式计算框架MapReduce
         *   *  4.1 Hadoop之MapReduce
             *   *  4.1.50 场景题之groupby功能开发及测试

![在这里插入图片描述][c96da71d14df4210883624ff224f4b2e.png_pic_center]

#### 第4章 入门到进阶：系统玩转分布式计算框架MapReduce ####

##### 4.1 Hadoop之MapReduce #####

> 1.  初识MapReduce
> 2.  MR优缺点
> 3.  MR核心思想
> 4.  官方词频统计案例分析
> 5.  MR编程规范
> 6.  MR数据类型
> 7.  词频统计分析详解
> 8.  模板方法设计模式
> 9.  序列化
> 10. Hadoop序列化之Writable
> 11. 自定义序列化案例实战
> 12. 初识InputFormat&InputSplit
> 13. InputSplit和Block的关系
> 14. InputFormat系列编程
> 15. Partitioner
> 16. Combiner
> 17. 排序
> 18. 初识OutputFormat
> 19. 自定义OutputFormat实战
> 20. 场景题之group by
> 21. 场景题之distinct
> 22. 场景题之join
> 23. Shuffle详解
> 24. MR核心组件总结

###### 4.1.50 场景题之groupby功能开发及测试 ######

OK，前面我们已经完成了需求分析

![在这里插入图片描述][69e763e33f634d6b8052ce16c165ea1d.png_pic_center]

下面进行实现：

package com.dingjiaxiong.mr.scenarios;
    
    import com.dingjiaxiong.utils.FileUtils;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    
    import java.io.IOException;
    
    /**
     * @Author Ding Jiaxiong
     * @Date 5/8/23 11:02 AM
     */
    public class GroupByApp {
        
    
        public static void main(String[] args) throws Exception {
        
            String input = "data/emp.txt";
            String output = "out";
    
            Configuration configuration = new Configuration();
            Job job = Job.getInstance(configuration);
    
            FileUtils.delele(configuration, output);
    
            job.setJarByClass(GroupByApp.class);
    
            job.setMapperClass(MyMapper.class);
            job.setReducerClass(MyReducer.class);
    
            job.setMapOutputKeyClass(IntWritable.class);
            job.setMapOutputValueClass(IntWritable.class);
    
            job.setOutputKeyClass(IntWritable.class);
            job.setOutputValueClass(IntWritable.class);
    
            FileInputFormat.setInputPaths(job, new Path(input));
            FileOutputFormat.setOutputPath(job, new Path(output));
    
            boolean result = job.waitForCompletion(true);
            System.exit(result ? 0 : 1);
        }
    
        public static class MyMapper extends Mapper<LongWritable, Text, IntWritable, IntWritable> {
        
    
            IntWritable ONE = new IntWritable(1);
    
            @Override
            protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, IntWritable, IntWritable>.Context context) throws IOException, InterruptedException {
        
                String[] splits = value.toString().split("\t");
                if (8 == splits.length) {
         // 8个长度才是正确的数据
                    int deptno = Integer.parseInt(splits[7].trim());
                    context.write(new IntWritable(deptno), ONE);
                }
            }
        }
    
        public static class MyReducer extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable> {
        
            @Override
            protected void reduce(IntWritable key, Iterable<IntWritable> values, Reducer<IntWritable, IntWritable, IntWritable, IntWritable>.Context context) throws IOException, InterruptedException {
        
                int count = 0;
    
                for (IntWritable value : values) {
        
                    count += value.get();
                }
    
                context.write(key, new IntWritable(count));
            }
        }
    }

![在这里插入图片描述][da2ce786c7024492b8526d9583f3c5c4.png_pic_center]

运行结果

![在这里插入图片描述][9e5d70421ea34bf1a8606e5441dce1d1.png_pic_center]

没问题。

这就是MR 实现groupby。

[c96da71d14df4210883624ff224f4b2e.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/17af782b5e8d47a8b65fdda6b1420786.png
[69e763e33f634d6b8052ce16c165ea1d.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/59a4039ce98e4ddebef7c657c5986652.png
[da2ce786c7024492b8526d9583f3c5c4.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/d3679d1a5fe441e0853112235adfef29.png
[9e5d70421ea34bf1a8606e5441dce1d1.png_pic_center]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/74918fe9fc5f4053b6db38a3214075e3.png