MapReduce编程开发之数据排序

傷城~ 2024-04-17 20:28 63阅读 0赞

MapReduce的数据排序，其实没有很复杂的实现，默认在shuffle阶段，MapReduce就帮我们将数据排好序了，我们在Map和Reduce阶段，无需做额外的操作。

MapReduce在shuffle阶段，默认帮我们将数据做了排序，并且是做了合并，相同的数据归为一组了。在reduce的时候，我们需要注意的是：因为相同的键做了合并，为了将所有的数据输出，还需要遍历每一个键，如果values集合长度大于1，那么就有可能是重复的数据。另外为了记录数据的顺序，我们的程序里面设定了一个全局静态变量lineNum，记录所有集合的长度总和，同时也是数据的输出顺序。

mapreduce程序：

package com.xxx.hadoop.mapred;
    import java.io.IOException;
    import java.util.StringTokenizer;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.FileSystem;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    
    /**
     * 数据排序
     *
     */
    public class DataSortApp {
    	
    	public static class Map extends Mapper<Object, Text, IntWritable, IntWritable>{
    		protected void map(Object key, Text value, 
    				Context context) throws IOException ,InterruptedException {
    			StringTokenizer tokenizer = new StringTokenizer(value.toString(),"\n");
    			while(tokenizer.hasMoreElements()) {
    				IntWritable data = new IntWritable(Integer.parseInt(tokenizer.nextToken()));
    				context.write(data, new IntWritable(1));
    			}
    		}
    	}
    	
    	public static class Reduce extends Reducer<IntWritable, IntWritable, IntWritable, IntWritable>{
    		private static IntWritable lineNum = new IntWritable(1);
    		@SuppressWarnings("unused")
    		protected void reduce(IntWritable key, Iterable<IntWritable> values, 
    				Context context) throws IOException ,InterruptedException {
    			for(IntWritable val:values) {
    				context.write(lineNum, key);
    				lineNum.set(lineNum.get()+1);
    			}
    		}
    	}
    
    	public static void main(String[] args) throws Exception{
    		String input="/user/root/datasort/input",
    		       output="/user/root/datasort/output";
    		System.setProperty("HADOOP_USER_NAME", "root");
    		Configuration conf = new Configuration();
    		conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
    		
    		FileSystem fs = FileSystem.get(conf);
    		boolean exists = fs.exists(new Path(output));
    		if(exists) {
    			fs.delete(new Path(output), true);
    		}
    		
    		Job job = Job.getInstance(conf);
    		job.setJarByClass(DataSortApp.class);
    		job.setMapperClass(Map.class);
    		job.setReducerClass(Reduce.class);
    		job.setMapOutputKeyClass(IntWritable.class);
    		job.setMapOutputValueClass(IntWritable.class);
    		job.setOutputKeyClass(IntWritable.class);
    		job.setOutputValueClass(IntWritable.class);
    		
    		FileInputFormat.addInputPath(job, new Path(input));
    		FileOutputFormat.setOutputPath(job, new Path(output));
    		
    		System.exit(job.waitForCompletion(true)?0:1);
    	}
    
    }

准备数据：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70][]

运行程序，控制台打印信息：

2019-08-31 22:22:35 [INFO ]  [main]  [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
    2019-08-31 22:22:35 [INFO ]  [main]  [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
    2019-08-31 22:22:35 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
    2019-08-31 22:22:35 [WARN ]  [main]  [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
    2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
    2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
    2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local1111900714_0001
    2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
    2019-08-31 22:22:36 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Running job: job_local1111900714_0001
    2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter set in config null
    2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
    2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
    2019-08-31 22:22:36 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for map tasks
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_m_000000_0
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@42324622
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/datasort/input/1.txt:0+14
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
    2019-08-31 22:22:36 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 32; bufvoid = 104857600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_m_000000_0 is done. And is in the process of committing
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_m_000000_0' done.
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_m_000000_0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_m_000001_0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@1e445343
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/datasort/input/2.txt:0+13
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 32; bufvoid = 104857600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_m_000001_0 is done. And is in the process of committing
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_m_000001_0' done.
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_m_000001_0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_m_000002_0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3bd4bc73
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/datasort/input/3.txt:0+7
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] 
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Starting flush of map output
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Spilling map output
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 16; bufvoid = 104857600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.MapTask] Finished spill 0
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_m_000002_0 is done. And is in the process of committing
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] map
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_m_000002_0' done.
    2019-08-31 22:22:37 [INFO ]  [LocalJobRunner Map Task Executor #0]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_m_000002_0
    2019-08-31 22:22:37 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] map task executor complete.
    2019-08-31 22:22:37 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] Waiting for reduce tasks
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1111900714_0001_r_000000_0
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task]  Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@5cb84ffa
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.ReduceTask] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@37f7c67f
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] MergerManager: memoryLimit=1265788544, maxSingleShuffleLimit=316447136, mergeThreshold=835420480, ioSortFactor=10, memToMemMergeOutputsThreshold=10
    2019-08-31 22:22:37 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] attempt_local1111900714_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
    2019-08-31 22:22:37 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1111900714_0001 running in uber mode : false
    2019-08-31 22:22:37 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 0%
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1111900714_0001_m_000000_0 decomp: 42 len: 46 to MEMORY
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 42 bytes from map-output for attempt_local1111900714_0001_m_000000_0
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 42, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->42
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1111900714_0001_m_000001_0 decomp: 42 len: 46 to MEMORY
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 42 bytes from map-output for attempt_local1111900714_0001_m_000001_0
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 42, inMemoryMapOutputs.size() -> 2, commitMemory -> 42, usedMemory ->84
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1111900714_0001_m_000002_0 decomp: 22 len: 26 to MEMORY
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 22 bytes from map-output for attempt_local1111900714_0001_m_000002_0
    2019-08-31 22:22:37 [INFO ]  [localfetcher#1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 22, inMemoryMapOutputs.size() -> 3, commitMemory -> 84, usedMemory ->106
    2019-08-31 22:22:37 [INFO ]  [EventFetcher for fetching Map Completion Events]  [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] EventFetcher is interrupted.. Returning
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 3 sorted segments
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 3 segments left of total size: 88 bytes
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merged 3 segments, 106 bytes to disk to satisfy reduce memory limit
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 1 files, 106 bytes from disk
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 0 segments, 0 bytes from memory into reduce
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Merging 1 sorted segments
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 1 segments left of total size: 96 bytes
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.conf.Configuration.deprecation] mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task:attempt_local1111900714_0001_r_000000_0 is done. And is in the process of committing
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task attempt_local1111900714_0001_r_000000_0 is allowed to commit now
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] Saved output of task 'attempt_local1111900714_0001_r_000000_0' to hdfs://192.168.56.202:9000/user/root/datasort/output/_temporary/0/task_local1111900714_0001_r_000000
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] reduce > reduce
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.Task] Task 'attempt_local1111900714_0001_r_000000_0' done.
    2019-08-31 22:22:37 [INFO ]  [pool-6-thread-1]  [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1111900714_0001_r_000000_0
    2019-08-31 22:22:37 [INFO ]  [Thread-3]  [org.apache.hadoop.mapred.LocalJobRunner] reduce task executor complete.
    2019-08-31 22:22:38 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job]  map 100% reduce 100%
    2019-08-31 22:22:38 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Job job_local1111900714_0001 completed successfully
    2019-08-31 22:22:38 [INFO ]  [main]  [org.apache.hadoop.mapreduce.Job] Counters: 35
    	File System Counters
    		FILE: Number of bytes read=4001
    		FILE: Number of bytes written=1098948
    		FILE: Number of read operations=0
    		FILE: Number of large read operations=0
    		FILE: Number of write operations=0
    		HDFS: Number of bytes read=109
    		HDFS: Number of bytes written=55
    		HDFS: Number of read operations=37
    		HDFS: Number of large read operations=0
    		HDFS: Number of write operations=10
    	Map-Reduce Framework
    		Map input records=10
    		Map output records=10
    		Map output bytes=80
    		Map output materialized bytes=118
    		Input split bytes=366
    		Combine input records=0
    		Combine output records=0
    		Reduce input groups=10
    		Reduce shuffle bytes=118
    		Reduce input records=10
    		Reduce output records=10
    		Spilled Records=20
    		Shuffled Maps =3
    		Failed Shuffles=0
    		Merged Map outputs=3
    		GC time elapsed (ms)=8
    		Total committed heap usage (bytes)=1486880768
    	Shuffle Errors
    		BAD_ID=0
    		CONNECTION=0
    		IO_ERROR=0
    		WRONG_LENGTH=0
    		WRONG_MAP=0
    		WRONG_REDUCE=0
    	File Input Format Counters 
    		Bytes Read=34
    	File Output Format Counters 
    		Bytes Written=55

查看运行结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70 1][]

数据排序也有点词频统计的感觉，只不过我们无需关注词的次数具体是多少，但是我们需要利用这个次数来输出重复的记录。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/c433c247ac7f4569a085e8a2a698cd1e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2ZlaW5pZmk_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/820a329ff1fa47bb86426b88a34d902f.png