HDPCD-Java-复习笔记（15）

柔情只为你懂 2022-06-07 09:09 115阅读 0赞

**Unit Testing**

MRUnit is an Apache project that provides Java classes for unit testing MapReduce jobs.

There are basically three types of tests that can be run with MRUnit:

<table> 
 <tbody> 
  <tr> 
   <td> <p><strong>Map test</strong></p> </td> 
   <td> <p>Test a Mapper class.</p> </td> 
  </tr> 
  <tr> 
   <td> <p><strong>Reduce test</strong></p> </td> 
   <td> <p>Test a Reducer class.</p> </td> 
  </tr> 
  <tr> 
   <td> <p><strong>MapReduce</strong></p> </td> 
   <td> <p>Send a record to a Mapper and verify the Reducer’s output.</p> </td> 
  </tr> 
 </tbody> 
</table>

The MRUnit classes are found in the org.apache.hadoop.mrunit package. The three types of unit tests areeach implemented by the driver classes found in the org.apache.hadoop.mrunit.mapreduce package:

•MapDriver  
•ReduceDriver  
•MapReduceDriver

**To setup and run an MRUnit test, there are four essential steps:**

1.Create **an instance of the driver class**, passing in an instance of the Mapper class, Reducer class, or, for a MapReduce test, both.

2.Use the **addInput** or **withInput** methods of the **driver class** to specify the < key ,value > pair to pass into the map or reduce method.

3.Use the **addOutput** or **withOutput**methods in the driver class to specify the expected output.

4.Invoke the run method of the driver to run the test.

**Using MRUnit with JUnit**

To use MRUnit with JUnit, add the junit and org.apache.mrunit frameworks to your build environment.

For example, if you are using Gradle the settings look like:

*  testCompile group: 'junit', name: 'junit', 
 *   version:'4.11' 
 *  
 *  testCompile group: 'org.apache.mrunit', name:
 *   'mrunit', version:'1.0.0', classifier:'hadoop2'

**Setting Up a Test**

A unit test class typically stores the driver instance as a field and initializes it in a @Before method.

For example, the following EmployeeJobTest class creates a ReduceDriver instance:

*  public class EmployeeJobTest \{ 
 *  ReduceDriver<EmployeeKey, Employee,
 *   Text, DoubleWritable> reduceDriver;
 *   @Before 
 *   public void init() \{ 
 *   PayrollReducer reducer = new PayrollReducer();
 *   reduceDriver = ReduceDriver.newReduceDriver(reducer);
 *   \} 
 *   //remainder of class definition...
 *  \}

**Testing a Mapper**

The example above uses the runTest method, which validates the input with the expected output. The test can also be run using the run method, which returns the results in a List<Pair> instance instead of validating the results.

*  public class EmployeeJobTest \{ 
 *  MapDriver<LongWritable, Text, 
 *   EmployeeKey, Employee> mapDriver;
 *   @Before 
 *   public void init() \{ 
 *   mapDriver = MapDriver.newMapDriver(
 *   new PayrollMapper()); 
 *   \} 
 *   @Test 
 *   public void testPayrollMapperInput() \{ 
 *   LongWritable inputKey = new LongWritable(0);
 *   Text inputValue = 
 *   new Text("111223333,John,Wayne,IT,50000.00");
 *   EmployeeKey outputKey = new EmployeeKey(111223333);
 *   Employee outputValue = 
 *   new Employee("John","Wayne","IT",50000.00);
 *   mapDriver.withInput(inputKey, inputValue);
 *   mapDriver.withOutput(outputKey, outputValue);
 *   mapDriver.runTest(); 
 *   \} 
 *  \}

**Testing a Reducer**

•public class EmployeeJobTest \{  
•ReduceDriver<EmployeeKey, Employee,  
• Text, DoubleWritable> reduceDriver;  
• @Before  
• public void init() \{  
• PayrollReducer reducer = new PayrollReducer();  
• reduceDriver = ReduceDriver.newReduceDriver(reducer);  
• \}  
• @Test  
• public void testPayrollReducer() throws IOException \{  
• List<Employee> values = new ArrayList<Employee>();  
• values.add(mapOutputValue);  
• reduceDriver.withInput(mapOutputKey, values);  
• List<Pair<Text,DoubleWritable>> result =  
• reduceDriver.run();  
• for(Pair<Text,DoubleWritable> pair :result) \{  
• System.out.println(pair.getFirst().toString()  
• \+ " " + pair.getSecond().toString());  
• \}  
• \}

•\}

**A MapReduce Test**

*  public class EmployeeJobTest \{ 
 *   MapReduceDriver<LongWritable, Text, EmployeeKey,
 *   Employee, Text, DoubleWritable> mrDriver;
 *   LongWritable inputKey = new LongWritable(0);
 *   Text inputValue = 
 *   new Text("111223333,John,Wayne,IT,50000.00");
 *   EmployeeKey mapOutputKey = new EmployeeKey(111223333);
 *   Employee mapOutputValue = 
 *   new Employee("John","Wayne","IT",50000.00);
 *   @Before 
 *   public void init() \{ 
 *   PayrollReducer reducer = new PayrollReducer();
 *   PayrollMapper mapper = new PayrollMapper();
 *   mrDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
 *   \} 
 *   @Test 
 *   public void testPayrollJob() \{ 
 *   mrDriver.withInput(inputKey, inputValue);
 *   Text outputKey = new Text("Wayne,John");
 *   DoubleWritable outputValue =
 *   new DoubleWritable(50000.00);
 *   mrDriver.withOutput(outputKey, outputValue);
 *   mrDriver.runTest(); 
 *   \} 
 *  \}