hadoop学习笔记2：eclipse开发MapReduce

水深无声 2022-07-13 07:57 109阅读 0赞

## hadoop学习笔记2：eclipse开发MapReduce ##

**伪分布式搭建见**：[Hadoop学习笔记1：伪分布式环境搭建][Hadoop_1]  
**环境：**  
虚拟机：Ubuntu16.0  
jdk1.8.0\_111,  
eclipse4.6.0,  
hadoop-2.7.3,  
下载地址见：  
[http://blog.csdn.net/yhhyhhyhhyhh/article/details/53945810][Hadoop_1]  
hadoop-eclipse-plugin-2.7.3,  
编译好的jar包下载地址:  
[http://download.csdn.net/download/yhhyhhyhhyhh/9726479][http_download.csdn.net_download_yhhyhhyhhyhh_9726479]

**1.eclipse安装及快捷方式**

1)解压并测试启动eclipse

sudo tar -zxvf ~/eclipse-jee-neon-1a-linux-gtk-x86_64.tar.gz -C/usr/local
    
    cd /usr/local/eclipse
    
    ./eclipse

2)eclipse快捷方式

创建eclipse.desktop

sudo vim /usr/share/applications/eclipse.desktop

在eclipse.desktop添加以下内容

[Desktop Entry]
    Version=1.0
    Encoding=UTF-8
    Name=Eclipse4.6.0
    Exec=eclipse
    TryExec=eclipse
    Comment=Eclipse4.6.0,EclipseSDK
    Exec=/usr/local/eclipse/eclipse
    Icon=/usr/local/eclipse/icon.xpm
    Terminal=false
    Type=Application
    Categories=Application;Development;

创建启动器

sudo vim /usr/bin/eclipse

在其中添加如下内容

#!/bin/sh
    export MOZILLA_FIVE_HOME="/usr/lib/mozilla/"
    export ECLIPSE_HOME="/usr/local/eclipse"
    $ECLIPSE_HOME/eclipse $*

添加可执行权限

sudo chmod +x /usr/bin/eclipse

注意：如果提示找到不jdk环境：在eclipse下建立软连接   
![SouthEast][]

yhh@ubuntu:/usr/local/eclipse$ sudo ln -s /usr/local/jdk1.8.0_111/jre/ jre

eclipse快捷方式创建成功如下：  
![SouthEast 1][]  
![SouthEast 2][]

**2. hadoop-eclipse-plugin-2.7.3的安装**

yhh@ubuntu:~$ cd /usr/local/eclipse/
    yhh@ubuntu:/usr/local/eclipse$ sudo cp ~/hadoop-eclipse-plugin-2.7.3.jar ./plugins/ 
    #复制到eclipse下的plugins 目录下
    yhh@ubuntu:/usr/local/eclipse$ ./eclipse -clean  # 添加插件后需要用这种方式使插件生效

**3. hadoop插件在eclipse中的配置**

安装好Hadoop-Eclipse-Plugin插件，启动 Eclipse 后就可以在左侧的Project Explorer中看到 DFS Locations（若看到的是 welcome 界面，点击左上角的 x 关闭就可以看到了）。  
注意：熟悉Java开发的,对eclipse视图切换应该很熟悉，不熟悉的话，如果DFS Locations或者Map/Reduce Locations面板没出现的话。只要配置了hadoop-eclispe插件，可以打开map/reduce视图，reset，见2）打开Map/Reduce视图，这里可以先配置，不影响。

1）配置hadoop安装目录  
![SouthEast 3][]

![SouthEast 4][]

2）打开Map/Reduce视图

![SouthEast 5][]

![SouthEast 6][]

注意：熟悉Java开发的,对eclipse视图切换应该很熟悉，不熟悉的话，如果DFS Locations或者Map/Reduce Locations面板没出现的话。只要配置了hadoop-eclispe插件，可以打开map/reduce视图，reset,如下图。

![SouthEast 7][]

![SouthEast 8][]

3）建立与hadoop集群的连接

![SouthEast 9][]

![SouthEast 10][]

注意：hadoop需要在终端中启动，这里才会链接上。

**hadoop启动前:**  
![SouthEast 11][]

**hadoop启动后:**

![SouthEast 12][]

**4.eclipse中建立hadoop工程**  
这里以hadoop自带的wordcount例子测试。  
在[Hadoop学习笔记1：伪分布式环境搭建][Hadoop_1]中，伪分布式、分布式的Hadoop数据是在hdfs下的目录。所以需要预先将数据上传到hdfs下的目录中。  
1）准备数据  
在/usr/local/input创建三个txt文件，f1,f2的内容随便输入，f3输入以下：

> 127.0.0.1,-,-,\[08/May/2014:13:42:40 +0800\],GET / HTTP/1.1,200,11444  
> 127.0.0.1,-,-,\[08/May/2014:13:42:42 +0800\],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurrentClassPlanVO HTTP/1.1,204,-  
> 127.0.0.1,-,-,\[08/May/2014:13:42:42 +0800\],GET /jygl/jaxrs/teachingManage/ClassBatchPlanService/getCurClassPlanVO HTTP/1.1,204,-  
> 127.0.0.1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jygl/jaxrs/right/isValidUserByType/1-admin-superadmin HTTP/1.1,200,20  
> 127.0.0.1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jygl/jaxrs/right/getUserByLoginName/admin HTTP/1.1,200,198  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/right\_login2home?loginName=admin&password=superadmin&type=1 HTTP/1.1,200,2525  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/style/style.css HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/js/tree.js HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/js/jquery.js HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/js/frame.js HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/images/logo.png HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/images/leftmenu\_bg.gif HTTP/1.1,404,1105  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/menuList.jsp HTTP/1.1,200,47603  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/style/images/header\_bg.jpg HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/images/allmenu.gif HTTP/1.1,404,1093  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:47 +0800\],GET /jyglFront/mainView/navigate/images/toggle\_menu.gif HTTP/1.1,404,1105  
> 127.0.0.1,-,-,\[08/May/2014:13:42:48 +0800\],GET /jygl/jaxrs/article/getArticleList/10-1 HTTP/1.1,200,20913  
> 127.0.0.1,-,-,\[08/May/2014:13:42:48 +0800\],GET /jygl/jaxrs/article/getTotalArticleRecords HTTP/1.1,200,22  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:48 +0800\],GET /jyglFront/baseInfo\_articleList?flag=1 HTTP/1.1,200,8989  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:42:48 +0800\],GET /jyglFront/mainView/studentView/style/images/nav\_10.png HTTP/1.1,404,1117  
> 127.0.0.1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jygl/jaxrs/right/isValidUserByType/1-admin-superadmin HTTP/1.1,200,20  
> 127.0.0.1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jygl/jaxrs/right/getUserByLoginName/admin HTTP/1.1,200,198  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/right\_login2home?loginName=admin&password=superadmin&type=1 HTTP/1.1,200,2525  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/js/tree.js HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/js/jquery.js HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/js/frame.js HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/style/style.css HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/menuList.jsp HTTP/1.1,200,47603  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/images/logo.png HTTP/1.1,304,-  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/images/leftmenu\_bg.gif HTTP/1.1,404,1105  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/images/toggle\_menu.gif HTTP/1.1,404,1105  
> 0:0:0:0:0:0:0:1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jyglFront/mainView/navigate/style/images/header\_bg.jpg HTTP/1.1,304,-  
> 127.0.0.1,-,-,\[08/May/2014:13:43:21 +0800\],GET /jygl/jaxrs/article/getArticleList/10-1 HTTP/1.1,200,20913

命令：

yhh@ubuntu:/usr/local/hadoop$ cd ..
    yhh@ubuntu:/usr/local$ sudo mkdir ./input
    yhh@ubuntu:/usr/local$ cd ./input
    yhh@ubuntu:/usr/local/input$ sudo vim f1.txt
    yhh@ubuntu:/usr/local/input$ sudo vim f2.txt
    yhh@ubuntu:/usr/local/input$ sudo vim f3.txt

在hdfs下建立文件夹 /user/yhh/input(\*\*为用户名我的是yhh)  
命令：

yhh@ubuntu:/usr/local$ cd ./hadoop/
    yhh@ubuntu:/usr/local/hadoop$ ./bin/hdfs dfs -ls /
    #可以先查看下hdfs下的文件级文件夹
    yhh@ubuntu:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir /user
    yhh@ubuntu:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir /user/yhh
    yhh@ubuntu:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir /user/yhh/input
    yhh@ubuntu:/usr/local/hadoop$ ./bin/hdfs dfs -put /usr/local/input/f*.txt /user/yhh/input
    #将数据上传到hdfs下的/user/yhh/input（如果已存在会有提示）

**上传数据到hdfs后：**

![SouthEast 13][]

2）Eclipse建立hadoop工程测试（以WordCount为例）

在Eclipse中创建MapReduce项目和创建普通的java项目一样。

![SouthEast 14][]

创建WordCount 项目后，添加类。  
选择 New -> Class需要填写两个地方：  
在 Package 处填写 org.apache.hadoop.examples；  
在 Name 处填写 WordCount。  
在WordCount.java：添加如下代码（可以将自动生成的删除）

package org.apache.hadoop.examples;
    import java.io.IOException;
    import java.util.StringTokenizer;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.util.GenericOptionsParser;
    
    @SuppressWarnings("unused")
    public class WordCount {
    
      public static class TokenizerMapper 
           extends Mapper<Object, Text, Text, IntWritable>{
    
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();
    
        public void map(Object key, Text value, Context context
                        ) throws IOException, InterruptedException {
          StringTokenizer itr = new StringTokenizer(value.toString());
          while (itr.hasMoreTokens()) {
            word.set(itr.nextToken());
            context.write(word, one);
          }
        }
      }
    
      public static class IntSumReducer 
           extends Reducer<Text,IntWritable,Text,IntWritable> {
        private IntWritable result = new IntWritable();
    
        public void reduce(Text key, Iterable<IntWritable> values, 
                           Context context
                           ) throws IOException, InterruptedException {
          int sum = 0;
          for (IntWritable val : values) {
            sum += val.get();
          }
          result.set(sum);
          context.write(key, result);
        }
      }
    
      public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        //通过配置工程输入参数
        //String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
        //代码中直接输入参数,从hdfs输入（具体数据在hdfs://localhost:9000/user/yhh/input）
        String[] otherArgs=new String[]{
       "input/f1.txt","output"}; //此处可以指定hdfs下具体的输入文件
        if (otherArgs.length != 2) {
          System.err.println("Usage: wordcount <in> <out>");
          System.exit(2);
        }
        //每次运行时自动删除输出目录
        Path outputPath = new Path(otherArgs[1]);
        outputPath.getFileSystem(conf).delete(outputPath, true);
        //1、设置job的基础属性  
        @SuppressWarnings("deprecation")
        Job job = new Job(conf, "WordCount");
        job.setJarByClass(WordCount.class);
        //2、设置Map与Reudce的类  
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        //3、设置map与reduce的输出键值类型  
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        //4、设置输入输出路径  
        FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
        FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
        //5、运行程序 
        System.exit(job.waitForCompletion(true) ? 0 : 1);
      }
    }

**注：如果需要通过配置工程输入参数:：**

//通过配置工程输入参数
    //String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

![这里写图片描述][SouthEast 15]

![这里写图片描述][SouthEast 16]  
注意：新建工程运行所需hadoop配置文件：   
在终端中通过命令复制到/workspace/WordCount/src下（**具体自己的，在命令中修改**）

cp /usr/local/hadoop/etc/hadoop/core-site.xml ~/workspace/WordCount/src
    cp /usr/local/hadoop/etc/hadoop/hdfs-site.xml ~/workspace/WordCount/src
    cp /usr/local/hadoop/etc/hadoop/log4j.properties ~/workspace/WordCount/src

注意：1）数据上传到hdfs  
2）终端执行复制配置文件  
3）运行成功后，在eclipse查看输出文件  
都需要Refresh或者重启eclipse工程，相关位置才会出现对应的文件  
4）输入输出已经在代码中指定，并且设置了运行自动删除上次的输出目录。hadoop job的简单属性设置在代码中。  
  
下图Refresh或者重启eclipse工程：  
![SouthEast 17][]

![SouthEast 18][]

测试成功：

![测试成功][SouthEast 19]

f3.txt的测试：**屏蔽掉测试f1.txt的代码，复制如下代码**

package org.apache.hadoop.examples;
    import java.io.IOException;
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.LongWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
    public class WordCount {
        enum Counter{  
            LINESKIP,  
        }     
        public static class CountMap extends Mapper<LongWritable,Text,Text,IntWritable>{  
            private static final IntWritable one = new IntWritable(1); 
            public void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException{  
                String line = value.toString(); 
                try{  
                    String[] lineSplit = line.split(","); 
                    String requestUrl = lineSplit[4]; 
                    requestUrl = requestUrl.substring(requestUrl.indexOf(' ')+1, requestUrl.lastIndexOf(' ')); 
                    Text out = new Text(requestUrl); 
                    context.write(out,one); 
                }catch(java.lang.ArrayIndexOutOfBoundsException e){  
                    context.getCounter(Counter.LINESKIP).increment(1); 
                }             
            }  
        }     
        public static class CountReduce extends Reducer<Text,IntWritable,Text,IntWritable>{          
            public void reduce(Text key, Iterable<IntWritable> values,Context context)throws IOException{  
                int count =  0; 
                for(IntWritable v : values){    
                    count = count + 1; 
                }    
                try {  
                    context.write(key, new IntWritable(count)); 
                } catch (InterruptedException e) {  
                     e.printStackTrace(); 
                }             
            }         
        }     
        public static void main(String[] args)throws Exception{     
            Configuration conf =  new Configuration();
            //通过配置工程输入参数
            //String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
            //代码中直接输入参数,从hdfs输入（具体数据在hdfs://localhost:9000/user/yhh/input）
            String[] otherArgs=new String[]{
       "input/f3.txt","output"}; 
            if (otherArgs.length != 2) {
              System.err.println("Usage: wordcount <in> <out>");
              System.exit(2);
            }
            //设置每次运行时自动删除输出目录
            Path outputPath = new Path(otherArgs[1]);
            outputPath.getFileSystem(conf).delete(outputPath, true);
            Job job = new Job(conf, "WordCount");
             //1、设置job的基础属性  
            job.setJarByClass(WordCount.class); 
            //2、设置Map与Reudce的类  
            job.setMapperClass(CountMap.class);
            job.setReducerClass(CountReduce.class);
            //3、设置map与reduce的输出键值类型  
            job.setOutputFormatClass(TextOutputFormat.class); 
            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class); 
            //4、设置输入输出路径  
            FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
            FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); 
            //5、运行程序 
            System.exit(job.waitForCompletion(true) ? 0 : 1);
        }
    }

测试成功：  
![这里写图片描述][SouthEast 20]

![这里写图片描述][SouthEast 21]

**参考：**  
大神的帖子：  
[http://www.powerxing.com/hadoop-build-project-using-eclipse/][http_www.powerxing.com_hadoop-build-project-using-eclipse]  
[https://my.oschina.net/132722/blog/168022][https_my.oschina.net_132722_blog_168022]  
[http://blog.csdn.net/jediael\_lu/article/details/43416751][http_blog.csdn.net_jediael_lu_article_details_43416751]  
[http://blog.csdn.net/liyong199012/article/details/25423221][http_blog.csdn.net_liyong199012_article_details_25423221]

[Hadoop_1]: http://blog.csdn.net/yhhyhhyhhyhh/article/details/53945810
[http_download.csdn.net_download_yhhyhhyhhyhh_9726479]: http://download.csdn.net/download/yhhyhhyhhyhh/9726479
[SouthEast]: /images/20220713/bad25be561d94a69a07063ecf0d58850.png
[SouthEast 1]: /images/20220713/29754195ed2446be9f42b822f618323c.png
[SouthEast 2]: /images/20220713/bb18ace3ed1c467d96d13ae474a39c7f.png
[SouthEast 3]: /images/20220713/023a6af051634375bbb8ce23f179aeb1.png
[SouthEast 4]: /images/20220713/4338a5de1da24fb09b5ca1e19e56c9fb.png
[SouthEast 5]: /images/20220713/ac73a55b59034ed2948b4f0ed99d67de.png
[SouthEast 6]: /images/20220713/2cdb63fa3a31445f8398a98f94453567.png
[SouthEast 7]: /images/20220713/d6dd9c3e695d400e992913d2bf5d2d41.png
[SouthEast 8]: /images/20220713/3784b9d19c604ba9b0b7fb11371f4932.png
[SouthEast 9]: /images/20220713/32f2e39ae76e401286a07eeb0d763acf.png
[SouthEast 10]: /images/20220713/25492f72235c4a959cf51dfadf7dcf47.png
[SouthEast 11]: /images/20220713/d28a520e3791433c83d448609fa0a371.png
[SouthEast 12]: /images/20220713/2c1905c48b244d19b311c16761aa82dd.png
[SouthEast 13]: /images/20220713/6aca3fe53e8a42e48cab75846ff7494f.png
[SouthEast 14]: /images/20220713/5a8d4b4b634a4d2e962a128bbd9c6349.png
[SouthEast 15]: /images/20220713/6c0f2de9921a41a48fa4029af0e4deeb.png
[SouthEast 16]: /images/20220713/9d05fe3960cc494c896c20abf2de5c30.png
[SouthEast 17]: /images/20220713/9cadc5df25304076b53cf0de286cf127.png
[SouthEast 18]: /images/20220713/fbc3c23c416b4947b3d169bd83be46f2.png
[SouthEast 19]: /images/20220713/9a4278fca1be41e898bfb7608ca66049.png
[SouthEast 20]: /images/20220713/72a3c6745b4146669237a6f344830219.png
[SouthEast 21]: /images/20220713/9f6f2a5ad46b4639b843409cd93fd4e0.png
[http_www.powerxing.com_hadoop-build-project-using-eclipse]: http://www.powerxing.com/hadoop-build-project-using-eclipse/
[https_my.oschina.net_132722_blog_168022]: https://my.oschina.net/132722/blog/168022
[http_blog.csdn.net_jediael_lu_article_details_43416751]: http://blog.csdn.net/jediael_lu/article/details/43416751
[http_blog.csdn.net_liyong199012_article_details_25423221]: http://blog.csdn.net/liyong199012/article/details/25423221