Linux下编译Hadoop的Eclipse插件

古城微笑少年丶 2022-04-14 06:41 213阅读 0赞

\[b\]\[color=green\]\[size=x-large\]虽然使用Java编写MapReduce作业有点繁琐，但是对于刚入门hadoop的新手来说，有利于理解hadoop的一些底层实现，这无疑也是一种巨大的收获。目前大部分的MapReduce作业都是使用Hive，Pig完成的，当然也有用其他语言实现的，通过使用HadoopStreaming的方式，用原生Java写MR作业的只有很少一部分。在Hadoop2.x中，已经不支持eclipse的插件了，不过可以在配置的XML文件里，加入远程监控的端口，来监控hadoop集群。\[/size\]\[/color\]\[/b\]  
  
  
\[b\]\[color=olive\]\[size=x-large\]由于1.0之后hadoop已经不自带eclipse的插件包了，所以得需要我们自己编译源码生成插件包，建议在Linux下编译，Windows下可能有时候会报一些莫名其妙的错误，先介绍下散仙的环境，Centos6.4的版本，hadoop1.2.0的版本，  
hadoop的目录在/root/hadoop-1.2.0下面  
eclipse的目录在/root/eclipse下面  
总结一下如下的四步来完成编译eclipse插件的过程  
\[/size\]\[/color\]\[/b\]  
  
\[b\]\[color=blue\]\[size=x-large\]\[table\]  
|步骤|操作  
|1|进入/root/hadoop-1.2.0/src/contrib/eclipse-plugin下面，修改build.xml  
|2|进入/root/hadoop-1.2.0/src/contrib下面，修改build-contrib.xml  
|3|/root/hadoop-1.2.0/src/contrib/eclipse-plugin/META-INF下面，修改MANIFEST.MF文件  
|4|使用shell命令进入/root/hadoop-1.2.0/src/contrib/eclipse-plugin下面，执行ant命令进行构建  
\[/table\]\[/size\]\[/color\]\[/b\]  
  
\[b\]\[color=olive\]\[size=x-large\]下面散仙就来详细介绍下，每一步具体怎么操作，在第一步里我们需要向build.xml里面，加入的内容有  
1,eclipse的根目录.  
2.hadoop的版本号.  
3.hadoop的一些引用包.  
4，在javac里加入 includeantruntime="on".  
内容如下所示：  
\[/size\]\[/color\]\[/b\]

<property name="eclipse.home" location="/root/eclipse"/>
      <property name="version" value="1.2.0"/>
      
      <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  tofile="${build.dir}/lib/commons-cli.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-configuration-1.6.jar"  tofile="${build.dir}/lib/commons-configuration.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar"  tofile="${build.dir}/lib/commons-httpclient.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-lang-2.4.jar"  tofile="${build.dir}/lib/commons-lang.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-core-asl.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-mapper-asl.jar" verbose="true"/>
     
     includeantruntime="on">

\[b\]\[color=olive\]\[size=x-large\]散仙会在下文贴出整个XML，现在我们看第二大步的配置，修改build-contrib.xml，内容如下：\[/size\]\[/color\]\[/b\]

<property name="javac.deprecation" value="on"/>

\[b\]\[color=olive\]\[size=x-large\]第三步的修改MANIFEST.MF内容如下：\[/size\]\[/color\]\[/b\]

Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar

\[b\]\[color=green\]\[size=x-large\]第四步执行命令：截图如下：\[/size\]\[/color\]\[/b\]  
  
\[img\]http://dl2.iteye.com/upload/attachment/0090/9706/575a3a9a-82be-38bc-ad3c-e8c03f6b069b.jpg\[/img\]  
\[b\]\[color=green\]\[size=x-large\]最后，插件类生成的目录，截图如下：\[/size\]\[/color\]  
\[/b\]  
  
\[img\]http://dl2.iteye.com/upload/attachment/0090/9708/77a37c76-dba2-329f-b5f0-fce95ace2d12.jpg\[/img\]  
  
\[b\]\[color=olive\]\[size=x-large\]下面散仙给出所有配置的全部信息，以供参考，下图的是build.xml里面的内容\[/size\]  
\[/color\]\[/b\]

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
    
    
    
                 
    <project default="jar" name="eclipse-plugin">
    
      <import file="../build-contrib.xml"/>
    
    
      <property name="eclipse.home" location="/root/eclipse"/>
      <property name="version" value="1.2.0"/>
    
      <path id="eclipse-sdk-jars">
        <fileset dir="${eclipse.home}/plugins/">
          <include name="org.eclipse.ui*.jar"/>
          <include name="org.eclipse.jdt*.jar"/>
          <include name="org.eclipse.core*.jar"/>
          <include name="org.eclipse.equinox*.jar"/>
          <include name="org.eclipse.debug*.jar"/>
          <include name="org.eclipse.osgi*.jar"/>
          <include name="org.eclipse.swt*.jar"/>
          <include name="org.eclipse.jface*.jar"/>
    
          <include name="org.eclipse.team.cvs.ssh2*.jar"/>
          <include name="com.jcraft.jsch*.jar"/>
        </fileset> 
      </path>
    
      
      <path id="classpath">
        <pathelement location="${build.classes}"/>
        <pathelement location="${hadoop.root}/build/classes"/>
    	 <fileset dir="${hadoop.root}">
            <include name="**/*.jar" />
        </fileset>
        <path refid="eclipse-sdk-jars"/>
      </path>
    
      
      <target name="check-contrib" unless="eclipse.home">
        <property name="skip.contrib" value="yes"/>
        <echo message="eclipse.home unset: skipping eclipse plugin"/>
      </target>
    
     <target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">
        <echo message="contrib: ${name}"/>
        <javac
         encoding="${build.encoding}"
         srcdir="${src.dir}"
         includes="**/*.java"
         destdir="${build.classes}"
         debug="${javac.debug}"
         deprecation="${javac.deprecation}"
    	 includeantruntime="on">
         <classpath refid="classpath"/>
        </javac>
      </target>
    
      
      <target name="jar" depends="compile" unless="skip.contrib">
        <mkdir dir="${build.dir}/lib"/>
      
       
      <copy file="${hadoop.root}/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-cli-${commons-cli.version}.jar"  tofile="${build.dir}/lib/commons-cli.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-configuration-1.6.jar"  tofile="${build.dir}/lib/commons-configuration.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar"  tofile="${build.dir}/lib/commons-httpclient.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/commons-lang-2.4.jar"  tofile="${build.dir}/lib/commons-lang.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-core-asl.jar" verbose="true"/>
    <copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar"  tofile="${build.dir}/lib/jackson-mapper-asl.jar" verbose="true"/>
    
    
      <jar
          jarfile="${build.dir}/hadoop-${name}-${version}.jar"
          manifest="${root}/META-INF/MANIFEST.MF">
          <fileset dir="${build.dir}" includes="classes/ lib/"/>
          <fileset dir="${root}" includes="resources/ plugin.xml"/>
        </jar>
      </target>
    
    </project>

\[b\]\[color=cyan\]\[size=x-large\]build-contrib.xml的核心配置如下：只有一个地方，notepad++，定位截图如下：\[/size\]\[/color\]\[/b\]

\[img\]http://dl2.iteye.com/upload/attachment/0090/9713/3240219a-e2c5-355e-9228-693e6b968a24.jpg\[/img\]  
\[b\]\[color=green\]\[size=x-large\]下面是MANIFEST.MF里面的内容\[/size\]\[/color\]\[/b\]

Manifest-Version: 1.0
    Bundle-ManifestVersion: 2
    Bundle-Name: MapReduce Tools for Eclipse
    Bundle-SymbolicName: org.apache.hadoop.eclipse;singleton:=true
    Bundle-Version: 0.18
    Bundle-Activator: org.apache.hadoop.eclipse.Activator
    Bundle-Localization: plugin
    Require-Bundle: org.eclipse.ui,
     org.eclipse.core.runtime,
     org.eclipse.jdt.launching,
     org.eclipse.debug.core,
     org.eclipse.jdt,
     org.eclipse.jdt.core,
     org.eclipse.core.resources,
     org.eclipse.ui.ide,
     org.eclipse.jdt.ui,
     org.eclipse.debug.ui,
     org.eclipse.jdt.debug.ui,
     org.eclipse.core.expressions,
     org.eclipse.ui.cheatsheets,
     org.eclipse.ui.console,
     org.eclipse.ui.navigator,
     org.eclipse.core.filesystem,
     org.apache.commons.logging
    Eclipse-LazyStart: true
    Bundle-ClassPath: classes/,lib/commons-cli.jar,lib/commons-httpclient.jar,lib/hadoop-core.jar,lib/jackson-mapper-asl.jar,lib/commons-configuration.jar,lib/commons-lang.jar,lib/jackson-core-asl.jar
    Bundle-Vendor: Apache Hadoop

\[b\]\[color=olive\]\[size=x-large\]至此，已经ant编译完毕，截图如下\[/size\]  
\[/color\]\[/b\]  
  
\[img\]http://dl2.iteye.com/upload/attachment/0090/9719/4701c572-0285-37ac-83aa-4436a950854a.jpg\[/img\]  
\[b\]\[color=olive\]\[size=x-large\]启动eclipse后，截图如下：\[/size\]  
\[/color\]\[/b\]  
\[img\]http://dl2.iteye.com/upload/attachment/0090/9725/376c4dd8-8ec9-3744-add8-8f5e7d8f8382.jpg\[/img\]  
  
  
\[b\]\[color=olive\]\[size=x-large\]至此，插件已经完美生成，并运行，因为受上传限制的影响，所以散仙在这里上传不了这个jar包，有需要的朋友们，可以联系散仙。\[/size\]\[/color\]\[/b\]