hadoop编译 绝地灬酷狼 2022-06-07 04:41 119阅读 0赞 1、检查是否安装过,是否有残留,检查hosts配置 ps -ef | grep hadoop find / -name hadoop \[hadoop@hadoop001 ~\]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.137.11 hadoop001 192.168.137.12 hadoop002 192.168.137.13 hadoop003 2、创建目录 在/opt 下 mkdir sourcecode 和 software \[root@hadoop001 ~\]\# mkdir sourcecode software \[root@hadoop001 opt\]\# cd /opt/sourcecode/ ======================================以下为hadoop编译======================================================= 3、上传hadoop-2.8.1-src.tar.gz包 \[root@hadoop001 sourcecode\]\# ll total 33720 drwxr-xr-x. 17 root root 4096 Jun 2 14:13 hadoop-2.8.1-src \-rw-r--r--. 1 root root 34523353 Aug 20 2017 hadoop-2.8.1-src.tar.gz \[root@hadoop001 sourcecode\]\# tar xzvf hadoop-2.8.1-src.tar.gz 可以在github上看hadoop源码 搜hadoop 在building.txt 文件中会让我们安装相关的软件: java maven 相关的gcc等包 protobuf 4、下载JDK 并 配置环境变量 将jdk 下载到/usr/java 下,如果java不存在,则创建 \[root@hadoop001 java\]\# tar -xzvf jdk-8u144-linux-x64.tar.gz 修改用户和用户组 \[root@hadoop001 java\]\# chown -R root.root jdk1.8.0\_144/ \[root@hadoop001 java\]\# ll total 181172 drwxr-xr-x. 8 root root 4096 Jul 22 13:11 jdk1.8.0\_144 \[root@hadoop001 java\]\# vi /etc/profile 在最下面添加: export JAVA\_HOME=/usr/java/jdk1.8.0\_144 export PATH=$JAVA\_HOME/bin:$PATH 保存后source \[root@hadoop001 java\]\# source /etc/profile \[root@hadoop001 java\]\# java -version java version "1.8.0\_144" Java(TM) SE Runtime Environment (build 1.8.0\_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) 5、安装maven包 并 配环境变量 安装maven 3.3.30,放在/opt/software/下 \[root@hadoop001 software\]\# ls \[root@hadoop001 software\]\# rz rz waiting to receive. 开始 zmodem 传输。 按 Ctrl+C 取消。 100% 8415 KB 8415 KB/s 00:00:01 0 Errors 解压: \[root@hadoop001 software\]\# unzip apache-maven-3.3.9-bin.zip 添加环境变量: \[root@hadoop001 software\]\# vi /etc/profile export MAVEN\_HOME=/opt/software/apache-maven-3.3.9 在PATH中添加,添加后为: export PATH=$MAVEN\_HOME/bin:$JAVA\_HOME/bin:$PATH \[root@hadoop001 software\]\# source /etc/profile \[root@hadoop001 software\]\# mvn -version Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) Maven home: /opt/software/apache-maven-3.3.9 Java version: 1.8.0\_144, vendor: Oracle Corporation Java home: /usr/java/jdk1.8.0\_144/jre Default locale: en\_US, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-431.el6.x86\_64", arch: "amd64", family: "unix" 6、安装protobuf-2.5.0.tar.gz 并 配环境变量(protobuf下载地址:https://code.google.com/p/protobuf/downloads/list) 上传到/opt/software 解压: \[root@hadoop001 software\]\# tar xzvf protobuf-2.5.0.tar.gz \[root@hadoop001 software\]\# chown -R root.root protobuf-2.5.0/ 安装gcc gcc-c++ make cmake 等一系列的依赖包 \[root@hadoop001 software\]\# yum install -y gcc gcc-c++ make cmake \[root@hadoop001 software\]\# yum -y install autoconf automake libtool curl make g++ unzip 进入protobuf \[root@hadoop001 software\]\# cd protobuf-2.5.0/ 进行编译: \[root@hadoop001 protobuf-2.5.0\]\# ./configure --prefix=/usr/local/protobuf \[root@hadoop001 protobuf-2.5.0\]\# make && make install make 以后会出现文件 \[root@hadoop001 local\]\# ll -d protobuf/ drwxr-xr-x. 5 root root 4096 Aug 17 18:32 protobuf/ 配环境变量 export PROTOC\_HOME=/usr/local/protobuf 在PATH中添加,添加后为: export PATH=$PROTOC\_HOME/bin:$MAVEN\_HOME/bin:$JAVA\_HOME/bin:$PATH \[root@hadoop001 protobuf-2.5.0\]\# source /etc/profile \[root@hadoop001 protobuf-2.5.0\]\# protoc --version libprotoc 2.5.0 7、安装findbugs 并 配置环境变量 在/opt/software 中上传软件 findbugs-1.3.9.zip(下载地址:http://findbugs.sourceforge.net/downloads.html) \[root@hadoop001 software\]\# unzip findbugs-1.3.9.zip 配置环境变量 export FINDBUGS\_HOME=/opt/software/findbugs-1.3.9 在PATH中添加,添加后为: export PATH=$FINDBUGS\_HOME/bin:$PROTOC\_HOME/bin:$MAVEN\_HOME/bin:$JAVA\_HOME/bin:$PATH \[root@hadoop001 software\]\# source /etc/profile \[root@hadoop001 software\]\# findbugs -version 1.3.9 8、安装其他依赖包 yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake 这里遇到lzo lzo-devel lzop 这三个包在CentOS6.5的另一个ISO里,这里我手动下载了2个包 \[root@hadoop001 mnt\]\# rpm -ivh lzo-minilzo-2.03-3.1.el6.x86\_64.rpm warning: lzo-minilzo-2.03-3.1.el6.x86\_64.rpm: Header V4 DSA/SHA1 Signature, key ID 192a7d7d: NOKEY Preparing... \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \[100%\] 1:lzo-minilzo \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \[100%\] \[root@hadoop001 mnt\]\# yum install -y lzo-devel-2.03-3.1.el6.x86\_64.rpm \[root@hadoop001 mnt\]\# yum install -y lzo lzo-devel lzop 再次安装,确认所有包都安装了,一定要做这一步! yum install -y snappy snappy-devel bzip2 bzip2-devel lzo lzo-devel lzop autoconf automake 9、编译 保证能ping通外网 进入/opt/sourcecode/hadoop-2.8.1-src \[root@hadoop001 hadoop-2.8.1-src\]\# pwd /opt/sourcecode/hadoop-2.8.1-src \[root@hadoop001 hadoop-2.8.1-src\]\# mvn clean package -Pdist,native -DskipTests -Dtar 根据网速,可能要1小时及以上。。。。 开始:21:00 Last login: Thu Aug 17 19:06:52 2017 from 192.168.137.1 结束: 提醒: 1、有时候编译过程中会出现下载某个包的时间太久,这是由于连接网站的过程中会出现假死, 此时按ctrl+c,重新运行编译命令。 2、如果出现缺少了某个文件的情况,则要先清理maven(使用命令 mvn clean) 再重新编译。 10、打包后,最终文件在此目录下 /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz 遇到的坑: 1.jdk1.7版本 2. ![这里写图片描述][SouthEast] 执行: mvn clean validate -Denforcer.skip=true 看到 BUILD SSUCCESS 后执行下面的命令: mvn clean package -Denforcer.skip=true -DskipTests -Pdist,native -Dtar 最后就会编译成功 ![Image 1][] ![Image 1][] [SouthEast]: /images/20220607/41e9cc7365754f0e8f2291f235565ba8.png [Image 1]:
还没有评论,来说两句吧...