阿里云hadoop安装

分手后的思念是犯贱 2021-11-23 01:14 401阅读 0赞

# 一、JAVA安装 #

解压jdk安装包到指定路径

root@hadoop:~# tar -zxvf jdk1.8.0.tar.gz -C /usr

配置环境变量

root@hadoop:~# vim /etc/profile
    #文件底部加入
    export JAVA_HOME=/usr/jdk1.8.0_151
    export JRE_HOME=${ JAVA_HOME}/jre
    export CLASSPATH=.:${ JAVA_HOME}/lib:${ JRE_HOME}/lib
    export PATH=.:${ JAVA_HOME}/bin:$PATH

使环境变量生效

root@hadoop:~# source /etc/profile

查看是否安装成功

root@hadoop:~# java -version
    java version "1.8.0_151"
    Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
    Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

## 二、hadoop安装 ##

官网下载地址：[https://www.apache.org/dist/hadoop/common/][https_www.apache.org_dist_hadoop_common]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70 1]  
解压到指定路径

root@hadoop:~# tar -zxvf hadoop-2.7.6.tar.gz -C /usr
    #重命名
    root@hadoop:/usr# mv hadoop-2.7.6 hadoop

配置环境变量

root@hadoop:~# vim /etc/profile
    #文件底部加入
    export HADOOP_HOME=/usr/hadoop
    export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

使环境变量生效

root@hadoop:~# source /etc/profile

输入hadoop  
出现以下信息则安装成功

root@hadoop:~# hadoop
    Usage: hadoop [--config confdir] [COMMAND | CLASSNAME]
      CLASSNAME            run the class named CLASSNAME
     or
      where COMMAND is one of:
      fs                   run a generic filesystem user client
      version              print the version
      jar <jar>            run a jar file
                           note: please use "yarn jar" to launch
                                 YARN applications, not this command.
      checknative [-a|-h]  check native hadoop and compression libraries availability
      distcp <srcurl> <desturl> copy file or directories recursively
      archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive
      classpath            prints the class path needed to get the
      credential           interact with credential providers
                           Hadoop jar and the required libraries
      daemonlog            get/set the log level for each daemon
      trace                view and modify Hadoop tracing settings
    
    Most commands print help when invoked w/o parameters.

## 三、配置hadoop ##

1、在hosts文件中加入你的阿里云ip地址，内网外网都加，映射名更改  
![在这里插入图片描述][20190623105211325.png]  
2、切换到hadoop中etc/hadoop目录下编辑hadoop-env.sh文件，将其中的JAVA\_HOME改为jdk安装目录，我们的jdk安装目录为/usr/bin/java/jdk1.8.0\_151。修改后如下图所示：

root@hadoop:/usr/hadoop/etc/hadoop# vim hadoop-env.sh
    export JAVA_HOME=/usr/java/jdk1.8.0_151

3、修改hadoop中etc/hadoop/目录下的core-site.xml文件。在 configuration 中添加如下配置

root@hadoop:/usr/hadoop/etc/hadoop# vim core-site.xml
    <configuration>
            <property>
                    <name>fs.defaultFS</name>
                    <value>hdfs://localhost:9000</value>
            </property>
            <property>
                    <name>hadoop.tmp.dir</name>
                    <value>/usr/hadoop/tmp</value>
            </property>
    </configuration>

> 注：localhost改为你的阿里云内网地址，也可为映射名

4、修改hadoop中etc/hadoop目录下的hdfs-site.xml文件。先在hadoop下建立一个tmp文件，用来存放中间数据文件。

root@hadoop:/usr/hadoop/etc/hadoop# vim hdfs-site.xml
    <configuration>
            <property>
                    <name>dfs.replication</name>
                    <value>1</value>
            </property>
            <property>
                    <name>dfs.namenode.name.dir</name>
                    <value>/usr/hadoop/tmp/dfs/name</value>
            </property>
            <property>
                    <name>dfs.datanode.data.dir</name>
                    <value>/usr/hadoop/tmp/dfs/data</value>
            </property>
            <property>
                   <name>dfs.permissions</name>
                   <value>false</value>
            </property>
    </configuration>

5、修改hadoop中etc/hadoop目录下的yarn-site.xml文件。

root@hadoop:/usr/hadoop/etc/hadoop# vim yarn-site.xml
    <configuration>
    
    
            <property>
                    <name>yarn.nodemanager.aux-services</name>
                    <value>mapreduce_shuffle</value>
            </property>
            <property>
                    <name>yarn.nodemanager.env-whitelist</name>
                    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
            </property>
    </configuration>

6、修改hadoop中etc/hadoop目录下的mapred-site.xml文件。

root@hadoop:/usr/hadoop/etc/hadoop# vim mapred-site.xml
    <configuration>
            <property>
                    <name>mapreduce.framework.name</name>
                    <value>yarn</value>
            </property>
    </configuration>

## 四、设置SSH免密登录 ##

现在当我们每次使用ssh localhost访问的时候，都需要输入密码。此时我们可以改成无密码登录。  
输入ssh-keygen 连续敲三个回车  
之后可重新连接已是无密码登录

root@hadoop:~# ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/usera/.ssh/id_rsa): 
    Created directory '/home/usera/.ssh'.
    Enter passphrase (empty for no passphrase): 
    Enter same passphrase again: 
    Your identification has been saved in /home/usera/.ssh/id_rsa.
    Your public key has been saved in /home/usera/.ssh/id_rsa.pub.
    The key fingerprint is:
    39:f2:fc:70:ef:e9:bd:05:40:6e:64:b0:99:56:6e:01 usera@serverA
    The key's randomart image is:
    +--[ RSA 2048]----+
    |          Eo*    |
    |           @ .   |
    |          = *    |
    |         o o .   |
    |      . S     .  |
    |       + .     . |
    |        + .     .|
    |         + . o . |
    |          .o= o. |
    +-----------------+
    切换到/root/.ssh下
    将公钥放到许可文件中
    root@hadoop:~/.ssh# cp id_rsa.pub authorized_keys

## 五、格式化hdfs ##

root@hadoop:/usr/hadoop/bin# hdfs namenode -format

## 六、启动进程 ##

root@hadoop:/usr/hadoop/bin# start-dfs.sh
    #查看进程启动
    root@hadoop:/usr/hadoop/bin# jps
    3306 NameNode
    6215 Jps
    3068 SecondaryNameNode
    2893 DataNode

出现四个进程安装成功

## 七、关闭防火墙 ##

查看防火墙状态：sudo ufw status 关闭防火墙：sudo ufw disable 开启防火墙：sudo ufw enable sudo ufw default deny。  
systemctl stop firewalled  
systemctl disable firewalled  
systemctl status firewalled  
当前禁用不会立即改变防火墙的状态，需要重启才行，所以需要我们手动的关闭一次。

## 八、阿里云开放端口号 ##

![在这里插入图片描述][20190623103223491.png]

## 九、访问 ##

在浏览器中输入http://<hadoop服务器IP地址>:50070  
如:[http://192.168.30.130:50070][http_192.168.30.130_50070] 能看到启动进程页面，说明搭建成功。页面如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70 2]  
输入8088端口号 能看到此页面，说明yarn搭建成功。页面如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70 3]

[https_www.apache.org_dist_hadoop_common]: https://www.apache.org/dist/hadoop/common/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70]: /images/20211122/46d9bc2fdc4b43b0b1ced5cb118ba8cc.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70 1]: /images/20211122/9dc429641d39423e9861cc8159358f96.png
[20190623105211325.png]: /images/20211122/0b9aa384511b44039ac4d14a1a625b30.png
[20190623103223491.png]: /images/20211122/7a7b37b25be44bfca379b954c2025d48.png
[http_192.168.30.130_50070]: http://192.168.30.130:50070
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70 2]: /images/20211122/1b066b8966d348cb8dffa66d24981f1e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTg0MTQ2_size_16_color_FFFFFF_t_70 3]: /images/20211122/b41598347202490795c40054bc2e3e83.png