hadoop 安装系列教程二——伪分布式

清疚 2021-09-21 08:04 338阅读 0赞

1.安装Java jdk

vi /etc/profile

> export JAVA\_HOME=/opt/jdk1.8.0\_251  
> export CLASSPATH=.:$JAVA\_HOME/lib/dt.jar:$JAVA\_HOME/lib/tools.jar  
> export PATH=$PATH:$JAVA\_HOME/bin

source /etc/profile

2.安装hadoop

步骤一：下载 [https://hadoop.apache.org/releases.html][https_hadoop.apache.org_releases.html]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70][]

步骤二：配置环境变量

vi /etc/profile

> export JAVA\_HOME=/opt/jdk1.8.0\_251  
> export HADOOP\_HOME=/opt/hadoop-2.9.2  
> export CLASSPATH=.:$JAVA\_HOME/lib/dt.jar:$JAVA\_HOME/lib/tools.jar  
> export PATH=$PATH:$JAVA\_HOME/bin:$HADOOP\_HOME/bin:$HADOOP\_HOME/sbin

source /etc/profile

测试：配置成功

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 1][]

步骤三：免密登录

(1)生成秘钥

ssh-keygen -t dsa  -f ~/.ssh/id_dsa

(2) 公钥存储

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

测试：ssh localhost

![2020060120210363.png][]

步骤四：修改配置文件，hadoop伪分布模式需要修改5个配置文件，配置文件目录为/opt/hadoop-2.9.2/etc/hadoop

(1)hadoop-env.sh

export JAVA\_HOME=/opt/jdk1.8.0\_251  
(2)core-site.xml

> <configuration>  
> <property>  
> <name>fs.default.name</name>  
> <value>hdfs://127.0.0.1:9000</value>  
> </property>  
> <property>  
> <name>hadoop.tmp.dir</name>  
> <value>/data/hadoop/tmp</value>  
> </property>  
> </configuration>

(3)hdfs-site.xml

> <configuration>  
> <property>  
> <name>dfs.namenode.name.dir</name>  
> <value>file:/data/hadoop/hdfs/name</value>  
> </property>  
> <property>  
> <name>dfs.datanode.name.dir</name>  
> <value>file:/data/hadoop/hdfs/data</value>  
> </property>  
> <property>  
> <name>dfs.replication</name>  
> <value>1</value>  
> </property>  
> </configuration>

(4)修改mapred-site.xml,首先去掉.template后缀，指令：mv mapred-site.xml.template mapred-site.xml

> <configuration>  
> <property>  
> <name>mapreduce.framework.name</name>  
> <value>yarn</value>  
> </property>  
> </configuration>

(5)yarn-site.xml

> <configuration>  
> <property>  
> <name>yarn.nodemanager.aux-services</name>  
> <value>mapreduce\_shuffle</value>  
> </property>  
> </configuration>

步骤五：格式化HDFS文件系统 /opt/hadoop-2.9.2/bin

> ./hdfs namenode -format

步骤六：启动 /opt/hadoop-2.9.2/sbin

> ./start-all.sh

测试是否成功 jps

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 2][]

访问地址：

[http://ip:8088/][http_ip_8088]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 3][]

[http://ip:50070/][http_ip_50070]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 4][]

[https_hadoop.apache.org_releases.html]: https://hadoop.apache.org/releases.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70]: /images/20210920/db36ab43045c458bae8e1079b63eed4c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 1]: /images/20210920/133970608b9448c6964d60eba85c73aa.png
[2020060120210363.png]: /images/20210920/f15693da145340d98d027d58fd324373.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 2]: /images/20210920/f492f9a1e81c4214b4c65b40e2b0cb79.png
[http_ip_8088]: http://122.112.182.244:8088/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 3]: /images/20210920/84c80c157cc548a28d6c73213ed005d7.png
[http_ip_50070]: http://122.112.182.244:50070/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzI5MTA1NQ_size_16_color_FFFFFF_t_70 4]: /images/20210920/4ec5b1858dc34fd99a745843f26acd06.png