Hadoop的学习

偏执的太偏执、 2022-12-08 15:45 125阅读 0赞

1、首先从[http://hadoop.apache.org/][http_hadoop.apache.org]下载最新版本的hadoop，关闭防火墙和selinux

2、解压到linux，配置环境变量在/etc/profile

export JAVA_HOME=/mine/jdk1.8.0_201
    export HADOOP_HOME=/mine/hadoop-3.2.0
    export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

3、cd /mine/hadoop-3.3.0/etc/hadoop

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70][]

4、修改其中的一些文件

（1）hadoop-env.sh，修改JAVS\_HOME为具体的路径

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 1][]

（2）core-site.xml

<property>
      <name>fs.defaultFS</name>
      <value>hdfs://hadoop01:9000</value>
    </property>
    
    
    <property>
      <name>hadoop.tmp.dir</name>
      <value>/mine/hadoop-3.2.0/datas</value>
    </property>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 2][]

（3）hdfs-site.xml

<property>
      <name>dfs.replication</name>
      <value>3</value>
    </property>
    
    <property>
      <name>dfs.permissions.enabled</name>
      <value>false</value>
    </property>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 3][]

（4）mapred-site.xml

<property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
    </property>
    <property>
      <name>yarn.app.mapreduce.am.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.map.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
      <name>mapreduce.reduce.env</name>
      <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 4][]

（5）yarn-site.xml

<property>
      <name>yarn.resourcemanager.hostname</name>
      <value>hadoop01</value>
    </property>
    
    
    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
    </property>

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 5][]

5、对namenode节点进行格式化

hdfs namenode -format

6、在sbin目录里找到hadoop-daemon.sh进行执行，

hadoop-daemon.sh start namenode

在[http://hadoop01:9870][http_hadoop01_9870]（HDFS管理界面）里可以看到namenode已经让启动！！！

在别的节点启动datanode：

hadoop-daemon.sh start datanode

也可以使用start-dfs.sh进行启动，但是要配置etc/hadoop里的workers文件：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 6][]

hadoop02
    hadoop03
    hadoop04

![20200920212301925.png][]

start-dfs.sh报错  
处理：  
vim /mine/hadoop-3.3.0/sbin/start-dfs.sh，stop-dfs.sh  
添加以下几行：

HDFS_DATANODE_USER=root
    HDFS_DATANODE_SECURE_USER=hdfs
    HDFS_NAMENODE_USER=root
    HDFS_SECONDARYNAMENODE_USER=root

start-yarn.sh报错  
处理：  
vim /mine/hadoop-3.3.0/sbin/start-yarn.sh，stop-yarn.sh  
添加以下几行：

YARN_RESOURCEMANAGER_USER=root
    HADOOP_SECURE_DN_USER=yarn
    YARN_NODEMANAGER_USER=root

7、hdfs的命令使用

#上传本地文件到hdfs
    hdfs dfs -put /mine/ideaIU-2019.2.4.tar.gz /
    #查看hdfs上文件的文件内容
    hdfs dfs -cat /hello.txt
    #下载hdfs上的文件到本地
    hdfs dfs -get /test /mine/test
    #列出所有的hdfs上/（根）的文件
    hdfs dfs -ls -h /
    #在hdfs上创建文件夹
    hdfs dfs -mkdir -p /a/b/c
    #在hdfs上删除文件或者文件夹
    hdfs dfs -rm -r -f /a/b/c

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 7][]

[http_hadoop.apache.org]: http://hadoop.apache.org/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70]: /images/20221123/86aec254a89c4898ac9fc35c4a780f99.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 1]: /images/20221123/23b44e9bdda546e883f5c8e367c50c12.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 2]: /images/20221123/58da23043b674cb3887c1a84e11292a7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 3]: /images/20221123/7461e469fda94ab5bffd73da6ab64a54.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 4]: /images/20221123/be284a79fee34508be356573381353fb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 5]: /images/20221123/acd903a285e34c3c85241815f853fd85.png
[http_hadoop01_9870]: http://hadoop01:9870
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 6]: /images/20221123/f38af005343b449cbd6d453b1d18d763.png
[20200920212301925.png]: /images/20221123/a1015fc759b84e6081977692830f6964.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d0bDE5OTI_size_16_color_FFFFFF_t_70 7]: /images/20221123/5a1b028a4efc4588bdda0dc1a3486f04.png