【Hadoop】HDFS的运行原理

古城微笑少年丶 2022-07-15 05:58 137阅读 0赞

**简介**

**HDFS**（Hadoop Distributed File System ）Hadoop分布式文件系统。是根据google发表的论文翻版的。论文为GFS（Google File System）Google 文件系统（[中文][Link 1]，[英文][Link 2]）。

**HDFS有很多特点**：

**①** 保存多个副本，且提供容错机制，副本丢失或宕机自动恢复。默认存3份。

**②** 运行在廉价的机器上。

**③** 适合大数据的处理。多大？多小？HDFS默认会将文件分割成block，64M为1个block。然后将block按键值对存储在HDFS上，并将键值对的映射存到内存中。如果小文件太多，那内存的负担会很重。

![26104230-8109ac513de14fe1b115b775581751f2.jpg][]

如上图所示，HDFS也是按照Master和Slave的结构。分NameNode、SecondaryNameNode、DataNode这几个角色。

***NameNode***：是Master节点，是大领导。管理数据块映射；处理客户端的读写请求；配置副本策略；管理HDFS的名称空间；

***SecondaryNameNode***：是一个小弟，分担大哥namenode的工作量；是NameNode的冷备份；合并fsimage和fsedits然后再发给namenode。

***DataNode**：Slave*节点，奴隶，干活的。负责存储client发来的数据块block；执行数据块的读写操作。

***热备份***：b是a的热备份，如果a坏掉。那么b马上运行代替a的工作。

***冷备份***：b是a的冷备份，如果a坏掉。那么b不能马上代替a工作。但是b上存储a的一些信息，减少a坏掉之后的损失。

***fsimage***:元数据镜像文件（文件系统的目录树。）

***edits***：元数据的操作日志（针对文件系统做的修改操作记录）

**namenode内存中存储的是=fsimage+edits。**

SecondaryNameNode负责定时默认1小时，从namenode上，获取fsimage和edits来进行合并，然后再发送给namenode。减少namenode的工作量。

--------------------

**工作原理**

**写操作：**

![26162921-2de9d28df9b54fe6a97a6fd88f1cb03f.jpg][]

有一个文件FileA，100M大小。Client将FileA写入到HDFS上。

HDFS按默认配置。

HDFS分布在三个机架上Rack1，Rack2，Rack3。

**a.** Client将FileA按64M分块。分成两块，block1和Block2;

**b.** Client向nameNode发送写数据请求，如图蓝色虚线①\------>。

**c.** NameNode节点，记录block信息。并返回可用的DataNode，如粉色虚线②\--------->。

Block1: host2,host1,host3

Block2: host7,host8,host4

原理：

NameNode具有RackAware机架感知功能，这个可以配置。

若client为DataNode节点，那存储block时，规则为：副本1，同client的节点上；副本2，不同机架节点上；副本3，同第二个副本机架的另一个节点上；其他副本随机挑选。

若client不为DataNode节点，那存储block时，规则为：副本1，随机选择一个节点上；副本2，不同副本1，机架上；副本3，同副本2相同的另一个节点上；其他副本随机挑选。

**d.** client向DataNode发送block1；发送过程是以流式写入。

流式写入过程，

**1>**将64M的block1按64k的package划分;

**2>**然后将第一个package发送给host2;

**3>**host2接收完后，将第一个package发送给host1，同时client想host2发送第二个package；

**4>**host1接收完第一个package后，发送给host3，同时接收host2发来的第二个package。

**5>**以此类推，如图红线实线所示，直到将block1发送完毕。

**6>**host2,host1,host3向NameNode，host2向Client发送通知，说“消息发送完了”。如图粉红颜色实线所示。

**7>**client收到host2发来的消息后，向namenode发送消息，说我写完了。这样就真完成了。如图黄色粗实线

**8>**发送完block1后，再向host7，host8，host4发送block2，如图蓝色实线所示。

**9>**发送完block2后，host7,host8,host4向NameNode，host7向Client发送通知，如图浅绿色实线所示。

**10>**client向NameNode发送消息，说我写完了，如图黄色粗实线。。。这样就完毕了。

**分析，**通过写过程，我们可以了解到：

**①**写1T文件，我们需要3T的存储，3T的网络流量贷款。

**②**在执行读或写的过程中，NameNode和DataNode通过HeartBeat进行保存通信，确定DataNode活着。如果发现DataNode死掉了，就将死掉的DataNode上的数据，放到其他节点去。读取时，要读其他节点去。

**③**挂掉一个节点，没关系，还有其他节点可以备份；甚至，挂掉某一个机架，也没关系；其他机架上，也有备份。

**读操作：**

![26163017-fc613879835c402886b75e0593ca52ed.jpg][]

读操作就简单一些了，如图所示，client要从datanode上，读取FileA。而FileA由block1和block2组成。

那么，读操作流程为：

**a.** client向namenode发送读请求。

**b.** namenode查看Metadata信息，返回fileA的block的位置。

block1:host2,host1,host3

block2:host7,host8,host4

**c.** block的位置是有先后顺序的，先读block1，再读block2。而且block1去host2上读取；然后block2，去host7上读取；

上面例子中，client位于机架外，那么如果client位于机架内某个DataNode上，例如,client是host6。那么读取的时候，遵循的规律是：

**优选读取本机架上的数据**。

--------------------

**HDFS中常用到的命令**

1、hadoop fs

<table style="border:1px solid silver; width:790px; border-collapse:collapse; word-break:break-word; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; font-size:12px!important; min-height:auto!important; background:none!important"> 
 <tbody style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; background:none!important"> 
  <tr style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; background:none!important"> 
   <td style="border:1px solid silver; padding:3px; border-collapse:collapse; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:35px!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; color:rgb(175,175,175)!important; background:none!important"> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      1 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      2 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      3 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      4 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      5 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      6 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      7 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      8 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      9 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      10 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      11 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      12 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      13 
    </div> </td> 
   <td style="border:1px solid silver; padding:3px; border-collapse:collapse; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important"> 
    <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:relative!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; background:none!important"> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -ls /</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -lsr</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -mkdir /user/hadoop</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -put a.txt /user/hadoop/</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -get /user/hadoop/a.txt /</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -cp src dst</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -mv src dst</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -cat /user/hadoop/a.txt</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -rm /user/hadoop/a.txt</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -rmr /user/hadoop/a.txt</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -text /user/hadoop/a.txt</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -copyFromLocal localsrc dst 与hadoop fs -put功能类似。</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop fs -moveFromLocal localsrc dst 将本地文件上传到hdfs，同时删除本地文件。</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

2、hadoop fsadmin

<table style="border:1px solid silver; width:790px; border-collapse:collapse; word-break:break-word; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; font-size:12px!important; min-height:auto!important; background:none!important"> 
 <tbody style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; background:none!important"> 
  <tr style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; background:none!important"> 
   <td style="border:1px solid silver; padding:3px; border-collapse:collapse; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:35px!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; color:rgb(175,175,175)!important; background:none!important"> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      1 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important">
      2 
    </div> 
    <div style="border-width:0px 2px 0px 0px!important; border-right-style:solid!important; border-right-color:rgb(108,226,108)!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 0.5em!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important">
      3 
    </div> </td> 
   <td style="border:1px solid silver; padding:3px; border-collapse:collapse; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important"> 
    <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.1em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:relative!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; background:none!important"> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop dfsadmin -report</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop dfsadmin -safemode enter | leave | get | wait</code> 
     </div> 
     <div style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px 1em!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:nowrap!important; background:none rgb(244,244,244)!important"> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; background:none!important">hadoop dfsadmin -setBalancerBandwidth&nbsp;</code> 
      <code style="border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.8em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,'Bitstream Vera Sans Mono','Courier New',Courier,monospace!important; min-height:auto!important; color:rgb(0,153,0)!important; background:none!important">1000</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

3、hadoop fsck

4、start-balancer.sh

[Link 1]: http://www.open-open.com/lib/view/open1328763454608.html
[Link 2]: http://static.googleusercontent.com/media/research.google.com/zh-CN//archive/gfs-sosp2003.pdf
[26104230-8109ac513de14fe1b115b775581751f2.jpg]: /images/20220715/b2edfd5568f64102aed638cd36564074.png
[26162921-2de9d28df9b54fe6a97a6fd88f1cb03f.jpg]: /images/20220715/5aa39b2ef83c4847b15b0045091ffc9b.png
[26163017-fc613879835c402886b75e0593ca52ed.jpg]: /images/20220715/43a51803adaa42a88e1a3fe272b4634c.png

发表评论取消回复

表情：

评论列表（有 0 条评论，137人围观）

还没有评论，来说两句吧...

相关阅读

相关 Servlet 的运行原理

系列文章目录 [如何创建一个Servlet项目（Maven）？\_crazy\_xieyi的博客-CSDN博客][Servlet_Maven_crazy_xieyi_-C

r囧r小猫/ 2024年04月06日 12:03/ 0 赞/ 21 阅读

相关 servlet的运行原理

Servlet在容器中的执行过程 ![c0c08d0ba7564efda36ffb04fa7a133d.png][] 1.浏览器向服务器发出GET请求 2.服务器上的

古城微笑少年丶/ 2023年10月11日 23:47/ 0 赞/ 22 阅读

相关 servlet的运行原理

Servlet在容器中的执行过程 ![c37deb6af3f663e6182562494e7bda1c.png][] 1.浏览器向服务器发出GET请求 2.服务器上的

雨点打透心脏的1/2处/ 2023年10月09日 23:13/ 0 赞/ 7 阅读

相关 babel原理_Babel运行原理

前言之前翻博客园的时候,看到有人朋友分享阿里巴巴的面试题,其中有一道题就是关于ES6转ES5 原理的,当时我看到感觉到自己离去阿里巴巴的路还很远啊,像我们大部分做开发的时

朴灿烈づ我的快乐病毒、/ 2022年12月31日 15:21/ 0 赞/ 195 阅读

相关 Sea.Js的运行原理

Sea.Js的运行原理 1.CMD(Common Module Definition)规范 Sea.js采用了和Node相似的CMD规范，使用require、exports

深碍√TFBOYSˉ_/ 2022年06月08日 03:24/ 0 赞/ 112 阅读

相关 java程序的运行原理

java命令会启动java虚拟机（jvm），jvm启动就相当于启动了一个应用程序，进而也就相当于启动了一个进程，该进程会主动启动一个“主线程”，主线程会调用某个类的main方法

电玩女神/ 2022年05月18日 01:46/ 0 赞/ 190 阅读

相关 Maven的内部运行原理

maven至今还是Java编程语言构建的事实标准，大部分项目还在使用maven来进行构建，因此了解maven内部运行的原理对定位和分析问题还是很有裨益的。本篇文章主要介绍一

╰+攻爆jí腚メ/ 2022年04月10日 06:19/ 0 赞/ 207 阅读

相关 SpringMVC的运行原理

1、客户端发送一个请求到web应用时请求首先会经过SpringMVC的核心控制器DispatcherServlet。 2、DispatcherServlet会调用处理器映射H

约定不等于承诺〃/ 2022年02月04日 12:53/ 0 赞/ 229 阅读

相关 Mybatis的SqlSession运行原理

1、SqlSession简单介绍　　（1）SqlSession简单原理介绍　　SqlSession提供select/insert/update/delete方法，在

Myth丶恋晨/ 2022年02月01日 06:13/ 0 赞/ 201 阅读

相关 Shiro的运行原理

前言： ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L

﹏ヽ暗。殇╰゛Y/ 2021年12月07日 12:51/ 0 赞/ 248 阅读