HDFS Balancer负载均衡器

短命女 2024-03-24 08:27 4阅读 0赞

#### 文章目录 ####

*  1、背景
 *  2、什么是平衡
 *   *  2.1 每个DataNode的利用率计算
     *  2.2 集群的利用率
     *  2.3 平衡
 *  3、hdfs balancer语法
 *  4、运行一个简单的balance案例
 *   *  4.1 设置平衡数据传输带宽
     *  4.2 执行banalce
 *  5、参考文档

## 1、背景 ##

当我们的hadoop集群运行了一段时间之后，各个`DataNode`上的`数据分布`并`不一定`是`均匀分布`的。**比如说：** 我们向现有集群中添加了一个新的DataNode。

![DataNode数据不均衡][DataNode]

## 2、什么是平衡 ##

> 此处是我自己的一个简单的理解

所谓的平衡指的是 `每个DataNode的利用率` 与 `集群的利用率` 之间相差不超过给定的阈值百分比。**此处的平衡** 指的是各个DataNode之间的平衡，同一个DataNode之间的各个磁盘是不会平衡的。

### 2.1 每个DataNode的利用率计算 ###

![每个DataNode的利用率计算][DataNode 1]  
`DataNode的利用率=` dfs已用的空间 / 分配给dfs的空间。

**注意：** `分配给dfs的空间` 不是磁盘的总空间。

### 2.2 集群的利用率 ###

![集群的利用率][0c71df7845fb4594a1f9fd1bd1d834d3.png]  
`集群的利用率=` 各datanode dfs已使用的空间 / 各datanode总空间

### 2.3 平衡 ###

假设平衡的`阈值`是 5%，集群的利用率是 `37.5`，那么每个节点的利用率在`32.5%`到`42.5%`之间都认为是均衡的。也就是说，极端情况下，DataNode的利用率最大相差`10%`。

## 3、hdfs balancer语法 ##

[hadoopdeploy@hadoop01 ~]$ hdfs balancer --help
    Usage: hdfs balancer
    	[-policy <policy>]	the balancing policy: datanode or blockpool
    	[-threshold <threshold>]	Percentage of disk capacity
    	[-exclude [-f <hosts-file> | <comma-separated list of hosts>]]	Excludes the specified datanodes.
    	[-include [-f <hosts-file> | <comma-separated list of hosts>]]	Includes only the specified datanodes.
    	[-source [-f <hosts-file> | <comma-separated list of hosts>]]	Pick only the specified datanodes as source nodes.
    	[-blockpools <comma-separated list of blockpool ids>]	The balancer will only run on blockpools included in this list.
    	[-idleiterations <idleiterations>]	Number of consecutive idle iterations (-1 for Infinite) before exit.
    	[-runDuringUpgrade]	Whether to run the balancer during an ongoing HDFS upgrade.This is usually not desired since it will not affect used space on over-utilized machines.
    	[-asService]	Run as a long running service.
    
    Generic options supported are:
    -conf <configuration file>        specify an application configuration file
    -D <property=value>               define a value for a given property
    -fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
    -jt <local|resourcemanager:port>  specify a ResourceManager
    -files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
    -libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
    -archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machines
    
    The general command line syntax is:
    command [genericOptions] [commandOptions]

<table> 
 <thead> 
  <tr> 
   <th align="left">参数</th> 
   <th align="left">描述</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="left">threshold</td> 
   <td align="left">磁盘容量的百分比。默认值为10%，表示上下浮动10%。</td> 
  </tr> 
  <tr> 
   <td align="left">policy</td> 
   <td align="left">平衡策略。<br> datanode（默认）：当每一个DataNode是平衡的时候，集群就是平衡的。<br>blockpool：当每一个DataNode中的blockpool是平衡的，集群就是平衡的。</td> 
  </tr> 
  <tr> 
   <td align="left">exclude</td> 
   <td align="left">不参与平衡的DataNode节点</td> 
  </tr> 
  <tr> 
   <td align="left">include</td> 
   <td align="left">参与平衡的DataNode节点</td> 
  </tr> 
  <tr> 
   <td align="left">source</td> 
   <td align="left">仅选取指定的数据节点作为源节点</td> 
  </tr> 
  <tr> 
   <td align="left">blockpools</td> 
   <td align="left">Balancer仅在指定的blockpools中运行</td> 
  </tr> 
  <tr> 
   <td align="left">idleiterations</td> 
   <td align="left">退出前的连续空闲迭代次数（-1表示无限）</td> 
  </tr> 
  <tr> 
   <td align="left">-runDuringUpgrade</td> 
   <td align="left">是否在正在进行的HDFS升级过程中运行平衡器。通常不需要这样做，因为这不会影响过度使用的计算机上的已用空间。</td> 
  </tr> 
  <tr> 
   <td align="left">-asService</td> 
   <td align="left">作为长期运行的服务运行</td> 
  </tr> 
 </tbody> 
</table>

## 4、运行一个简单的balance案例 ##

### 4.1 设置平衡数据传输带宽 ###

[hadoopdeploy@hadoop01 ~]$ hdfs dfsadmin  -setBalancerBandwidth 10485760
    Balancer bandwidth is set to 10485760
    [hadoopdeploy@hadoop01 ~]$

当我们的集群负载需要调低这个值，当我们的集群负载较低时，可以适当调高这个值。

### 4.2 执行banalce ###

[hadoopdeploy@hadoop01 ~]$ hdfs balancer -policy datanode -threshold 5
    2023-03-26 14:10:09,785 INFO balancer.Balancer: Using a threshold of 5.0
    2023-03-26 14:10:09,786 INFO balancer.Balancer: namenodes  = [hdfs://hadoop01:8020]
    2023-03-26 14:10:09,786 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
    2023-03-26 14:10:09,786 INFO balancer.Balancer: included nodes = []
    2023-03-26 14:10:09,786 INFO balancer.Balancer: excluded nodes = []
    2023-03-26 14:10:09,786 INFO balancer.Balancer: source nodes = []
    Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved  NameNode
    2023-03-26 14:10:09,787 INFO balancer.NameNodeConnector: getBlocks calls for hdfs://hadoop01:8020 will be rate-limited to 20 per second
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.namenode.get-blocks.max-qps = 20 (default=20)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 100 (default=100)
    2023-03-26 14:10:10,392 INFO balancer.Balancer: dfs.datanode.balance.bandwidthPerSec = 104857600 (default=104857600)
    2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
    2023-03-26 14:10:10,395 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)
    2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.141:9866
    2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.140:9866
    2023-03-26 14:10:10,401 INFO net.NetworkTopology: Adding a new node: /default-rack/192.168.121.142:9866
    2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 over-utilized: []
    2023-03-26 14:10:10,402 INFO balancer.Balancer: 0 underutilized: []
    2023-3-26 14:10:10                0                  0 B                 0 B                0 B                  0  hdfs://hadoop01:8020
    The cluster is balanced. Exiting...
    2023-3-26 14:10:10       Balancing took 810.0 milliseconds
    [hadoopdeploy@hadoop01 ~]$

## 5、参考文档 ##

1、[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html\#Balancer][https_hadoop.apache.org_docs_current_hadoop-project-dist_hadoop-hdfs_HdfsUserGuide.html_Balancer]  
2、[https://help.aliyun.com/document\_detail/449686.html][https_help.aliyun.com_document_detail_449686.html]

[DataNode]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/1f15bc3726b74ee89f715a3efb5edb31.png
[DataNode 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/74611f339a914e4a836978b69138f697.png
[0c71df7845fb4594a1f9fd1bd1d834d3.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/24/db169dba48ab4d259376c7fab6f105b2.png
[https_hadoop.apache.org_docs_current_hadoop-project-dist_hadoop-hdfs_HdfsUserGuide.html_Balancer]: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
[https_help.aliyun.com_document_detail_449686.html]: https://help.aliyun.com/document_detail/449686.html