YARN的Memory和CPU调优配置详解

叁歲伎倆 2022-06-06 02:36 98阅读 0赞

Hadoop YARN同时支持内存和CPU两种资源的调度，本文介绍如何配置YARN对内存和CPU的使用。

YARN作为一个资源调度器，应该考虑到集群里面每一台机子的计算资源，然后根据application申请的资源进行分配Container。Container是YARN里面资源分配的基本单位，具有一定的内存以及CPU资源。

在YARN集群中，平衡内存、CPU、磁盘的资源的很重要的，根据经验，每两个container使用一块磁盘以及一个CPU核的时候可以使集群的资源得到一个比较好的利用。

# 内存配置 #

关于内存相关的配置可以参考hortonwork公司的文档[Determine HDP Memory Configuration Settings][]来配置你的集群。

> YARN以及MAPREDUCE所有可用的内存资源应该要除去系统运行需要的以及其他的hadoop的一些程序，总共保留的内存=系统内存+HBASE内存。

可以参考下面的表格确定应该保留的内存：

<table style="word-wrap:break-word; font-family:宋体,Arial; font-size:12px; color:rgb(51,51,51)"> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word"> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 每台机子内存</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 系统需要的内存</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> HBase需要的内存</th> 
  </tr> 
 </tbody> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 4GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 2GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 16GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 2GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 2GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 24GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 4GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 4GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 48GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 6GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 64GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 72GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 96GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 12GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 16GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 128GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 24GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 24GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 255GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 32GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 32GB</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 512GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 64GB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 64GB</td> 
  </tr> 
 </tbody> 
</table>

计算每台机子最多可以拥有多少个container，可以使用下面的公式:

containers = min (2\*CORES, 1.8\*DISKS, (Total available RAM) / MIN\_CONTAINER\_SIZE)

说明：

*  CORES为机器CPU核数
 *  DISKS为机器上挂载的磁盘个数
 *  Total available RAM为机器总内存
 *  MIN\_CONTAINER\_SIZE是指container最小的容量大小，这需要根据具体情况去设置，可以参考下面的表格：

每个container的平均使用内存大小计算方式为：

RAM-per-container = max(MIN\_CONTAINER\_SIZE, (Total Available RAM) / containers))

通过上面的计算，YARN以及MAPREDUCE可以这样配置：

<table style="word-wrap:break-word; font-family:宋体,Arial; font-size:12px; color:rgb(51,51,51)"> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word"> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 配置文件</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 配置设置</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 默认值</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 计算值</th> 
  </tr> 
 </tbody> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.nodemanager.resource.memory-mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8192 MB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = containers * RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.scheduler.minimum-allocation-mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1024MB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.scheduler.maximum-allocation-mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 8192 MB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = containers * RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml (check)</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.app.mapreduce.am.resource.mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1536 MB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 2 * RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml (check)</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.app.mapreduce.am.command-opts</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> -Xmx1024m</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 0.8 * 2 * RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.map.memory.mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1024 MB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.reduce.memory.mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> 1024 MB</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 2 * RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.map.java.opts</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> &nbsp;</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 0.8 * RAM-per-container</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.reduce.java.opts</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> &nbsp;</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 0.8 * 2 * RAM-per-container</td> 
  </tr> 
 </tbody> 
</table>

举个例子：对于128G内存、32核CPU的机器，挂载了7个磁盘，根据上面的说明，系统保留内存为24G，不适应HBase情况下，系统剩余可用内存为104G，计算containers值如下：

containers = min (2\*32, 1.8\* 7 , (128-24)/2) = min (64, 12.6 , 51) = 13

计算RAM-per-container值如下：

RAM-per-container = max (2, (124-24)/13) = max (2, 8) = 8

你也可以使用脚本[yarn-utils.py][]来计算上面的值：

点击(此处)折叠或打开

1.  \#!/usr/bin/env python  
    
2.  import optparse  
    
3.  from pprint import pprint  
    
4.  import logging  
    
5.  import sys  
    
6.  import math  
    
7.  import ast  
    
8.    
    
9.  ''' Reserved for OS + DN + NM, Map: Memory => Reservation '''  
    
10. reservedStack = \{  4:1, 8:2, 16:2, 24:4, 48:6, 64:8, 72:8, 96:12,   
    
11.                    128:24, 256:32, 512:64\}  
    
12. ''' Reserved for HBase. Map: Memory => Reservation '''  
    
13.     
    
14. reservedHBase = \{ 4:1, 8:1, 16:2, 24:4, 48:8, 64:8, 72:8, 96:16,   
    
15.                    128:24, 256:32, 512:64\}  
    
16. GB = 1024  
    
17.   
    
18. def getMinContainerSize(memory):  
    
19.   if (memory <= 4):  
    
20.     return 256  
    
21.   elif (memory <= 8):  
    
22.     return 512  
    
23.   elif (memory <= 24):  
    
24.     return 1024  
    
25.   else:  
    
26.     return 2048  
    
27.   pass  
    
28.   
    
29. def getReservedStackMemory(memory):  
    
30.   if (reservedStack.has\_key(memory)):  
    
31.     return reservedStack\[memory\]  
    
32.   if (memory <= 4):  
    
33.     ret = 1  
    
34.   elif (memory >= 512):  
    
35.     ret = 64  
    
36.   else:  
    
37.     ret = 1  
    
38.   return ret  
    
39.   
    
40. def getReservedHBaseMem(memory):  
    
41.   if (reservedHBase.has\_key(memory)):  
    
42.     return reservedHBase\[memory\]  
    
43.   if (memory <= 4):  
    
44.     ret = 1  
    
45.   elif (memory >= 512):  
    
46.     ret = 64  
    
47.   else:  
    
48.     ret = 2  
    
49.   return ret  
    
50.                       
    
51. def main():  
    
52.   log = logging.getLogger(\_\_name\_\_)  
    
53.   out\_hdlr = logging.StreamHandler(sys.stdout)  
    
54.   out\_hdlr.setFormatter(logging.Formatter(' %(message)s'))  
    
55.   out\_hdlr.setLevel(logging.INFO)  
    
56.   log.addHandler(out\_hdlr)  
    
57.   log.setLevel(logging.INFO)  
    
58.   parser = optparse.OptionParser()  
    
59.   memory = 0  
    
60.   cores = 0  
    
61.   disks = 0  
    
62.   hbaseEnabled = True  
    
63.   parser.add\_option('-c', '--cores', default = 16,  
    
64.                      help = 'Number of cores on each host')  
    
65.   parser.add\_option('-m', '--memory', default = 64,   
    
66.                     help = 'Amount of Memory on each host in GB')  
    
67.   parser.add\_option('-d', '--disks', default = 4,   
    
68.                     help = 'Number of disks on each host')  
    
69.   parser.add\_option('-k', '--hbase', default = "True",  
    
70.                     help = 'True if HBase is installed, False is not')  
    
71.   (options, args) = parser.parse\_args()  
    
72.     
    
73.   cores = int (options.cores)  
    
74.   memory = int (options.memory)  
    
75.   disks = int (options.disks)  
    
76.   hbaseEnabled = ast.literal\_eval(options.hbase)  
    
77.     
    
78.   log.info("Using cores=" \+ str(cores) \+ " memory=" \+ str(memory) \+ "GB" \+  
    
79.             " disks=" \+ str(disks) \+ " hbase=" \+ str(hbaseEnabled))  
    
80.   minContainerSize = getMinContainerSize(memory)  
    
81.   reservedStackMemory = getReservedStackMemory(memory)  
    
82.   reservedHBaseMemory = 0  
    
83.   if (hbaseEnabled):  
    
84.     reservedHBaseMemory = getReservedHBaseMem(memory)  
    
85.   reservedMem = reservedStackMemory \+ reservedHBaseMemory  
    
86.   usableMem = memory \- reservedMem  
    
87.   memory \-= (reservedMem)  
    
88.   if (memory < 2):  
    
89.     memory = 2  
    
90.     reservedMem = max(0, memory \- reservedMem)  
    
91.       
    
92.   memory \*= GB  
    
93.     
    
94.   containers = int (min(2 \* cores,  
    
95.                          min(math.ceil(1.8 \* float(disks)),  
    
96.                               memory/minContainerSize)))  
    
97.   if (containers <= 2):  
    
98.     containers = 3  
    
99.   
    
100.   log.info("Profile: cores=" \+ str(cores) \+ " memory=" \+ str(memory) \+ "MB"  
    
101.            \+ " reserved=" \+ str(reservedMem) \+ "GB" \+ " usableMem="  
    
102.            \+ str(usableMem) \+ "GB" \+ " disks=" \+ str(disks))  
    
103.       
    
104.   container\_ram = abs(memory/containers)  
    
105.   if (container\_ram > GB):  
    
106.     container\_ram = int(math.floor(container\_ram / 512)) \* 512  
    
107.   log.info("Num Container=" \+ str(containers))  
    
108.   log.info("Container Ram=" \+ str(container\_ram) \+ "MB")  
    
109.   log.info("Used Ram=" \+ str(int (containers\*container\_ram/float(GB))) \+ "GB")  
    
110.   log.info("Unused Ram=" \+ str(reservedMem) \+ "GB")  
    
111.   log.info("yarn.scheduler.minimum-allocation-mb=" \+ str(container\_ram))  
    
112.   log.info("yarn.scheduler.maximum-allocation-mb=" \+ str(containers\*container\_ram))  
    
113.   log.info("yarn.nodemanager.resource.memory-mb=" \+ str(containers\*container\_ram))  
    
114.   map\_memory = container\_ram  
    
115.   reduce\_memory = 2\*container\_ram if (container\_ram <= 2048) else container\_ram  
    
116.   am\_memory = max(map\_memory, reduce\_memory)  
    
117.   log.info("mapreduce.map.memory.mb=" \+ str(map\_memory))  
    
118.   log.info("mapreduce.map.java.opts=-Xmx" \+ str(int(0.8 \* map\_memory)) \+"m")  
    
119.   log.info("mapreduce.reduce.memory.mb=" \+ str(reduce\_memory))  
    
120.   log.info("mapreduce.reduce.java.opts=-Xmx" \+ str(int(0.8 \* reduce\_memory)) \+ "m")  
    
121.   log.info("yarn.app.mapreduce.am.resource.mb=" \+ str(am\_memory))  
    
122.   log.info("yarn.app.mapreduce.am.command-opts=-Xmx" \+ str(int(0.8\*am\_memory)) \+ "m")  
    
123.   log.info("mapreduce.task.io.sort.mb=" \+ str(int(0.4 \* map\_memory)))  
    
124.   pass  
    
125.   
    
126. if \_\_name\_\_ == '\_\_main\_\_':  
    
127.   try:  
    
128.     main()  
    
129.   except(KeyboardInterrupt, EOFError):  
    
130.     print("\\nAborting ... Keyboard Interrupt.")  
    
131.     sys.exit(1)

执行下面命令：

python yarn-utils.py -c 32 -m 128 -d 7 -k False

返回结果如下：

点击(此处)折叠或打开

1.  Using cores=32 memory=128GB disks=7 hbase=False  
    
2.   Profile: cores=32 memory=106496MB reserved=24GB usableMem=104GB disks=7  
    
3.   Num Container=13  
    
4.   Container Ram=8192MB  
    
5.   Used Ram=104GB  
    
6.   Unused Ram=24GB  
    
7.   yarn.scheduler.minimum\-allocation\-mb=8192  
    
8.   yarn.scheduler.maximum\-allocation\-mb=106496  
    
9.   yarn.nodemanager.resource.memory\-mb=106496  
    
10.  mapreduce.map.memory.mb=8192  
    
11.  mapreduce.map.java.opts=\-Xmx6553m  
    
12.  mapreduce.reduce.memory.mb=8192  
    
13.  mapreduce.reduce.java.opts=\-Xmx6553m  
    
14.  yarn.app.mapreduce.am.resource.mb=8192  
    
15.  yarn.app.mapreduce.am.command\-opts=\-Xmx6553m  
    
16.  mapreduce.task.io.sort.mb=3276

这样的话，每个container内存为8G，似乎有点多，我更愿意根据集群使用情况任务将其调整为2G内存，则集群中下面的参数配置值如下：

<table style="word-wrap:break-word; font-family:宋体,Arial; font-size:12px; color:rgb(51,51,51)"> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word"> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 配置文件</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 配置设置</th> 
   <th style="word-wrap:break-word; margin:0px; padding:6px; border-width:0px 1px 2px; border-style:none solid solid; text-align:left; vertical-align:bottom"> 计算值</th> 
  </tr> 
 </tbody> 
 <tbody style="word-wrap:break-word"> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.nodemanager.resource.memory-mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 52 * 2 =104 G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.scheduler.minimum-allocation-mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 2G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.scheduler.maximum-allocation-mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 52 * 2 = 104G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml (check)</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.app.mapreduce.am.resource.mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 2 * 2=4G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn-site.xml (check)</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> yarn.app.mapreduce.am.command-opts</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 0.8 * 2 * 2=3.2G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.map.memory.mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 2G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.reduce.memory.mb</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 2 * 2=4G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.map.java.opts</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 0.8 * 2=1.6G</td> 
  </tr> 
  <tr style="word-wrap:break-word"> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapred-site.xml</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> mapreduce.reduce.java.opts</td> 
   <td style="word-wrap:break-word; margin:0px; padding:6px; vertical-align:middle"> = 0.8 * 2 * 2=3.2G</td> 
  </tr> 
 </tbody> 
</table>

对应的xml配置为：

点击(此处)折叠或打开

1.  <property>  
    
2.        <name>yarn.nodemanager.resource.memory-mb</name>  
    
3.        <value>106496</value>  
    
4.    </property>  
    
5.    <property>  
    
6.        <name>yarn.scheduler.minimum-allocation-mb</name>  
    
7.        <value>2048</value>  
    
8.    </property>  
    
9.    <property>  
    
10.       <name>yarn.scheduler.maximum-allocation-mb</name>  
    
11.       <value>106496</value>  
    
12.   </property>  
    
13.   <property>  
    
14.       <name>yarn.app.mapreduce.am.resource.mb</name>  
    
15.       <value>4096</value>  
    
16.   </property>  
    
17.   <property>  
    
18.       <name>yarn.app.mapreduce.am.command-opts</name>  
    
19.       <value>\-Xmx3276m</value>  
    
20.   </property>

另外，还有一下几个参数：

*  yarn.nodemanager.vmem-pmem-ratio：任务每使用1MB物理内存，最多可使用虚拟内存量，默认是2.1。
 *  yarn.nodemanager.pmem-check-enabled：是否启动一个线程检查每个任务正使用的物理内存量，如果任务超出分配值，则直接将其杀掉，默认是true。
 *  yarn.nodemanager.vmem-pmem-ratio：是否启动一个线程检查每个任务正使用的虚拟内存量，如果任务超出分配值，则直接将其杀掉，默认是true。

第一个参数的意思是当一个map任务总共分配的物理内存为2G的时候，该任务的container最多内分配的堆内存为1.6G，可以分配的虚拟内存上限为2\*2.1=4.2G。另外，照这样算下去，每个节点上YARN可以启动的Map数为104/2=52个。

# CPU配置 #

YARN中目前的CPU被划分成虚拟CPU（CPU virtual Core），这里的虚拟CPU是YARN自己引入的概念，初衷是，考虑到不同节点的CPU性能可能不同，每个CPU具有的计算能力也是不一样的，比如某个物理CPU的计算能力可能是另外一个物理CPU的2倍，这时候，你可以通过为第一个物理CPU多配置几个虚拟CPU弥补这种差异。用户提交作业时，可以指定每个任务需要的虚拟CPU个数。

在YARN中，CPU相关配置参数如下：

*  yarn.nodemanager.resource.cpu-vcores：表示该节点上YARN可使用的虚拟CPU个数，默认是8，注意，目前推荐将该值设值为与物理CPU核数数目相同。如果你的节点CPU核数不够8个，则需要调减小这个值，而YARN不会智能的探测节点的物理CPU总数。
 *  yarn.scheduler.minimum-allocation-vcores：单个任务可申请的最小虚拟CPU个数，默认是1，如果一个任务申请的CPU个数少于该数，则该对应的值改为这个数。
 *  yarn.scheduler.maximum-allocation-vcores：单个任务可申请的最多虚拟CPU个数，默认是32。

对于一个CPU核数较多的集群来说，上面的默认配置显然是不合适的，在我的测试集群中，4个节点每个机器CPU核数为31，留一个给操作系统，可以配置为：

点击(此处)折叠或打开

1.  <property>  
    
2.        <name>yarn.nodemanager.resource.cpu-vcores</name>  
    
3.        <value>31</value>  
    
4.    </property>  
    
5.    <property>  
    
6.        <name>yarn.scheduler.maximum-allocation-vcores</name>  
    
7.        <value>124</value>  
    
8.    </property>

[Determine HDP Memory Configuration Settings]: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-11.html
[yarn-utils.py]: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-9.html