CDH 中hbaese, mapreduce内存控制

青旅半醒 2022-03-17 11:56 186阅读 0赞

# 1, hbase启动失败：OutOfMemoryError #

由于内存不够，导入大批量数据时发生内存溢出，所以需要调整内存分配，在浏览器打开7180集群管理界面，进入到hbase

## 1, 配置–>输入框搜索：java ##

*  优化值： master堆大小–> 4G
 *  优化值： regionserver堆大小–> 15G  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]

# 2，mapreduce运行卡顿 #

查看/var/log/ hadoop-cmf-yarn-JOBHISTORY-node1.cluster.log.out ,提示gc频繁，所以想到给mapreduce分配的内存不足, 需要加大内存

*  nodemanager.resource.memory-mb = scheduler.maximum-allocation-mb =60G
 *  优化值： map堆大小–> 10G, map容器内存–>12G
 *  优化值： reduce堆大小–> 20G, reduce容器内存–>24G

2019-02-23 08:00:21,593 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[JOBHISTORY], CATEGORY=[LOG_MESSAGE], ROLE=[yarn-JOBHISTORY-a6c222f47368c75014f30b4eb3a45e95], SEVERITY=[IMPORTANT], SERVICE=[yarn], HOST_IDS=[f64f5e51-45af-4278-88e1-b18ccaea3de6], SERVICE_TYPE=[YARN], LOG_LEVEL=[WARN], HOSTS=[node1.cluster], EVENTCODE=[EV_L
    OG_EVENT]}, content=Detected pause in JVM or host machine (eg GC): pause of approximately 14944ms
    GC pool 'ParNew' had collection(s): count=1 time=14631ms, timestamp=1550926817327}
    2019-02-23 10:19:51,816 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1009ms
    No GCs detected
    2019-02-23 10:35:18,805 INFO org.apache.hadoop.util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 5783ms
    GC pool 'ParNew' had collection(s): count=1 time=5852ms

修改mapred-site.xml中的map , reduce的内存参数

*  io.sort.mb map缓冲区大小//默认：256m -->512
 *  io.sort.factor 并行处理spill的个数//默认：64 -->80  
    在浏览器打开7180集群管理界面，进入到yarn

## 1,配置–>输入框搜索：java ##

会出现对应的java堆内存参数，修改后保存  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 2]

## 2, 配置–>输入框搜索：memory ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 3]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: /images/20220317/58c9f6cbdec04b1da0feea4e005e8be9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]: /images/20220317/32a05040b017427f97b9ebbe5119ac5b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 2]: /images/20220317/5f6db940395e43faa2704fd137ef2d93.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 3]: /images/20220317/84ed824626434a9f9fbc645a8e099029.png