Hbase-Trying to load more than 32 hfiles to one family of one region

系统管理员 2024-04-17 05:49 82阅读 0赞

*  [BulkLoad方式优化][BulkLoad]
    
     *  [背景][Link 1]
     *  [问题分析][Link 2]
     *  [解决方案][Link 3]
        
         *  [方式一：spark DataFrame写入Hive临时表前进行coalesce(30)][spark DataFrame_Hive_coalesce_30]
         *  [方式二：][Link 4]
     *  [参考链接][Link 5]

# BulkLoad方式优化 #

## 背景 ##

> 项目中使用到Hbase BulkLoad方式插入数据，当数据量较大时或小文件过多时，任务失败： 报错：Exception in thread "main" java.io.IOException: Trying to load more than 32 hfiles to one family of one region

## 问题分析 ##

> 代码逻辑是将临时存放在hive临时表的数据 分别写入到Hive正式表和Hbase中， Hbase是使用BulkLoad方式插入的, 查看了Hive临时表中的数据大小，parquet格式文件，每个15MB左右，在网上查相关异常，也给出了相关的解决方案

## 解决方案 ##

### 方式一：spark DataFrame写入Hive临时表前进行coalesce(30) ###

> 抱着试下的心态，在写入hive之前减少了分区，结果运行成功

### 方式二： ###

hbase-site.xml中设置

<property> 
        <name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name> 
        <value>3200</value>
        </property>

重新启动集群

## 参考链接 ##

[https://blog.csdn.net/weixin\_40861707/article/details/99976526][https_blog.csdn.net_weixin_40861707_article_details_99976526]

[BulkLoad]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#bulkload%E6%96%B9%E5%BC%8F%E4%BC%98%E5%8C%96
[Link 1]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#%E8%83%8C%E6%99%AF
[Link 2]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#%E9%97%AE%E9%A2%98%E5%88%86%E6%9E%90
[Link 3]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88
[spark DataFrame_Hive_coalesce_30]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#%E6%96%B9%E5%BC%8F%E4%B8%80spark-dataframe%E5%86%99%E5%85%A5hive%E4%B8%B4%E6%97%B6%E8%A1%A8%E5%89%8D%E8%BF%9B%E8%A1%8Ccoalesce30
[Link 4]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#%E6%96%B9%E5%BC%8F%E4%BA%8C
[Link 5]: https://note.youdao.com/md/preview.html?file=%2Fyws%2Fapi%2Fpersonal%2Ffile%2FD00585D331EE4366839B09698356A00C%3Fmethod%3Ddownload%26read%3Dtrue%26shareKey%3D5f2d744463f040e5a2dbfc943dd0e730#%E5%8F%82%E8%80%83%E9%93%BE%E6%8E%A5
[https_blog.csdn.net_weixin_40861707_article_details_99976526]: https://blog.csdn.net/weixin_40861707/article/details/99976526