hbase读写流程, regionserver内存调节

男娘i 2022-12-20 08:50 407阅读 0赞

### 文章目录 ###

*  1，遇到的问题
 *  2，hbase读写流程
 *  3，regionserver 内存调节
 *  4，hbase垃圾回收策略

# 1，遇到的问题 #

*  hbase数据写入频繁，触发memstore刷写次数多，导致region , RS 下线，写入失败
 *  hbase 发生compact ，导致RS下线，查询失败

# 2，hbase读写流程 #

*  hbase数据的存储结构：[https://blog.csdn.net/eyeofeagle/article/details/105506606][https_blog.csdn.net_eyeofeagle_article_details_105506606]
 *  hbase读写流程图：[https://sematext.com/blog/hbase-memstore-what-you-should][https_sematext.com_blog_hbase-memstore-what-you-should]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70_pic_center]
 *  [https://blog.cloudera.com/guide-to-using-apache-hbase-ports/][https_blog.cloudera.com_guide-to-using-apache-hbase-ports]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70_pic_center 1]
 *  表数据读写：  
    1 (通过zk, 获取meta区域的服务器地址）  
    2 (找到表所在的regionserver，开始读写）
 *  管理 Hbase表( memstore刷写，hfile合并 ）:  
    1 (通过zk, 获取meta区域的服务器地址）  
    3 (通过master节点，创建/修改表)  
    4 (master节点通知regionserver打开/关闭/移动/分割/刷新/压缩区域)  
    5 (master节点把数据：\[meta区域的服务器地址,创建日志拆分任务等\]，存入zk)  
    6,7,8 (regionserver 读取zk数据，执行拆分日志等任务，并报告给master节点)

通过zookeeper-client 命令行登录，查看zk中的数据：

[zk: localhost:2181(CONNECTED) 2] ls /hbase
    [ meta-region-server, rs,  master, namespace, hbaseid, table .... ]
    
    [zk: localhost:2181(CONNECTED) 3] get /hbase/master
    �master:60000-��3�gNNPBUF
    
    hadoop1�����.��
    ...
    
    [zk: localhost:2181(CONNECTED) 4] ls /hbase/rs
    [hadoop1,60020,1605056703882]
    
    
    [zk: localhost:2181(CONNECTED) 5] get /hbase/meta-region-server
    �regionserver:60020]�i~p�<PBUF
    
    hadoop1�����.
    ...
    
    
    [zk: localhost:2181(CONNECTED) 6] ls /hbase/namespace
    [default, t1, hbase]
    
    
    [zk: localhost:2181(CONNECTED) 7] get /hbase/namespace/default
    �master:60000J3�g�e
    
    default
    ...

# 3，regionserver 内存调节 #

思路：CDH 管理的Hbase Region Server 内存调节

//zookeeper设置： 
    	maxSessionTimeout=300000 //默认40*1000ms 
    	maxClientCnxns=10000 //默认60 (0表示无限制)
    
    //hbase配置 ：
        zookeeper.session.timeout=300000 //默认(2-40)*1000ms
    	hbase.client.scanner.timeout.period=240000 //默认60*1000ms 
    	hbase.rpc.timeout=280000 //默认60*1000ms
    	hbase.regionserver.handler.count=100 //默认25 
    	hbase.hstore.blockingStoreFiles=10默认 --> 1000
    	hbase.regionserver.maxlogs=32默认 --> 16

*  保存数据的是Region Server，要加大其java 堆内存:(4G)； Master节点可小:(1G)
 *  Region Server内存主要消耗在缓存上：(以下两者之和不能大于0.8, 否则会报错)  
    hfile.block.cache.size（磁盘HFile缓存），默认0.4  
    hbase.regionserver.global.memstore.size （hbase写入数据到缓存), 默认0.4

MemStore刷写触发的条件：

*  hbase.hregion.memstore.flush.size默认128M :每10s检查, 当memstore大小超出时flush
 *  hbase.regionserver.global.memstore.size：当该Region Server管理的所有memstore总大小超出时flush
 *  WAL大小超出hbase.regionserver.maxlogs \* hbase.regionserver.hlog.blocksize（默认为32\*128M=4GB)，会导致Memstore刷新，应配置上面的两数之积略大于hbase.regionserver.global.memstore.lowerLimit \* HBASE\_HEAPSIZE。（默认4G \* 0.4=1.6G )  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70_pic_center 2]

# 4，hbase垃圾回收策略 #

hbase 垃圾回收策略设置：[https://blog.cloudera.com/tuning-java-garbage-collection-for-hbase/][https_blog.cloudera.com_tuning-java-garbage-collection-for-hbase]  
测试环境：cpu: 40 个逻辑cpu， 内存：256GB  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]

Error: VM option 'G1NewSizePercent' is experimental and must be enabled via -XX:+UnlockExperimentalVMOptions.
    Error: Could not create the Java Virtual Machine.
    Error: A fatal exception has occurred. Program will exit.
    
    
    #-ResizePLAB :Promotion Local Allocation Buffers，它被用于年轻代回收。PLAB的作用是避免多线程竞争相同的数据，处理方式是每个线程拥有独立的PLAB，用于针对幸存者和老年空间。当应用开启的线程较多时，最好使用-XX:-ResizePlaB来关闭PLAB()的大小调整，以避免大量的线程通信所导致的性能下降
    #+ParallelRefProcEnabled : 应用有很多的Reference or finalizable objects
    -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=23 
    
    #使用java util分析堆内存使用
    [root@localhost jms]# ps -ef |grep -i hregion
    hbase     62612  22652  3 Aug05 ?        00:29:41 /usr/java/jdk1.8.0_211-amd64/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Djava.net.preferIPv4Stack=true -Xms2147483648 -Xmx2147483648 -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+ParallelRefProcEnabled -XX:-ResizePLAB -XX:ParallelGCThreads=23 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/cdh/tmp/hbase_hbase-REGIONSERVER-144a0ada48aafa3485e33bb69df57f5c_pid62612.hprof -XX:OnOutOfMemoryError=/opt/cm-5.12.2/lib64/cmf/service/common/killparent.sh -Dhbase.log.dir=/home/cdh/var/log/hbase -Dhbase.log.file=hbase-cmf-hbase-REGIONSERVER-eadage.log.out -Dhbase.home.dir=/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hbase -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hbase/lib/native/Linux-amd64-64 -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.regionserver.HRegionServer start
    
    [root@localhost jms]# jstat -gcutil 62612 1s
      S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
      0.00 100.00  25.09  74.62  98.16  96.66    602   24.008     0    0.000   24.008
      0.00 100.00  25.20  74.62  98.16  96.66    602   24.008     0    0.000   24.008
      0.00 100.00  25.32  74.62  98.16  96.66    602   24.008     0    0.000   24.008
    #-gcutil option
    # Summary of garbage collection statistics.
    # S0: Survivor space 0 utilization as a percentage of the space's current capacity.
    # S1: Survivor space 1 utilization as a percentage of the space's current capacity.
    # E: Eden space utilization as a percentage of the space's current capacity.
    # O: Old space utilization as a percentage of the space's current capacity.
    # M: Metaspace utilization as a percentage of the space's current capacity.
    # CCS: Compressed class space utilization as a percentage.
    # YGC: Number of young generation GC events.
    # YGCT: Young generation garbage collection time.
    # FGC: Number of full GC events.
    # FGCT: Full garbage collection time.
    # GCT: Total garbage collection time.
    
    [root@localhost jms]# jmap -heap 62612
    Attaching to process ID 62612, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.211-b12
    
    using thread-local object allocation.
    Garbage-First (G1) GC with 23 thread(s)
    
    Heap Configuration:
       MinHeapFreeRatio         = 40
       MaxHeapFreeRatio         = 70
       MaxHeapSize              = 2147483648 (2048.0MB)
       NewSize                  = 1363144 (1.2999954223632812MB)
       MaxNewSize               = 1287651328 (1228.0MB)
       OldSize                  = 5452592 (5.1999969482421875MB)
       NewRatio                 = 2
       SurvivorRatio            = 8
       MetaspaceSize            = 21807104 (20.796875MB)
       CompressedClassSpaceSize = 1073741824 (1024.0MB)
       MaxMetaspaceSize         = 17592186044415 MB
       G1HeapRegionSize         = 1048576 (1.0MB)
    
    Heap Usage:
    G1 Heap:
       regions  = 2048
       capacity = 2147483648 (2048.0MB)
       used     = 1730277376 (1650.12109375MB)
       free     = 417206272 (397.87890625MB)
       80.57231903076172% used
    G1 Young Generation:
    Eden Space:
       regions  = 745
       capacity = 907018240 (865.0MB)
       used     = 781189120 (745.0MB)
       free     = 125829120 (120.0MB)
       86.1271676300578% used
    Survivor Space:
       regions  = 88
       capacity = 92274688 (88.0MB)
       used     = 92274688 (88.0MB)
       free     = 0 (0.0MB)
       100.0% used
    G1 Old Generation:
       regions  = 818
       capacity = 1148190720 (1095.0MB)
       used     = 856813568 (817.12109375MB)
       free     = 291377152 (277.87890625MB)
       74.62293093607306% used

[https_blog.csdn.net_eyeofeagle_article_details_105506606]: https://blog.csdn.net/eyeofeagle/article/details/105506606
[https_sematext.com_blog_hbase-memstore-what-you-should]: https://sematext.com/blog/hbase-memstore-what-you-should
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70_pic_center]: /images/20221120/89bb24d7547840539f333b7f1a9defa3.png
[https_blog.cloudera.com_guide-to-using-apache-hbase-ports]: https://blog.cloudera.com/guide-to-using-apache-hbase-ports/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70_pic_center 1]: https://img-blog.csdnimg.cn/20201111170139461.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU=,size_16,color_FFFFFF,t_70#pic_center
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70_pic_center 2]: https://img-blog.csdnimg.cn/2020111116544547.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU=,size_16,color_FFFFFF,t_70#pic_center
[https_blog.cloudera.com_tuning-java-garbage-collection-for-hbase]: https://blog.cloudera.com/tuning-java-garbage-collection-for-hbase/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/24cc095920c54c88acf7dee158d401dc.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU=,size_16,color_FFFFFF,t_70