JVM调优之---一次GC调优实战

一时失言乱红尘 2021-05-18 20:36 623阅读 0赞

某系统反馈『性能抖动，响应时间会突然飙高，TP999 MAX会到3000+』，初步怀疑是JVM FULL GC导致的 STW，观察FULL GC日志默认的JVM参数：

-Xms4096m -Xmx4096m -XX:PermSize=512M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=1024M -XX:+UseCodeCacheFlushing

从线上down下来的GC LOG如下：

1768.617: [GC [PSYoungGen: 1313280K->31072K(1341440K)] 3990240K->2729238K(4137984K), 0.0992420 secs] [Times: user=0.36 sys=0.01, real=0.10 secs] 
    1770.525: [GC [PSYoungGen: 1316704K->27632K(1345536K)] 4014870K->2750306K(4142080K), 0.0552640 secs] [Times: user=0.20 sys=0.00, real=0.06 secs] 
    1772.437: [GC [PSYoungGen: 1319408K->51424K(1343488K)] 4042082K->2795338K(4140032K), 0.0578170 secs] [Times: user=0.23 sys=0.00, real=0.06 secs] 
    1774.583: [GC [PSYoungGen: 1343200K->30912K(1340416K)] 4087114K->2796362K(4136960K), 0.0746000 secs] [Times: user=0.27 sys=0.00, real=0.07 secs] 
    1776.521: [GC [PSYoungGen: 1316535K->27328K(1341952K)] 4081986K->2817152K(4138496K), 0.0792640 secs] [Times: user=0.26 sys=0.01, real=0.08 secs] 
    1776.601: [Full GC [PSYoungGen: 27328K->0K(1341952K)] [ParOldGen: 2789824K->150473K(2796544K)] 2817152K->150473K(4138496K) 
    [PSPermGen: 60521K->60480K(524288K)], 0.5549000 secs] [Times: user=1.64 sys=0.00, real=0.55 secs] 
    1779.045: [GC [PSYoungGen: 1285632K->51105K(1334784K)] 1436105K->201578K(4131328K), 0.0313710 secs] [Times: user=0.12 sys=0.00, real=0.03 secs] 
    1781.122: [GC [PSYoungGen: 1330593K->30368K(1338880K)] 1481066K->202953K(4135424K), 0.0522210 secs] [Times: user=0.17 sys=0.00, real=0.05 secs] 
    1782.995: [GC [PSYoungGen: 1309856K->27488K(1341440K)] 1482441K->223926K(4137984K), 0.0416840 secs] [Times: user=0.13 sys=0.01, real=0.04 secs]
    .
    .
    .
    2007.041: [GC [PSYoungGen: 1328928K->27808K(1351168K)] 4059412K->2783016K(4147712K), 0.0694050 secs] [Times: user=0.24 sys=0.00, real=0.07 secs] 
    2008.972: [GC [PSYoungGen: 1330848K->47079K(1350144K)] 4086056K->2827611K(4146688K), 0.0648990 secs] [Times: user=0.23 sys=0.00, real=0.07 secs] 
    2009.037: [Full GC [PSYoungGen: 47079K->0K(1350144K)] [ParOldGen: 2780532K->191662K(2796544K)] 2827611K->191662K(4146688K)
     [PSPermGen: 60530K->60530K(524288K)], 3.4921610 secs] [Times: user=13.39 sys=0.08, real=3.49 secs] 
    2014.630: [GC [PSYoungGen: 1303040K->31712K(1328640K)] 1494702K->223374K(4125184K), 0.0672680 secs] [Times: user=0.21 sys=0.00, real=0.07 secs] 
    2016.521: [GC [PSYoungGen: 1312223K->27136K(1339392K)] 1503886K->243740K(4135936K), 0.0297230 secs] [Times: user=0.09 sys=0.00, real=0.03 secs] 
    2018.442: [GC [PSYoungGen: 1307632K->47901K(1335296K)] 1524236K->285700K(4131840K), 0.0524640 secs] [Times: user=0.18 sys=0.00, real=0.05 secs] 
    2020.530: [GC [PSYoungGen: 1325341K->30272K(1337856K)] 1563140K->289717K(4134400K), 0.0615270 secs] [Times: user=0.22 sys=0.00, real=0.06 secs]

分析上述日志，可以看出两个问题：

1. 每次ygc，晋升至老年代的对象体积较大，平均为20m+，这导致fgc频率较高；

2. FGC的时间很长，上面看到的是550ms，实际上还有2000+的,这导致FGC开销很大；

因而调优思路很明确：

1. 减少每次ygc晋升大小；

2. 尽可能的减少每次fgc的时间开销;

因而进行了如下的尝试:

一. 使用parallel scavenge GC,但是加入如下参数 :

-Xmn1350m -XX:-UseAdaptiveSizePolicy  -XX:SurvivorRatio=6  
         调优思路:  
               ygc每次晋升到old gen的内容较多,而这很可能是因为JVM 动态的调整eden 和 survivor区,导致它们空间过小,部分本该在new gen呆着的对象直接跳到old gen

了(此现象在survivor区较为明显,因为其本来就较小)

调优效果:

![format_png][]

可以看到调优后full gc频率大为减少(由4min一次--->变为30h一次),同时因为少了 频繁调整new gen的开销,ygc耗时也略微减少了.

遗留问题:

虽然fgc频率大为降低,但是每次fgc的耗时还是一样,500ms+~2000ms

二. 改用传说中的 CMS GC,加入jvm参数如下(原来的配置不变):

-XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly   
       -XX:CMSInitiatingOccupancyFraction=70  
      调优思路:  
           使用CMS GC,启用碎片整理,降低fgc的耗时

调优效果:

![format_png 1][]

遗留问题:

91832.693: [GC [1 CMS-initial-mark: 1968357K(2811904K)] 2123963K(4021504K), 0.0732300 secs] [Times: user=0.06 sys=0.01, real=0.08 secs]
    91832.767: [CMS-concurrent-mark-start]91834.022: [CMS-concurrent-mark: 1.256/1.256 secs] [Times: user=4.06 sys=0.94, real=1.25 secs]
    91834.022: [CMS-concurrent-preclean-start]
    91834.040: [GC91834.040: [ParNew: 1091621K->50311K(1209600K), 0.0469420 secs] 3059979K->2018697K(4021504K), 0.0473540 secs] [Times: user=0.16 sys=0.01, real=0.05 secs]
    91834.123: [CMS-concurrent-preclean: 0.051/0.101 secs] [Times: user=0.31 sys=0.05, real=0.10 secs]
    91834.123: [CMS-concurrent-abortable-preclean-start]91834.900: [CMS-concurrent-abortable-preclean: 0.769/0.777 secs] [Times: user=2.36 sys=0.53, real=0.78 secs]
    91834.903: [GC[YG occupancy: 595674 K (1209600 K)]91834.904: [Rescan (parallel) , 0.6762340 secs]
    91835.580: [weak refs processing, 0.0728400 secs]91835.653: [scrub string table, 0.0009380 secs] [1 CMS-remark: 1968386K(2811904K)] 2564060K(4021504K), 0.7555510 secs] [Times: user=2.73 sys=0.03, real=0.76 secs]
    91835.659: [CMS-concurrent-sweep-start]91836.401: [GC91836.401: [ParNew: 1087111K->47187K(1209600K), 0.0723650 secs] 2884822K->1844924K(4021504K), 0.0727230 secs] [Times: user=0.26 sys=0.00, real=0.07 secs]
    91837.799: [GC91837.800: [ParNew: 1083987K->45402K(1209600K), 0.0663740 secs] 2609145K->1570570K(4021504K), 0.0667940 secs] [Times: user=0.23 sys=0.01, real=0.07 secs]
    91839.374: [GC91839.374: [ParNew: 1082202K->45240K(1209600K), 0.0673220 secs] 2353716K->1316904K(4021504K), 0.0677680 secs] [Times: user=0.23 sys=0.00, real=0.07 secs]
    91840.710: [GC91840.710: [ParNew: 1082040K->19453K(1209600K), 0.0215620 secs] 2188694K->1126165K(4021504K), 0.0219340 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
    91842.338: [GC91842.338: [ParNew: 1056253K->52429K(1209600K), 0.0507020 secs] 1780994K->777227K(4021504K), 0.0511040 secs] [Times: user=0.19 sys=0.00, real=0.05 secs]
    91843.636: [GC91843.636: [ParNew: 1089229K->21084K(1209600K), 0.0408210 secs] 1657456K->589323K(4021504K), 0.0412140 secs] [Times: user=0.13 sys=0.00, real=0.05 secs]
    91845.222: [GC91845.223: [ParNew: 1057884K->42762K(1209600K), 0.0419470 secs] 1315163K->300052K(4021504K), 0.0423230 secs] [Times: user=0.13 sys=0.00, real=0.04 secs]
    91845.925: [CMS-concurrent-sweep: 9.875/10.266 secs] [Times: user=31.77 sys=6.94, real=10.26 secs]
    91845.936: [CMS-concurrent-reset-start]91845.985: [CMS-concurrent-reset: 0.049/0.049 secs] [Times: user=0.16 sys=0.03, real=0.05 secs]

三. 降低remark的时间开销,加入参数:-XX:+CMSScavengeBeforeRemark          oldgen gc开销还是较大,虽然比ps gc略好,而且通过gc日志发现,主要耗时都是在remark的rescan阶段

调优思路:

通常情况下进行remark会先对new gen进行一次扫描,而且这个开销占比挺大,所以加上这个参数,在remark之前强制进行一次ygc

调优效果：

直接看日志吧，从下面日志看到remark的时间缩短的了很多。

91718.963: [GC [1 CMS-initial-mark: 1968409K(2811904K)] 2019966K(4021504K), 0.0419650 secs] [Times: user=0.02 sys=0.01, real=0.04 secs] 
     91719.005: [CMS-concurrent-mark-start]
     91720.142: [CMS-concurrent-mark: 1.137/1.137 secs] [Times: user=3.72 sys=0.83, real=1.14 secs] 
     91720.142: [CMS-concurrent-preclean-start]
     91720.183: [CMS-concurrent-preclean: 0.041/0.041 secs] [Times: user=0.13 sys=0.03, real=0.04 secs] 
     91720.183: [CMS-concurrent-abortable-preclean-start]
     91720.525: [GC 91720.525: [ParNew: 1074573K->72397K(1209600K), 0.0613610 secs] 3042982K->2040991K(4021504K), 0.0617860 secs] 
    [Times: user=0.23 sys=0.00, real=0.06 secs] 
    91721.314: [CMS-concurrent-abortable-preclean: 1.063/1.131 secs] [Times: user=3.59 sys=0.82, real=1.13 secs] 
     91721.316: [GC[YG occupancy: 657780 K (1209600 K)][GC2017-03-25T19:52:47.124+0800: 91721.317: [ParNew: 657780K->30915K(1209600K), 0.1393790 secs] 
    2626374K->1999535K(4021504K), 0.1398100 secs] [Times: user=0.52 sys=0.00, real=0.14 secs] 
     91721.456: [Rescan (parallel) , 0.0302040 secs]91721.487: [weak refs processing, 0.0691870 secs]
     91721.556: [scrub string table, 0.0009370 secs] [1 CMS-remark: 1968619K(2811904K)] 1999535K(4021504K), 0.2452990 secs] [Times: user=0.72 sys=0.00, real=0.25 secs] 
     91721.562: [CMS-concurrent-sweep-start][CMS-concurrent-sweep: 8.561/8.898 secs] [Times: user=27.82 sys=5.98, real=8.89 secs] 
    [CMS-concurrent-reset-start]

至此，jvm调优告一段落，目前线上大约30H生一次 CMS GC（“FGC”），每次耗时在300ms以内，算是比较完美了。

JVM参数如下：

-Xms4096m -Xmx4096m -XX:PermSize=256M -XX:MaxPermSize=256M -XX:ReservedCodeCacheSize=1024M -XX:+UseCodeCacheFlushing

\-Xmn1350m -XX:-UseAdaptiveSizePolicy  -XX:SurvivorRatio=6 -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+UseCMSInitiatingOccupancyOnly     -XX:CMSInitiatingOccupancyFraction=70  -XX:+CMSScavengeBeforeRemark

[format_png]: /images/20210518/4ae0bc7e64804d1a8253c31262a548b5.png
[format_png 1]: /images/20210518/b7bef3e54d9d45a589bc5c7ef398f20f.png