Arthas实战全录

雨点打透心脏的1/2处 2022-10-30 12:28 379阅读 0赞

## 一、Arthas基础认识 ##

1. 可以做什么？

Arthas 是Alibaba开源的Java诊断工具，采用命令行交互模式，提供了丰富的功能，是排查jvm相关问题的利器。  
在逛github时，发现了这款利器，深入了解之后，简直惊为天人。下面先列举一下它能做的一些事情：

*  提供性能看板，包括线程、cpu、内存等信息，并且会定时的刷新。
 *  根据各种条件查看线程快照。比如找出cpu占用率最高的n个线程等
 *  输出jvm的各种信息，如gc算法、jdk版本、ClassPath等
 *  查看/设置sysprop和sysenv
 *  查看某个类的静态属性，也可以通过ognl语法执行一些语句
 *  查看已加载的类的详细信息，比如这个类从哪个jar包加载的。也可以查看类的方法的信息
 *  dump某个类的字节码到指定目录
 *  直接反编译指定的类
 *  查看类加载器的一些信息
 *  可以让jvm重新加载某个类
 *  监控方法的执行，同时可以获取到执行的入参、出参以及抛出的异常
 *  追踪方法执行的调用栈，以及各个方法的调用时间

2. 运行原理？

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZFTkdRSVlVTlJBTg_size_16_color_FFFFFF_t_70][]

3.支持的命令？

NAME         DESCRIPTION                                                                                                                                      
     help         Display Arthas Help                                                                                                                              
     keymap       Display all the available keymap for the specified connection.                                                                                   
     sc           Search all the classes loaded by JVM                                                                                                             
     sm           Search the method of classes loaded by JVM                                                                                                       
     classloader  Show classloader info                                                                                                                            
     jad          Decompile class                                                                                                                                  
     getstatic    Show the static field of a class                                                                                                                 
     monitor      Monitor method execution statistics, e.g. total/success/failure count, average rt, fail rate, etc.                                               
     stack        Display the stack trace for the specified class and method                                                                                       
     thread       Display thread info, thread stack                                                                                                                
     trace        Trace the execution time of specified method invocation.                                                                                         
     watch        Display the input/output parameter, return object, and thrown exception of specified method invocation                                           
     tt           Time Tunnel                                                                                                                                      
     jvm          Display the target JVM information                                                                                                               
     perfcounter  Display the perf counter information.                                                                                                            
     ognl         Execute ognl expression.                                                                                                                         
     mc           Memory compiler, compiles java files into bytecode and class files in memory.                                                                    
     redefine     Redefine classes. @see Instrumentation#redefineClasses(ClassDefinition...)                                                                       
     retransform  Retransform classes. @see Instrumentation#retransformClasses(Class...)                                                                           
     dashboard    Overview of target jvm's thread, memory, gc, vm, tomcat info.                                                                                    
     dump         Dump class byte array from JVM                                                                                                                   
     heapdump     Heap dump                                                                                                                                        
     options      View and change various Arthas options                                                                                                           
     cls          Clear the screen                                                                                                                                 
     reset        Reset all the enhanced classes                                                                                                                   
     version      Display Arthas version                                                                                                                           
     session      Display current session information                                                                                                              
     sysprop      Display, and change the system properties.                                                                                                       
     sysenv       Display the system env.                                                                                                                          
     vmoption     Display, and update the vm diagnostic options.                                                                                                   
     logger       Print logger info, and update the logger level                                                                                                   
     history      Display command history                                                                                                                          
     cat          Concatenate and print files                                                                                                                      
     base64       Encode and decode using Base64 representation                                                                                                    
     echo         write arguments to the standard output                                                                                                           
     pwd          Return working directory name                                                                                                                    
     mbean        Display the mbean information                                                                                                                    
     grep         grep command for pipes.                                                                                                                          
     tee          tee command for pipes.                                                                                                                           
     profiler     Async Profiler. https://github.com/jvm-profiling-tools/async-profiler                                                                            
     stop         Stop/Shutdown Arthas server and exit the console.

## 二、下载和安装 ##

参考地址： [https://arthas.aliyun.com/en-us/][https_arthas.aliyun.com_en-us]

1. Use `arthas-boot`(Recommended)

Download `arthas-boot.jar`，Start with `java` command:

<table> 
 <tbody> 
  <tr> 
   <td>&nbsp;</td> 
   <td> <p>curl -O https://arthas.aliyun.com/arthas-boot.jar</p> </td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td> <p>java -jar arthas-boot.jar</p> </td> 
  </tr> 
 </tbody> 
</table>

Print usage:

java -jar arthas-boot.jar -h

2. Use `as.sh`

You can install Arthas with one single line command on Linux, Unix, and Mac. Pls. copy the following command and paste it into the command line, then press *Enter* to run:

curl -L https://arthas.aliyun.com/install.sh | sh

The command above will download the bootstrap script `as.sh` to the current directory. You can move it to any other place you want, or put its location in `$PATH`.

You can enter its interactive interface by executing `as.sh`, or execute `as.sh -h` for more help information.

第二种方法更加简单，但是第一种更为通用，尤其是在目前的微服务的状况下，服务运行在docker里，可能并没有安装telnet，就不能直接使用第二种。

## 三、初级使用 ##

1. 安装并查利用help查看可供使用的命令

sh-4.2$ curl -O https://arthas.aliyun.com/arthas-boot.jar
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  135k  100  135k    0     0  30152      0  0:00:04  0:00:04 --:--:-- 30156
    sh-4.2$ ls
    arthas-boot.jar  arthas-output  live-server.jar  run.sh
    sh-4.2$ java -jar arthas-boot.jar 
    [INFO] arthas-boot version: 3.4.6
    [INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
    * [1]: 14 /data/code/live-server.jar
    1
    [INFO] arthas home: /home/shareit/.arthas/lib/3.4.6/arthas
    [INFO] Try to attach process 14
    [INFO] Attach process 14 success.
    [INFO] arthas-client connect 127.0.0.1 3658
      ,---.  ,------. ,--------.,--.  ,--.  ,---.   ,---.                           
     /  O  \ |  .--. ''--.  .--'|  '--'  | /  O  \ '   .-'                          
    |  .-.  ||  '--'.'   |  |   |  .--.  ||  .-.  |`.  `-.                          
    |  | |  ||  |\  \    |  |   |  |  |  ||  | |  |.-'    |                         
    `--' `--'`--' '--'   `--'   `--'  `--'`--' `--'`-----'                          
                                                                                    
    
    wiki      https://arthas.aliyun.com/doc                                         
    tutorials https://arthas.aliyun.com/doc/arthas-tutorials.html                   
    version   3.4.6                                                                 
    pid       14                                                                    
    time      2021-02-19 07:05:28                                                   
    
    [arthas@14]$ help
     NAME         DESCRIPTION                                                                                                                                      
     help         Display Arthas Help

2. Arthas具体命令讲解，其命令主要分为3类：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZFTkdRSVlVTlJBTg_size_16_color_FFFFFF_t_70 1][]

1）dashboard命令

[arthas@14]$ dashboard
    ID     NAME                                   GROUP               PRIORITY      STATE        %CPU         DELTA_TIME   TIME          INTERRUPTED  DAEMON       
    -1     C2 CompilerThread0                     -                   -1            -            0.0          0.000        1:1.643       false        true         
    78     DestroyJavaVM                          main                5             RUNNABLE     0.0          0.000        0:19.976      false        false        
    -1     C1 CompilerThread1                     -                   -1            -            0.0          0.000        0:16.397      false        true         
    302    RxComputationScheduler-1               main                5             TIMED_WAITIN 0.0          0.000        0:9.905       false        true         
    -1     VM Periodic Task Thread                -                   -1            -            0.0          0.000        0:8.563       false        true         
    -1     VM Thread                              -                   -1            -            0.0          0.000        0:5.847       false        true         
    24     Abandoned connection cleanup thread    main                5             TIMED_WAITIN 0.0          0.000        0:5.832       false        true         
    76     scheduling-1                           main                5             TIMED_WAITIN 0.0          0.000        0:4.140       false        false        
    43     pollingConfigurationSource             main                5             TIMED_WAITIN 0.0          0.000        0:1.986       false        true         
    34     I/O dispatcher 1                       main                5             RUNNABLE     0.0          0.000        0:1.301       false        false        
    57     http-nio-8080-exec-10                  main                5             WAITING      0.0          0.000        0:1.257       false        true         
    58     http-nio-8080-ClientPoller             main                5             RUNNABLE     0.0          0.000        0:1.109       false        true         
    60     Catalina-utility-1                     main                1             WAITING      0.0          0.000        0:1.100       false        false        
    21     Catalina-utility-1                     main                1             WAITING      0.0          0.000        0:1.082       false        false        
    55     http-nio-8080-exec-8                   main                5             WAITING      0.0          0.000        0:1.043       false        true         
    22     Catalina-utility-2                     main                1             TIMED_WAITIN 0.0          0.000        0:1.015       false        false        
    48     http-nio-8080-exec-1                   main                5             WAITING      0.0          0.000        0:0.984       false        true         
    Memory                            used        total      max         usage      GC                                                                             
    heap                              350M        966M       966M        36.27%     gc.copy.count                           79                                     
    eden_space                        257M        266M       266M        96.40%     gc.copy.time(ms)                        2795                                   
    survivor_space                    9M          33M        33M         28.09%     gc.marksweepcompact.count               4                                      
    tenured_gen                       84M         666M       666M        12.63%     gc.marksweepcompact.time(ms)            1035                                   
    nonheap                           183M        199M       -1          91.68%                                                                                    
    code_cache                        42M         51M        240M        17.87%                                                                                    
    metaspace                         124M        131M       -1          94.93%                                                                                    
    compressed_class_space            15M         16M        1024M       1.53%                                                                                     
    direct                            314K        314K       -           100.00%                                                                                   
    mapped                            0K          0K         -           0.00%                                                                                     
    Runtime                                                                                                                                                        
    os.name                                                                         Linux                                                                          
    os.version                                                                      3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64                                      
    java.version                                                                    1.8.0_202                                                                      
    java.home                                                                       /App/java/jre                                                                  
    systemload.average                                                              2.02                                                                           
    processors                                                                      1                                                                              
    timestamp/uptime                                                                Fri Feb 19 08:54:52 UTC 2021/18966s

*  ID: JVM thread ID, pls. note this ID is different from the nativeID in jstack
 *  NAME: thread name
 *  GROUP: thread group name
 *  PRIORITY: thread priority, ranged from 1 to 10. The greater number, the higher priority
 *  STATE: thread state
 *  CPU%: the ratio of CPU usage for the thread. For example, the sampling interval is 1000ms, and the incremental cpu time of a thread is 100ms, then the cpu usage rate=100/1000=10%
 *  DELTA\_TIME: incremental CPU time of thread running after the last sampling in `second` format
 *  TIME: total CPU time of the thread in `minute:second` format
 *  INTERRUPTED: the thread interruption state
 *  DAEMON: daemon thread or not

2）thread命令

[arthas@14]$ thread -n 3
    "arthas-command-execute" Id=610 cpuUsage=0.68% deltaTime=1ms time=125ms RUNNABLE
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:448)
        at com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:199)
        at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:122)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
        at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    
    
    "C1 CompilerThread1" [Internal] cpuUsage=0.36% deltaTime=0ms time=16653ms
    
    
    "RxComputationScheduler-1" Id=302 cpuUsage=0.12% deltaTime=0ms time=11307ms TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22fb6f64
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22fb6f64
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    
    # By default, they are arranged in descending order of CPU increment time, and only the first page of data is displayed.
    [arthas@14]$ thread
    Threads Total: 269, NEW: 0, RUNNABLE: 19, BLOCKED: 0, WAITING: 222, TIMED_WAITING: 23, TERMINATED: 0, Internal threads: 5                                      
    ID     NAME                                   GROUP               PRIORITY      STATE        %CPU         DELTA_TIME   TIME          INTERRUPTED  DAEMON       
    610    arthas-command-execute                 system              5             RUNNABLE     0.6          0.001        0:0.130       false        true         
    36     cluster-ClusterId{value='602f32e9a621f main                5             TIMED_WAITIN 0.16         0.000        0:1.002       false        true         
    38     cluster-ClusterId{value='602f32e9a621f main                5             TIMED_WAITIN 0.15         0.000        0:0.926       false        true         
    302    RxComputationScheduler-1               main                5             TIMED_WAITIN 0.11         0.000        0:11.469      false        true         
    -1     VM Periodic Task Thread                -                   -1            -            0.04         0.000        0:9.062       false        true         
    24     Abandoned connection cleanup thread    main                5             TIMED_WAITIN 0.02         0.000        0:6.154       false        true         
    58     http-nio-8080-ClientPoller             main                5             RUNNABLE     0.01         0.000        0:1.166       false        true         
    47     http-nio-8080-BlockPoller              main                5             RUNNABLE     0.01         0.000        0:0.721       false        true         
    2      Reference Handler                      system              10            WAITING      0.0          0.000        0:0.156       false        true         
    3      Finalizer                              system              8             WAITING      0.0          0.000        0:0.214       false        true         
    4      Signal Dispatcher                      system              9             RUNNABLE     0.0          0.000        0:0.000       false        true         
    594    Attach Listener                        system              9             RUNNABLE     0.0          0.000        0:0.010       false        true         
    596    arthas-timer                           system              9             WAITING      0.0          0.000        0:0.000       false        true         
    599    arthas-NettyHttpTelnetBootstrap-3-1    system              5             RUNNABLE     0.0          0.000        0:0.037       false        true         
    600    arthas-NettyWebsocketTtyBootstrap-4-1  system              5             RUNNABLE     0.0          0.000        0:0.001       false        true         
    601    arthas-NettyWebsocketTtyBootstrap-4-2  system              5             RUNNABLE     0.0          0.000        0:0.001       false        true         
    602    arthas-shell-server                    system              9             TIMED_WAITIN 0.0          0.000        0:0.014       false        true         
    603    arthas-session-manager                 system              9             TIMED_WAITIN 0.0          0.000        0:0.007       false        true         
    604    arthas-UserStat                        system              9             WAITING      0.0          0.000        0:0.001       false        true         
    606    arthas-NettyHttpTelnetBootstrap-3-2    system              5             RUNNABLE     0.0          0.000        0:0.246       false        true         
    13     AsyncAppender-Worker-ASYNC_FILEINFO    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    14     AsyncAppender-Worker-ASYNC_FILEWARN    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    16     logback-2                              main                5             TIMED_WAITIN 0.0          0.000        0:0.046       false        true         
    17     AsyncAppender-Worker-ASYNC_STAT_USER   main                5             WAITING      0.0          0.000        0:0.001       false        true         
    18     AsyncAppender-Worker-ASYNC_STAT_USER_C main                5             WAITING      0.0          0.000        0:0.000       false        true         
    19     AsyncAppender-Worker-ASYNC_FILEINFO    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    20     AsyncAppender-Worker-ASYNC_FILEWARN    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    21     Catalina-utility-1                     main                1             WAITING      0.0          0.000        0:1.133       false        false        
    22     Catalina-utility-2                     main                1             TIMED_WAITIN 0.0          0.000        0:1.067       false        false        
    23     container-0                            main                5             TIMED_WAITIN 0.0          0.000        0:0.065       false        false        
    25     HikariPool-1 housekeeper               main                5             TIMED_WAITIN 0.0          0.000        0:0.104       false        true         
    28     lettuce-eventExecutorLoop-1-1          main                5             TIMED_WAITIN 0.0          0.000        0:0.227       false        true         
    29     lettuce-epollEventLoop-4-1             main                5             RUNNABLE     0.0          0.000        0:0.516       false        true         
    30     lettuce-epollEventLoop-4-2             main                5             RUNNABLE     0.0          0.000        0:0.189       false        true         
    31     lettuce-epollEventLoop-4-3             main                5             RUNNABLE     0.0          0.000        0:0.234       false        true
    
    # thread id, show the running stack for the target thread
    [arthas@14]$ thread 29
    "lettuce-epollEventLoop-4-1" Id=29 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:148)
        at io.netty.channel.epoll.Native.epollWait(Native.java:141)
        at io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
    
    # To diagnose such an issue, Arthas provides thread -b to find the problematic thread in one single command.
    [arthas@14]$ thread -b
    No most blocking thread found!
    
    # thread –state , view the special state theads
    [arthas@14]$ thread --state WAITING
    Threads Total: 264, NEW: 0, RUNNABLE: 19, BLOCKED: 0, WAITING: 222, TIMED_WAITING: 23, TERMINATED: 0                                                           
    ID     NAME                                   GROUP               PRIORITY      STATE        %CPU         DELTA_TIME   TIME          INTERRUPTED  DAEMON       
    2      Reference Handler                      system              10            WAITING      0.0          0.000        0:0.156       false        true         
    3      Finalizer                              system              8             WAITING      0.0          0.000        0:0.214       false        true         
    596    arthas-timer                           system              9             WAITING      0.0          0.000        0:0.000       false        true         
    604    arthas-UserStat                        system              9             WAITING      0.0          0.000        0:0.001       false        true         
    13     AsyncAppender-Worker-ASYNC_FILEINFO    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    14     AsyncAppender-Worker-ASYNC_FILEWARN    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    17     AsyncAppender-Worker-ASYNC_STAT_USER   main                5             WAITING      0.0          0.000        0:0.001       false        true         
    18     AsyncAppender-Worker-ASYNC_STAT_USER_C main                5             WAITING      0.0          0.000        0:0.000       false        true         
    19     AsyncAppender-Worker-ASYNC_FILEINFO    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    20     AsyncAppender-Worker-ASYNC_FILEWARN    main                5             WAITING      0.0          0.000        0:0.000       false        true         
    22     Catalina-utility-2                     main                1             WAITING      0.0          0.000        0:1.089       false        false        
    46     spring.cloud.inetutils                 main                5             WAITING      0.0          0.000        0:0.000       false        true         
    48     http-nio-8080-exec-1                   main                5             WAITING      0.0          0.000        0:1.018       false        true         
    49     http-nio-8080-exec-2                   main                5             WAITING      0.0          0.000        0:0.873       false        true         
    50     http-nio-8080-exec-3                   main                5             WAITING      0.0          0.000        0:0.667       false        true         
    51     http-nio-8080-exec-4                   main                5             WAITING      0.0          0.000        0:0.817       false        true
    
    
    # thread -i 1000: Count the thread cpu time of the last 1000ms.
    # thread -n 3 -i 1000: List the 3 busiest thread stacks in 1000ms.
    [arthas@14]$ thread -n 3 -i 1000
    "arthas-command-execute" Id=610 cpuUsage=0.2% deltaTime=2ms time=147ms RUNNABLE
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:448)
        at com.taobao.arthas.core.command.monitor200.ThreadCommand.processTopBusyThreads(ThreadCommand.java:199)
        at com.taobao.arthas.core.command.monitor200.ThreadCommand.process(ThreadCommand.java:122)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
        at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    
    
    "RxComputationScheduler-1" Id=302 cpuUsage=0.13% deltaTime=1ms time=12224ms TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22fb6f64
        at sun.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22fb6f64
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    
    
    "VM Thread" [Internal] cpuUsage=0.07% deltaTime=0ms time=6241ms

3）jvm命令

[arthas@14]$ jvm
     RUNTIME                                                                                                                                                       
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     MACHINE-NAME                                 14@game-server-76c7dcc4bc-9ntlf                                                                                  
     JVM-START-TIME                               2021-02-19 03:38:46                                                                                              
     MANAGEMENT-SPEC-VERSION                      1.2                                                                                                              
     SPEC-NAME                                    Java Virtual Machine Specification                                                                               
     SPEC-VENDOR                                  Oracle Corporation                                                                                               
     SPEC-VERSION                                 1.8                                                                                                              
     VM-NAME                                      Java HotSpot(TM) 64-Bit Server VM                                                                                
     VM-VENDOR                                    Oracle Corporation                                                                                               
     VM-VERSION                                   25.202-b08                                                                                                       
     INPUT-ARGUMENTS                              -Xms1000m                                                                                                        
                                                  -Xmx1000m                                                                                                        
                                                                                                                                                                   
     CLASS-PATH                                   /data/code/game-server.jar                                                                                       
     BOOT-CLASS-PATH                              /App/java/jre/lib/resources.jar:/App/java/jre/lib/rt.jar:/App/java/jre/lib/sunrsasign.jar:/App/java/jre/lib/jsse 
                                                  .jar:/App/java/jre/lib/jce.jar:/App/java/jre/lib/charsets.jar:/App/java/jre/lib/jfr.jar:/App/java/jre/classes    
     LIBRARY-PATH                                 /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib                                                     
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     CLASS-LOADING                                                                                                                                                 
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     LOADED-CLASS-COUNT                           23446                                                                                                            
     TOTAL-LOADED-CLASS-COUNT                     23450                                                                                                            
     UNLOADED-CLASS-COUNT                         4                                                                                                                
     IS-VERBOSE                                   false                                                                                                            
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     COMPILATION                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     NAME                                         HotSpot 64-Bit Tiered Compilers                                                                                  
     TOTAL-COMPILE-TIME                           103567                                                                                                           
     [time (ms)]                                                                                                                                                   
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     GARBAGE-COLLECTORS                                                                                                                                            
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     Copy                                         name : Copy                                                                                                      
     [count/time (ms)]                            collectionCount : 82                                                                                             
                                                  collectionTime : 2862                                                                                            
                                                                                                                                                                   
     MarkSweepCompact                             name : MarkSweepCompact                                                                                          
     [count/time (ms)]                            collectionCount : 4                                                                                              
                                                  collectionTime : 1035                                                                                            
                                                                                                                                                                   
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     MEMORY-MANAGERS                                                                                                                                               
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     CodeCacheManager                             Code Cache                                                                                                       
                                                                                                                                                                   
     Metaspace Manager                            Metaspace                                                                                                        
                                                  Compressed Class Space                                                                                           
                                                                                                                                                                   
     Copy                                         Eden Space                                                                                                       
                                                  Survivor Space                                                                                                   
                                                                                                                                                                   
     MarkSweepCompact                             Eden Space                                                                                                       
                                                  Survivor Space                                                                                                   
                                                  Tenured Gen                                                                                                      
                                                                                                                                                                   
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     MEMORY                                                                                                                                                        
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     HEAP-MEMORY-USAGE                            init : 1048576000(1000.0 MiB)                                                                                    
     [memory in bytes]                            used : 231360240(220.6 MiB)                                                                                      
                                                  committed : 1013645312(966.7 MiB)                                                                                
                                                  max : 1013645312(966.7 MiB)                                                                                      
                                                                                                                                                                   
     NO-HEAP-MEMORY-USAGE                         init : 2555904(2.4 MiB)                                                                                          
     [memory in bytes]                            used : 193959168(185.0 MiB)                                                                                      
                                                  committed : 209846272(200.1 MiB)                                                                                 
                                                  max : -1(-1 B)                                                                                                   
                                                                                                                                                                   
     PENDING-FINALIZE-COUNT                       0                                                                                                                
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     OPERATING-SYSTEM                                                                                                                                              
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     OS                                           Linux                                                                                                            
     ARCH                                         amd64                                                                                                            
     PROCESSORS-COUNT                             1                                                                                                                
     LOAD-AVERAGE                                 1.54                                                                                                             
     VERSION                                      3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64                                                                        
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     THREAD                                                                                                                                                        
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     COUNT                                        263                                                                                                              
     DAEMON-COUNT                                 253                                                                                                              
     PEAK-COUNT                                   266                                                                                                              
     STARTED-COUNT                                944                                                                                                              
     DEADLOCK-COUNT                               0                                                                                                                
                                                                                                                                                                   
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     FILE-DESCRIPTOR                                                                                                                                               
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     MAX-FILE-DESCRIPTOR-COUNT                    1048576                                                                                                          
     OPEN-FILE-DESCRIPTOR-COUNT                   116

## Thread related ##

*  COUNT: the count of active threads
 *  DAEMON-COUNT: the count of active daemon threads
 *  PEAK-COUNT: the maximum count of the live threads since JVM starts
 *  STARTED-COUNT: the total count of the created threads since JVM starts
 *  DEADLOCK-COUNT: the count of deadlocked threads

## File descriptor related ##

*  MAX-FILE-DESCRIPTOR-COUNT：the count of max file descriptor JVM process can open
 *  OPEN-FILE-DESCRIPTOR-COUNT：the current count of file descriptor JVM process open

4）sysprop命令

[arthas@14]$ sysprop
     KEY                             VALUE                                                                                                                         
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     awt.toolkit                     sun.awt.X11.XToolkit                                                                                                          
     file.encoding.pkg               sun.io                                                                                                                        
     java.specification.version      1.8                                                                                                                           
     sun.cpu.isalist                                                                                                                                               
     sun.jnu.encoding                UTF-8                                                                                                                         
     java.class.path                 /data/code/game-server.jar                                                                                                    
     java.vm.vendor                  Oracle Corporation                                                                                                            
     sun.arch.data.model             64                                                                                                                            
     java.vendor.url                 http://java.oracle.com/                                                                                                       
     catalina.useNaming              false                                                                                                                         
     user.timezone                   UTC                                                                                                                           
     os.name                         Linux                                                                                                                         
     java.vm.specification.version   1.8                                                                                                                           
     @appId                          application                                                                                                                   
     user.country                    CN                                                                                                                            
     sun.java.launcher               SUN_STANDARD                                                                                                                  
     sun.boot.library.path           /App/java/jre/lib/amd64                                                                                                       
     sun.java.command                /data/code/game-server.jar -Dname=game-server --spring.profiles.active=test --spring.config.additional-location=class 
                                     path:/prometheus/                                                                                                             
     sun.cpu.endian                  little                                                                                                                        
     user.home                       /home/game                                                                                                                 
     user.language                   zh                                                                                                                            
     java.specification.vendor       Oracle Corporation                                                                                                            
     java.home                       /App/java/jre                                                                                                                 
     spring.profiles.active          test                                                                                                                          
     file.separator                  /                                                                                                                             
     line.separator                                                                                                                                                
                                                                                                                                                                   
     java.vm.specification.vendor    Oracle Corporation                                                                                                            
     java.specification.name         Java Platform API Specification                                                                                               
     java.awt.graphicsenv            sun.awt.X11GraphicsEnvironment                                                                                                
     java.awt.headless               true                                                                                                                          
     sun.boot.class.path             /App/java/jre/lib/resources.jar:/App/java/jre/lib/rt.jar:/App/java/jre/lib/sunrsasign.jar:/App/java/jre/lib/jsse.jar:/App/jav 
                                     a/jre/lib/jce.jar:/App/java/jre/lib/charsets.jar:/App/java/jre/lib/jfr.jar:/App/java/jre/classes                              
     java.protocol.handler.pkgs      org.springframework.boot.loader                                                                                               
     sun.management.compiler         HotSpot 64-Bit Tiered Compilers                                                                                               
     java.runtime.version            1.8.0_202-b08                                                                                                                 
     user.name                       game                                                                                                                       
     path.separator                  :                                                                                                                             
     os.version                      3.10.0-862.14.1.5.h428.eulerosv2r7.x86_64                                                                                     
     java.endorsed.dirs              /App/java/jre/lib/endorsed                                                                                                    
     java.runtime.name               Java(TM) SE Runtime Environment                                                                                               
     file.encoding                   UTF-8                                                                                                                         
     spring.beaninfo.ignore          true                                                                                                                          
     sun.nio.ch.bugLevel                                                                                                                                           
     java.vm.name                    Java HotSpot(TM) 64-Bit Server VM                                                                                             
     java.vendor.url.bug             http://bugreport.sun.com/bugreport/                                                                                           
     java.io.tmpdir                  /tmp                                                                                                                          
     catalina.home                   /tmp/tomcat.1473713682152073004.8080                                                                                          
     com.zaxxer.hikari.pool_number   2                                                                                                                             
     java.version                    1.8.0_202                                                                                                                     
     user.dir                        /data/code                                                                                                                    
     os.arch                         amd64                                                                                                                         
     PID                             14                                                                                                                            
     java.vm.specification.name      Java Virtual Machine Specification                                                                                            
     java.awt.printerjob             sun.print.PSPrinterJob                                                                                                        
     sun.os.patch.level              unknown                                                                                                                       
     catalina.base                   /tmp/tomcat.3200595071363069719.10106                                                                                         
     java.library.path               /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib                                                                  
     java.vm.info                    mixed mode                                                                                                                    
     java.vendor                     Oracle Corporation                                                                                                            
     java.vm.version                 25.202-b08                                                                                                                    
     java.ext.dirs                   /App/java/jre/lib/ext:/usr/java/packages/lib/ext                                                                              
     sun.io.unicode.encoding         UnicodeLittle                                                                                                                 
     java.class.version              52.0
    
    [arthas@14]$ sysprop user.country
     KEY                             VALUE                                                                                                                         
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     user.country                    CN

4）sysenv命令

[arthas@14]$ sysenv
     KEY                             VALUE                                                                                                                         
    ---------------------------------------------------------------------------------------------------------------------------------------------------------------
     PATH                            /App/java/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin                                                    
     LIVE_CLIENT_SERVICE_PORT        80                                                                                                                            
     LIVE_ADMIN_PORT_80_TCP_PROTO    tcp                                                                                                                           
     LIVE_DATA_RECOVERY_PORT_80_TCP  10.247.8.245                                                                                                                  
     _ADDR                                                                                                                                                         
     LIVE_ENTRY_PORT_2021_TCP_PORT   2021                                                                                                                          
     BUILD_DEPS                      epel-release                 net-tools                 sudo                 unzip                 which                       
     LIVE_CLIENT_PORT_80_TCP_ADDR    10.247.20.154                                                                                                                 
     LIVE_DATA_RECOVERY_PORT_80_TCP  tcp://10.247.8.245:80                                                                                                         
     LIVE_QUALITY_PORT_80_TCP_ADDR   10.247.135.39                                                                                                                 
     LIVE_ADMIN_PORT                 tcp://10.247.157.140:80                                                                                                       
     JRE_HOME                        /App/java/jre                                                                                                                 
     LIVE_ADMIN_SERVICE_PORT         80                                                                                                                            
     LIVE_LOGIC_SERVICE_PORT_HTTP    80                                                                                                                            
     PWD                             /data/code                                                                                                                    
     LIVE_LOGIC_PORT_80_TCP          tcp://10.247.16.244:80                                                                                                        
     LIVE_ENTRY_SERVICE_PORT_WS      2021                                                                                                                          
     PAYMENT_CENTER_SERVICE_HOST     10.247.221.88                                                                                                                 
     LIVE_ANCHOR_PORT_80_TCP_PROTO   tcp                                                                                                                           
     LIVE_ENTRY_PORT                 tcp://10.247.165.86:80                                                                                                        
     PAYMENT_CENTER_PORT_80_TCP_ADD  10.247.221.88                                                                                                                 
     R                                                                                                                                                             
     LC_ALL                          zh_CN.UTF-8                                                                                                                   
     LIVE_ANCHOR_SERVICE_PORT_HTTP   80                                                                                                                            
     LIVE_CLIENT_SERVICE_HOST        10.247.20.154                                                                                                                 
     LIVE_ANCHOR_PORT_80_TCP_ADDR    10.247.50.225                                                                                                                 
     KUBERNETES_SERVICE_PORT_HTTPS   443                                                                                                                           
     LIVE_ANCHOR_SERVICE_HOST        10.247.50.225                                                                                                                 
     SHLVL                           2                                                                                                                             
     LIVE_ENTRY_PORT_80_TCP          tcp://10.247.165.86:80                                                                                                        
     PAYMENT_CENTER_PORT_80_TCP_PRO  tcp                                                                                                                           
     TO                                                                                                                                                            
     LIVE_INTERNAL_SERVER_PORT_80_T  80                                                                                                                            
     CP_PORT                                                                                                                                                       
     KUBERNETES_PORT                 tcp://10.247.0.1:443                                                                                                          
     JAVA_HOME                       /App/java                                                                                                                     
     LIVE_ADMIN_PORT_80_TCP_PORT     80                                                                                                                            
     LIVE_ENTRY_PORT_80_TCP_PORT     80                                                                                                                            
     LIVE_ENTRY_SERVICE_HOST         10.247.165.86                                                                                                                 
     LIVE_ENTRY_PORT_80_TCP_PROTO    tcp                                                                                                                           
     LIVE_SERVER_PORT                tcp://10.247.201.112:80                                                                                                       
     LIVE_SERVER_SERVICE_HOST        10.247.201.112                                                                                                                
     LIVE_LOGIC_PORT_80_TCP_PORT     80                                                                                                                            
     LIVE_QUALITY_SERVICE_PORT_HTTP  80                                                                                                                            
     _                               /App/java/bin/java                                                                                                            
     LIVE_ANCHOR_PORT                tcp://10.247.50.225:80                                                                                                        
     LIVE_ADMIN_SERVICE_HOST         10.247.157.140                                                                                                                
     KUBERNETES_PORT_443_TCP_ADDR    10.247.0.1                                                                                                                    
     LIVE_INTERNAL_SERVER_PORT       tcp://10.247.163.191:80                                                                                                       
     KUBERNETES_PORT_443_TCP_PROTO   tcp                                                                                                                           
     LIVE_QUALITY_PORT               tcp://10.247.135.39:80                                                                                                        
     LIVE_ENTRY_PORT_2021_TCP        tcp://10.247.165.86:2021                                                                                                      
     LIVE_CLIENT_PORT_80_TCP_PROTO   tcp                                                                                                                           
     LIVE_QUALITY_PORT_80_TCP        tcp://10.247.135.39:80                                                                                                        
     LIVE_ENTRY_PORT_2021_TCP_ADDR   10.247.165.86                                                                                                                 
     PAYMENT_CENTER_SERVICE_PORT     80                                                                                                                            
     LIVE_LOGIC_SERVICE_HOST         10.247.16.244                                                                                                                 
     LIVE_INTERNAL_SERVER_PORT_80_T  tcp://10.247.163.191:80                                                                                                       
     CP                                                                                                                                                            
     LIVE_CLIENT_PORT_80_TCP_PORT    80                                                                                                                            
     LIVE_SERVER_PORT_80_TCP_PROTO   tcp                                                                                                                           
     LIVE_SERVER_SERVICE_PORT        80                                                                                                                            
     LIVE_SERVER_PORT_80_TCP_PORT    80                                                                                                                            
     LIVE_DATA_RECOVERY_PORT_80_TCP  80                                                                                                                            
     _PORT                                                                                                                                                         
     AppUser                         game                                                                                                                       
     LIVE_INTERNAL_SERVER_SERVICE_H  10.247.163.191                                                                                                                
     OST                                                                                                                                                           
     LIVE_ENTRY_SERVICE_PORT         80                                                                                                                            
     LIVE_INTERNAL_SERVER_PORT_80_T  tcp                                                                                                                           
     CP_PROTO                                                                                                                                                      
     LIVE_CALLBACK_PORT              tcp://10.247.121.29:80                                                                                                        
     LIVE_QUALITY_PORT_80_TCP_PORT   80                                                                                                                            
     LIVE_CALLBACK_SERVICE_PORT      80                                                                                                                            
     LIVE_QUALITY_SERVICE_PORT       80                                                                                                                            
     LIVE_CALLBACK_PORT_80_TCP_PORT  80                                                                                                                            
     LIVE_CALLBACK_PORT_80_TCP_PROT  tcp                                                                                                                           
     O                                                                                                                                                             
     LIVE_CLIENT_PORT_80_TCP         tcp://10.247.20.154:80                                                                                                        
     LIVE_DATA_RECOVERY_SERVICE_HOS  10.247.8.245                                                                                                                  
     T                                                                                                                                                             
     KUBERNETES_PORT_443_TCP         tcp://10.247.0.1:443                                                                                                          
     LIVE_ADMIN_PORT_80_TCP          tcp://10.247.157.140:80                                                                                                       
     LIVE_ADMIN_SERVICE_PORT_HTTP    80                                                                                                                            
     LIVE_INTERNAL_SERVER_SERVICE_P  80                                                                                                                            
     ORT                                                                                                                                                           
     LIVE_ANCHOR_PORT_80_TCP_PORT    80                                                                                                                            
     LIVE_LOGIC_SERVICE_PORT         80                                                                                                                            
     PAYMENT_CENTER_PORT_80_TCP_POR  80                                                                                                                            
     T                                                                                                                                                             
     LIVE_SERVER_PORT_80_TCP         tcp://10.247.201.112:80                                                                                                       
     OLDPWD                          /data                                                                                                                         
     LIVE_DATA_RECOVERY_PORT_80_TCP  tcp                                                                                                                           
     _PROTO                                                                                                                                                        
     LIVE_ENTRY_PORT_2021_TCP_PROTO  tcp                                                                                                                           
     LIVE_DATA_RECOVERY_PORT         tcp://10.247.8.245:80                                                                                                         
     LIVE_CALLBACK_SERVICE_HOST      10.247.121.29                                                                                                                 
     PAAS_POD_ID                     a960be08-0975-492a-bed3-1c1dd975e9ed                                                                                          
     LIVE_QUALITY_SERVICE_HOST       10.247.135.39                                                                                                                 
     PAYMENT_CENTER_PORT             tcp://10.247.221.88:80                                                                                                        
     LIVE_ADMIN_PORT_80_TCP_ADDR     10.247.157.140                                                                                                                
     LIVE_ENTRY_PORT_80_TCP_ADDR     10.247.165.86                                                                                                                 
     LIVE_INTERNAL_SERVER_PORT_80_T  10.247.163.191                                                                                                                
     CP_ADDR                                                                                                                                                       
     LIVE_LOGIC_PORT_80_TCP_ADDR     10.247.16.244                                                                                                                 
     Env                             test                                                                                                                          
     KUBERNETES_SERVICE_HOST         10.247.0.1                                                                                                                    
     LIVE_ANCHOR_SERVICE_PORT        80                                                                                                                            
     LIVE_ENTRY_SERVICE_PORT_HTTP    80                                                                                                                            
     LIVE_DATA_RECOVERY_SERVICE_POR  80                                                                                                                            
     T_HTTP                                                                                                                                                        
     AppGroup                        game                                                                                                                       
     PAYMENT_CENTER_SERVICE_PORT_HT  80                                                                                                                            
     TP                                                                                                                                                            
     LIVE_SERVER_PORT_80_TCP_ADDR    10.247.201.112                                                                                                                
     LIVE_SERVER_SERVICE_PORT_HTTP   80                                                                                                                            
     CLASSPATH                       .:/App/java/jre/lib/rt.jar:/App/java/lib/dt.jar:/App/java/lib/tools.jar                                                       
     KUBERNETES_SERVICE_PORT         443                                                                                                                           
     LIVE_CALLBACK_PORT_80_TCP_ADDR  10.247.121.29                                                                                                                 
     LIVE_LOGIC_PORT                 tcp://10.247.16.244:80                                                                                                        
     HOSTNAME                        game-server-76c7dcc4bc-9ntlf                                                                                                  
     LIVE_CALLBACK_PORT_80_TCP       tcp://10.247.121.29:80                                                                                                        
     PAYMENT_CENTER_PORT_80_TCP      tcp://10.247.221.88:80                                                                                                        
     LIVE_QUALITY_PORT_80_TCP_PROTO  tcp                                                                                                                           
     LIVE_CLIENT_PORT                tcp://10.247.20.154:80                                                                                                        
     KUBERNETES_PORT_443_TCP_PORT    443                                                                                                                           
     LIVE_DATA_RECOVERY_SERVICE_POR  80                                                                                                                            
     T                                                                                                                                                             
     LIVE_LOGIC_PORT_80_TCP_PROTO    tcp                                                                                                                           
     LIVE_CLIENT_SERVICE_PORT_HTTP   80                                                                                                                            
     LIVE_INTERNAL_SERVER_SERVICE_P  80                                                                                                                            
     ORT_HTTP                                                                                                                                                      
     LIVE_ANCHOR_PORT_80_TCP         tcp://10.247.50.225:80                                                                                                        
     LIVE_CALLBACK_SERVICE_PORT_HTT  80                                                                                                                            
     P                                                                                                                                                             
     HOME                            /home/game

## 四、高阶使用 ##

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZFTkdRSVlVTlJBTg_size_16_color_FFFFFF_t_70]: /images/20221024/01f0ba2300604c389bce3df2d7c8a997.png
[https_arthas.aliyun.com_en-us]: https://arthas.aliyun.com/en-us/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0ZFTkdRSVlVTlJBTg_size_16_color_FFFFFF_t_70 1]: /images/20221024/ca2d51d004694c168795c18ceb18eb71.png