prometheus监控java程序

àì夳堔傛蜴生んèń 2022-01-06 01:53 175阅读 0赞

> 可参考链接，需找开发商量jmx端口暴露方式
> 
> [https://www.jianshu.com/p/8a5e681b18ce][https_www.jianshu.com_p_8a5e681b18ce] 或者 [http://www.mamicode.com/info-detail-2323750.html][http_www.mamicode.com_info-detail-2323750.html]
> 
> 上图两个链接为不同的jmx暴露方式，实则大同小异，需跟开发商量，以免程序启动不了

**下载jmx\_exporter的jar包**

[https://repo1.maven.org/maven2/io/prometheus/jmx/jmx\_prometheus\_javaagent/0.3.1/jmx\_prometheus\_javaagent-0.3.1.jar][https_repo1.maven.org_maven2_io_prometheus_jmx_jmx_prometheus_javaagent_0.3.1_jmx_prometheus_javaagent-0.3.1.jar]

创建配置文件namenode.yaml(datanode.yaml)放在任意位置，内容为你想要的metrics

参考配置：

---
    startDelaySeconds: 0
    hostPort: master:1234 #master为本机IP（一般可设置为localhost）；1234为想设置的jmx端口（可设置为未被占用的端口）
    #jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:1234/jmxrmi
    ssl: false
    lowercaseOutputName: false
    lowercaseOutputLabelNames: false

其他参数参考：

<table> 
 <thead> 
  <tr> 
   <th>Name</th> 
   <th>Description</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>startDelaySeconds</td> 
   <td>start delay before serving requests. Any requests within the delay period will result in an empty metrics set.</td> 
  </tr> 
  <tr> 
   <td>hostPort</td> 
   <td>The host and port to connect to via remote JMX. If neither this nor jmxUrl is specified, will talk to the local JVM.</td> 
  </tr> 
  <tr> 
   <td>username</td> 
   <td>The username to be used in remote JMX password authentication.</td> 
  </tr> 
  <tr> 
   <td>password</td> 
   <td>The password to be used in remote JMX password authentication.</td> 
  </tr> 
  <tr> 
   <td>jmxUrl</td> 
   <td>A full JMX URL to connect to. Should not be specified if hostPort is.</td> 
  </tr> 
  <tr> 
   <td>ssl</td> 
   <td>Whether JMX connection should be done over SSL. To configure certificates you have to set following system properties:<br><code>-Djavax.net.ssl.keyStore=/home/user/.keystore</code><br><code>-Djavax.net.ssl.keyStorePassword=changeit</code><br><code>-Djavax.net.ssl.trustStore=/home/user/.truststore</code><br><code>-Djavax.net.ssl.trustStorePassword=changeit</code></td> 
  </tr> 
  <tr> 
   <td>lowercaseOutputName</td> 
   <td>Lowercase the output metric name. Applies to default format and&nbsp;<code>name</code>. Defaults to false.</td> 
  </tr> 
  <tr> 
   <td>lowercaseOutputLabelNames</td> 
   <td>Lowercase the output metric label names. Applies to default format and&nbsp;<code>labels</code>. Defaults to false.</td> 
  </tr> 
  <tr> 
   <td>whitelistObjectNames</td> 
   <td>A list of&nbsp;<a href="http://docs.oracle.com/javase/6/docs/api/javax/management/ObjectName.html" rel="nofollow">ObjectNames</a>&nbsp;to query. Defaults to all mBeans.</td> 
  </tr> 
  <tr> 
   <td>blacklistObjectNames</td> 
   <td>A list of&nbsp;<a href="http://docs.oracle.com/javase/6/docs/api/javax/management/ObjectName.html" rel="nofollow">ObjectNames</a>&nbsp;to not query. Takes precedence over&nbsp;<code>whitelistObjectNames</code>. Defaults to none.</td> 
  </tr> 
  <tr> 
   <td>rules</td> 
   <td>A list of rules to apply in order, processing stops at the first matching rule. Attributes that aren‘t matched aren‘t collected. If not specified, defaults to collecting everything in the default format.</td> 
  </tr> 
  <tr> 
   <td>pattern</td> 
   <td>Regex pattern to match against each bean attribute. The pattern is not anchored. Capture groups can be used in other options. Defaults to matching everything.</td> 
  </tr> 
  <tr> 
   <td>attrNameSnakeCase</td> 
   <td>Converts the attribute name to snake case. This is seen in the names matched by the pattern and the default format. For example, anAttrName to an_attr_name. Defaults to false.</td> 
  </tr> 
  <tr> 
   <td>name</td> 
   <td>The metric name to set. Capture groups from the&nbsp;<code>pattern</code>&nbsp;can be used. If not specified, the default format will be used. If it evaluates to empty, processing of this attribute stops with no output.</td> 
  </tr> 
  <tr> 
   <td>value</td> 
   <td>Value for the metric. Static values and capture groups from the&nbsp;<code>pattern</code>&nbsp;can be used. If not specified the scraped mBean value will be used.</td> 
  </tr> 
  <tr> 
   <td>valueFactor</td> 
   <td>Optional number that&nbsp;<code>value</code>&nbsp;(or the scraped mBean value if&nbsp;<code>value</code>&nbsp;is not specified) is multiplied by, mainly used to convert mBean values from milliseconds to seconds.</td> 
  </tr> 
  <tr> 
   <td>labels</td> 
   <td>A map of label name to label value pairs. Capture groups from&nbsp;<code>pattern</code>&nbsp;can be used in each.&nbsp;<code>name</code>&nbsp;must be set to use this. Empty names and values are ignored. If not specified and the default format is not being used, no labels are set.</td> 
  </tr> 
  <tr> 
   <td>help</td> 
   <td>Help text for the metric. Capture groups from&nbsp;<code>pattern</code>&nbsp;can be used.&nbsp;<code>name</code>&nbsp;must be set to use this. Defaults to the mBean attribute decription and the full name of the attribute.</td> 
  </tr> 
  <tr> 
   <td>type</td> 
   <td>The type of the metric, can be&nbsp;<code>GAUGE</code>,&nbsp;<code>COUNTER</code>&nbsp;or&nbsp;<code>UNTYPED</code>.&nbsp;<code>name</code>&nbsp;must be set to use this. Defaults to&nbsp;<code>UNTYPED</code>.</td> 
  </tr> 
 </tbody> 
</table>

## 前提： ##

1.java程序的目录结构展示（因断网环境所以没办法用树状图命令tree来展示，只能截图）

![20190624112600229.png][]

如上图所示，一个服务器有一个java程序，其中子程序包括manager(管理端), server，client，PRM, Utilities，关联关系是agent和server要从manager管理端获取各种参数，server再向agent发包，这样程序就算跑起来了。

这里主要监控server和client

下图为manager和server以及其中一个client的目录结构

![2019062414172382.png][]

## ![20190624141733831.png][] ##

## ![20190624141825325.png][] ##

各conf目录下都有一个namenode.conf,内容如下

> startDelaySeconds: 10  
> jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:$\{JMX\_PORT\}/$\{APP\_NAME\}  
> ssl: false  
> lowercaseOutputName: false  
> lowercaseOutputLabelNames: false

各conf目录下都有一个namenode.yaml,内容如下(此为manager下conf中的)

![20190624142354774.png][]

各个子程序bin目录下都有关于程序的启动脚本，先执行1脚本启动，然后2脚本是让程序开启一个prometheus jmx端口，以便jmx exporter探针找到这个端口，收集到java程序的metrics信息，达到监控的目的

![20190624142841931.png][]

下图是npds-manager.sh脚本中实现开启jmx端口的脚本信息，完整脚本信息在最下面

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70][]

这样启动java程序后，访问ip加自定义的端口9210就能访问到manager的mertrics信息了,达到监控的目的

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70 1][]

然后找开发要他们想要监控的参数，以我为例要的参数如下图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70 2][]

再以此一个个配置，就能实现监控了

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70 3][]

> vim npds-manager.sh
> 
> \#!/bin/bash  
> then  
>     echo "=================================================================="  
>     echo "ERROR: Please set the JAVA\_HOME variable in your environment!"  
>     echo "=================================================================="  
>     exit 20  
> fi
> 
> cd=$(pwd)
> 
> if \[ $\{cd:0-3\} != "bin" \]  
> then  
>     echo "=================================================================="  
>     echo "ERROR: Please execute the script in the bin directory!"  
>     echo "=================================================================="  
>     exit 21  
> fi
> 
> APP\_HOME=$\{cd%/bin\}
> 
> APP\_NAME=NPDS-Manager
> 
> APP\_MAIN\_CLASS=cn.com.greattimes.npds.manager.boot.ManagerBootstrap
> 
> JPS\_FLAG=ManagerBootstrap
> 
> LIBRARY\_PATH=$APP\_HOME/jni
> 
>   
> JAVA\_OPTS="-server  
> \-Xmx1G  
> \-Xms1G"
> 
> DEBUG=0
> 
> RET\_CODE=0
> 
> FORCE\_KILL=0
> 
> CHECK\_STATUS=1
> 
> CHECK\_STATUS\_TIMEOUT=5
> 
> check\_env() \{  
>     local DIR=$\{APP\_HOME:POS+1\}
> 
>     if \[ ! $DIR == $APP\_NAME \]  
>     then  
>         echo "=================================================================="  
>         exit 22  
>     fi
> 
>     if \[ ! -d "$APP\_HOME/logs" \]  
>     then  
>         mkdir $APP\_HOME/logs  
>     fi
> 
>     if \[ $DEBUG -eq 1 \]  
>     then  
>         local DBG\_PORT=9310  
>     fi  
> \}
> 
> check\_prometheus() \{  
>     if \[ -e "$APP\_HOME/conf/namenode.conf" \]  
>     then  
>         local JMX\_PORT=15914  
>     fi
> 
>     if \[ -e "$APP\_HOME/conf/namenode.yaml" \]  
>     then  
>         local WEB\_PORT=9210  
>     fi  
> \}
> 
> check() \{  
>     PID=0  
>     local TMP=$(jps -v | grep $JPS\_FLAG | grep $APP\_HOME)  
>     TMP=$\{TMP%% \*\}
> 
>     if \[ $\{\#TMP\} -ne 0 \]  
>     then  
>         PID=$TMP  
>     fi
> 
>     if \[ $PID -ne 0 \]  
>     then  
>         RUNNING=1  
>     else  
>         RUNNING=0  
>     fi  
> \}
> 
> print() \{  
>     if \[ $QUIET\_MODE -eq 1 \]  
>     then  
>         return  
>     fi
> 
>     echo "$1" "$2"  
> \}  
>     if \[ ! -n "$1" \]  
>     then  
>         return 2  
>     fi
> 
>     ps -p $1 > /dev/null
> 
>     if \[ $? -eq 0 \]  
>     then  
>         kill -9 $1  
>     fi  
> \}
> 
> start() \{  
>     check
> 
>     if \[ $RUNNING -eq 1 \]  
>     then  
>         print "=================================================================="  
>         print "WARN: $APP\_NAME(PID=$PID) is running."  
>         print "=================================================================="  
>         return 1  
>     fi
> 
>     info
> 
>     if \[ $CHECK\_STATUS -eq 0 \]  
>     then  
>         print -n "..."  
>         print  
>         print "------------------------------------------------------------------"  
>         print "WARN: $APP\_NAME is started without checking status."  
>         print  
>         return 1  
>     fi
> 
>     local n=0
> 
>     for ((; n<$CHECK\_STATUS\_TIMEOUT; n++))  
>     do  
>         if \[ $n -lt 3 \]  
>         then  
>             sleep 1  
>             print -n "."  
>         else  
>             check
> 
>             if \[ $RUNNING -eq 1 \]  
>             then  
>                 print  
>                 print "Done!"  
>                 print  
>                 return 0  
>             else  
>                 sleep 1  
>             fi  
>         fi  
>     done
> 
>     print  
>     print "------------------------------------------------------------------"  
>     print "ERROR: $APP\_NAME failed to start for unknown reasons. "  
>     print "       Please refer to the log file for details."  
>     print  
>     return 2  
> stop() \{  
>     check
> 
>     if \[ $RUNNING -eq 0 \]  
>     then  
>         print "=================================================================="  
>         print "WARN: $APP\_NAME is stopped."  
>         print "=================================================================="  
>         return 1  
>     fi
> 
>     info  
>     print -n "Stopping $APP\_NAME"
> 
>     if \[ $FORCE\_KILL -eq 1 \]  
>     then  
>         print -n "..."  
>         print
> 
>         kill\_pid $PID
> 
>         print "------------------------------------------------------------------"  
>         print "WARN: $APP\_NAME was killed by force."  
>         print  
>         return 1
> 
>     if \[ $CHECK\_STATUS -eq 0 \]  
>     then  
>         print -n "..."  
>         print  
>         print "------------------------------------------------------------------"  
>         print "WARN: $APP\_NAME is stopped without checking status."  
>         print  
>         return 1  
>     fi
> 
>     local n=0
> 
>     for ((; n<$CHECK\_STATUS\_TIMEOUT; n++))  
>     do  
>         if \[ $n -lt 3 \]  
>         then  
>             sleep 1  
>             print -n "."  
>         else  
>             check
> 
>             if \[ $RUNNING -eq 0 \]  
>             then  
>                 print  
>                 print "Done!"  
>                 print  
>                 return 0  
>             else  
>                 sleep 1  
>             fi  
>         fi  
>     done
> 
>     kill\_pid $PID
> 
>     print  
>     print "------------------------------------------------------------------"  
>     print "WARN: $APP\_NAME was killed for stopping timeout."  
>     print  
>     return 1  
> \}
> 
> restart() \{  
>     check
> 
>     if \[ $RUNNING -eq 1 \]  
>     then  
>         if \[ $CHECK\_STATUS -eq 0 \]  
>         then  
>             FORCE\_KILL=1  
>         fi  
>         stop  
>     fi
> 
>     start  
> \}
> 
> status() \{  
>     check
> 
>     if \[ $RUNNING -eq 0 \]  
>     then  
>         print "=================================================================="  
>         print "INFO: $APP\_NAME is stopped."  
>         print "=================================================================="  
>         return 0  
>     else  
>         print "=================================================================="  
>         print "INFO: $APP\_NAME(PID=$PID) is running."  
>         print "=================================================================="  
>         return 1  
>     fi  
> \}
> 
> info() \{  
>     if \[ $QUIET\_MODE -eq 1 \]  
>     then  
>         return  
>     fi
> 
>     echo "=================================================================="  
>     echo "$APP\_NAME Information:"  
>     echo "------------------------------------------------------------------"  
>     echo "JAVA\_HOME=$JAVA\_HOME"  
>     echo "CLASS\_PATH=$CLASS\_PATH"  
>     echo  
>     echo \`$JAVA\_HOME/bin/java -version\`  
>     echo "APP\_HOME=$APP\_HOME"  
>     echo "APP\_MAIN\_CLASS=$APP\_MAIN\_CLASS"  
>     echo "=================================================================="  
> \}
> 
> check\_env
> 
> if \[ $\# -gt 1 \]  
> then  
>     for v in $@  
>     do  
>         if \[ "$v" == "$1" \]  
>         then  
>             continue  
>         fi  
>         case "$v" in  
>             '--quiet')  
>                 QUIET\_MODE=1  
>                 ;;  
>             '--nocheck')  
>                 CHECK\_STATUS=0  
>                 ;;  
>             '--force')  
>                 FORCE\_KILL=1  
>                 ;;  
>             \*)  
>                 ;;  
>         esac  
>     done  
> fi
> 
> case "$1" in  
>     'start')  
>         check\_prometheus  
>         start  
>         ;;  
>     'stop')  
>         check\_prometheus  
>         stop  
>         ;;  
>     'restart')  
>         check\_prometheus  
>         restart  
>         ;;  
>     'status')  
>         status  
>         ;;  
>     'info')  
>         info  
>         ;;  
>     \*)  
>         echo "=================================================================="  
>         echo "Usage: $0 <commands> \[options\]"  
>         echo "------------------------------------------------------------------"  
>         echo "where commands include:"  
>         echo  
>         echo -e "      start \[options\]"  
>         echo -e "            --quiet\\twithout console output"  
>         echo -e "            --nocheck\\twithout checking status"  
>         echo  
>         echo -e "       stop \[options\]"  
>         echo -e "            --quiet\\twithout console output"  
>         echo -e "            --nocheck\\twithout checking status"  
>         echo -e "            --force\\tforce-kill"  
>         echo  
>         echo -e "    restart \[options\]"  
>         echo -e "            --quiet\\twithout console output"  
>         echo -e "            --nocheck\\twithout checking status"  
>         echo -e "            --force\\tstart after force-kill"  
>         echo  
>         echo -e "     status \\t\\tdisplay running status"  
>         echo  
>         echo -e "       info \\t\\tdisplay environment information"  
>         echo "=================================================================="  
>         ;;  
> esac
> 
> RET\_CODE=\`echo $?\`
> 
> exit $RET\_CODE  
>

[https_www.jianshu.com_p_8a5e681b18ce]: https://www.jianshu.com/p/8a5e681b18ce
[http_www.mamicode.com_info-detail-2323750.html]: http://www.mamicode.com/info-detail-2323750.html
[https_repo1.maven.org_maven2_io_prometheus_jmx_jmx_prometheus_javaagent_0.3.1_jmx_prometheus_javaagent-0.3.1.jar]: https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
[20190624112600229.png]: /images/20211228/e6601a0a844443c984551f3a76ff7041.png
[2019062414172382.png]: /images/20211228/037a17aa35bf4f98b004020f3fd5e91c.png
[20190624141733831.png]: /images/20211228/cb8545aadc124bf6bd214f533419193a.png
[20190624141825325.png]: /images/20211228/9d5b3959fb1a48ffa0ba0f0c92173257.png
[20190624142354774.png]: /images/20211228/4941ab08ec224bec9c5021c9fcf31b02.png
[20190624142841931.png]: /images/20211228/1b1b4891d6e3468a88e455572d3ff818.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70]: /images/20211228/1da3343fe0e2428da10902c7f94ef435.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70 1]: /images/20211228/06f98b046f984eed8095158c276f44d1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70 2]: /images/20211228/ea80ca26ad774cbb9db07c7845ba57ec.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3pob3U3NTc3MTIxNw_size_16_color_FFFFFF_t_70 3]: /images/20211228/005e1d81df0048cb924c6adc1bb9b94a.png