Zabbix监控实战-Tomcat监控

左手的ㄟ右手 2022-11-08 14:09 297阅读 0赞

## 八、Zabbix监控实战-Tomcat监控 ##

### 1、方法一：开发java监控页面 ###

[root@qfedu.com tomcat8_1]# cat /application/tomcat/webapps/memtest/meminfo.jsp 
 <%
 Runtime rtm = Runtime.getRuntime();
 long mm = rtm.maxMemory()/1024/1024;
 long tm = rtm.totalMemory()/1024/1024;
 long fm = rtm.freeMemory()/1024/1024;
 
 out.println("JVM memory detail info : ");
 out.println("Max memory:"+mm+"MB"+" ");
 out.println("Total memory:"+tm+"MB"+" ");
 out.println("Free memory:"+fm+"MB"+" ");
 out.println("Available memory can be used is :"+(mm+fm-tm)+"MB"+" ");
 %>

### 2、方法二：使用jps命令进行监控 ###

[root@qfedu.com ~]# jps -lvm
    
    31906 org.apache.catalina.startup.Bootstrap start -Djava.util.logging.config.file=/application/tomcat8_1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/application/tomcat8_1/endorsed -Dcatalina.base=/application/tomcat8_1 -Dcatalina.home=/application/tomcat8_1 -Djava.io.tmpdir=/application/tomcat8_1/temp
    
    31812 org.apache.catalina.startup.Bootstrap start -Djava.util.logging.config.file=/application/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/application/tomcat/endorsed -Dcatalina.base=/application/tomcat -Dcatalina.home=/application/tomcat -Djava.io.tmpdir=/application/tomcat/temp
    
    31932 org.apache.catalina.startup.Bootstrap start -Djava.util.logging.config.file=/application/tomcat8_2/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/application/tomcat8_2/endorsed -Dcatalina.base=/application/tomcat8_2 -Dcatalina.home=/application/tomcat8_2 -Djava.io.tmpdir=/application/tomcat8_2/temp
    
    32079 sun.tools.jps.Jps -lvm -Denv.class.path=.:/application/jdk/lib:/application/jdk/jre/lib:/application/jdk/lib/tools.jar -Dapplication.home=/application/jdk1.8.0_60 -Xms8m

### 3、Tomcat 远程监控功能 ###

#### 1、修改配置文件，开启远程监控 ####

[root@qfedu.com ~]# vim /application/tomcat8_1/bin/catalina.sh +97
    CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=10.0.0.17"

#### 2、重启服务，检查12345端口是否开启 ####

[root@qfedu.com ~]# /application/tomcat8_1/bin/shutdown.sh 
    [root@qfedu.com ~]# /application/tomcat8_1/bin/startup.sh 
    [root@qfedu.com ~]# netstat -tunlp|grep 12345

#### 3、 检查端口 ####

[root@qfedu.com ~]# netstat -tunlp|grep 12345
    tcp6       0      0 :::12345           :::*          LISTEN      33158/java

4、在windows上监控tomcat

**注意：windwos需要安装jdk环境！**

查考：http://www.oracle.com/technetwork/java/javase/downloads/index.html

##### 1、软件路径 #####

C:\Program Files\Java\jdk1.8.0_31\bin
    jconsole.exe   jvisualvm.exe

2、jconsole.exe 使用说明

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70]

连接成功即可进行监控，连接的时候注意端口信息。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 1]

3、jvisualvm.exe 使用说明

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 2]  
**输入ip地址**

![在这里插入图片描述][20210313174104705.png]

**主机添加完成，添加JMX监控**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 3]  
**注意添加的时候输入端口信息**

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 4]

添加完成后就能够多tomcat程序进行监控。

### 4、zabbix 监控 tomcat 程序 ###

##### 1、zabbix 服务端安装配置 java监控服务 #####

[root@qfedu.com ~]# yum install zabbix-java-gateway -y

##### 2、查看配置文件 #####

配置文件路径：/etc/zabbix/zabbix_java_gateway.conf
    [root@qfedu.com ~]# sed -i -e '220a JavaGateway=127.0.0.1' -e '236a StartJavaPollers=5' /etc/zabbix/zabbix_server.conf

##### 3、启动zabbix-java-gateway服务，与zabbix服务 #####

[root@qfedu.com ~]# systemctl start zabbix-java-gateway.service
    [root@qfedu.com ~]# systemctl restart zabbix-server.service

##### 4、检查java端口是否开启 #####

[root@qfedu.com ~]# netstat -lntup |grep java
    tcp6       0      0 :::10052   :::*    LISTEN      72971/java

##### 5、检查java进程是否存在 #####

[root@qfedu.com ~]# ps -ef |grep [j]ava
    zabbix    72971      1  0 11:29 ?        00:00:00 java -server -Dlogback.configurationFile=/etc/zabbix/zabbix_java_gateway_logback.xml -classpath lib:lib/android-json-4.3_r3.1.jar:lib/logback-classic-0.9.27.jar:lib/logback-core-0.9.27.jar:lib/slf4j-api-1.6.1.jar:bin/zabbix-java-gateway-3.0.13.jar -Dzabbix.pidFile=/var/run/zabbix/zabbix_java.pid -Dzabbix.timeout=3 -Dsun.rmi.transport.tcp.responseTimeout=3000 com.zabbix.gateway.JavaGateway
    zabbix    73255  73226  0 11:35 ?        00:00:00 /usr/sbin/zabbix_server: java poller #1 [got 0 values in 0.000002 sec, idle 5 sec]
    zabbix    73256  73226  0 11:35 ?        00:00:00 /usr/sbin/zabbix_server: java poller #2 [got 0 values in 0.000002 sec, idle 5 sec]
    zabbix    73257  73226  0 11:35 ?        00:00:00 /usr/sbin/zabbix_server: java poller #3 [got 0 values in 0.000002 sec, idle 5 sec]
    zabbix    73258  73226  0 11:35 ?        00:00:00 /usr/sbin/zabbix_server: java poller #4 [got 0 values in 0.000002 sec, idle 5 sec]
    zabbix    73259  73226  0 11:35 ?        00:00:00 /usr/sbin/zabbix_server: java poller #5 [got 0 values in 0.000004 sec, idle 5 sec]

##### 6、web界面添加 #####

###### 1、添加主机 ######

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 5]  
2、主机管理模板，注意是JMX模板

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 6]  
3、监控完成

![在这里插入图片描述][20210313174218939.png]

### 5、排除 tomcat 故障步骤 ###

###### 1、查看 catalina.out ######

###### 2、使用 sh show-busy-java-threads.sh 脚本进行检测 ######

#!/bin/bash
 # @Function
 # Find out the highest cpu consumed threads of java, and print the stack of these threads.
 #
 # @Usage
 # $ ./show-busy-java-threads.sh
 #
 # @author Jerry Lee
 
 readonly PROG=`basename $0`
 readonly -a COMMAND_LINE=("$0" "$@")
 
 usage() { 
 cat <<EOF Usage: ${PROG} [OPTION]... Find out the highest cpu consumed threads of java, and print the stack of these threads. Example: ${PROG} -c 10 Options: -p, --pid find out the highest cpu consumed threads from the specifed java process, default from all java process. -c, --count set the thread count to show, default is 5 -h, --help display this help and exit EOF
 exit $1
 }
 
 readonly ARGS=`getopt -n "$PROG" -a -o c:p:h -l count:,pid:,help -- "$@"`
 [ $? -ne 0 ] && usage 1
 eval set -- "${ARGS}"
 
 while true; do
 case "$1" in
 -c|--count)
 count="$2"
 shift 2
 ;;
 -p|--pid)
 pid="$2"
 shift 2
 ;;
 -h|--help)
 usage
 ;;
 --)
 shift
 break
 ;;
 esac
 done
 count=${count:-5}
 
 redEcho() { 
 [ -c /dev/stdout ] && { 
 # if stdout is console, turn on color output.
 echo -ne "\033[1;31m"
 echo -n "$@"
 echo -e "\033[0m"
 } || echo "$@"
 }
 
 yellowEcho() { 
 [ -c /dev/stdout ] && { 
 # if stdout is console, turn on color output.
 echo -ne "\033[1;33m"
 echo -n "$@"
 echo -e "\033[0m"
 } || echo "$@"
 }
 
 blueEcho() { 
 [ -c /dev/stdout ] && { 
 # if stdout is console, turn on color output.
 echo -ne "\033[1;36m"
 echo -n "$@"
 echo -e "\033[0m"
 } || echo "$@"
 }
 
 # Check the existence of jstack command!
 if ! which jstack &> /dev/null; then
 [ -z "$JAVA_HOME" ] && { 
 redEcho "Error: jstack not found on PATH!"
 exit 1
 }
 ! [ -f "$JAVA_HOME/bin/jstack" ] && { 
 redEcho "Error: jstack not found on PATH and $JAVA_HOME/bin/jstack file does NOT exists!"
 exit 1
 }
 ! [ -x "$JAVA_HOME/bin/jstack" ] && { 
 redEcho "Error: jstack not found on PATH and $JAVA_HOME/bin/jstack is NOT executalbe!"
 exit 1
 }
 export PATH="$JAVA_HOME/bin:$PATH"
 fi
 
 readonly uuid=`date +%s`_${RANDOM}_$$
 
 cleanupWhenExit() { 
 rm /tmp/${uuid}_* &> /dev/null
 }
 trap "cleanupWhenExit" EXIT
 
 printStackOfThread() { 
 local line
 local count=1
 while IFS=" " read -a line ; do
 local pid=${line[0]}
 local threadId=${line[1]}
 local threadId0x=`printf %x ${ threadId}`
 local user=${line[2]}
 local pcpu=${line[4]}
 
 local jstackFile=/tmp/${uuid}_${pid}
 
 [ ! -f "${jstackFile}" ] && { 
 { 
 if [ "${user}" == "${USER}" ]; then
 jstack ${pid} > ${jstackFile}
 else
 if [ $UID == 0 ]; then
 sudo -u ${user} jstack ${pid} > ${jstackFile}
 else
 redEcho "[$((count++))] Fail to jstack Busy(${pcpu}%) thread(${threadId}/0x${threadId0x}) stack of java process(${pid}) under user(${user})."
 redEcho "User of java process($user) is not current user($USER), need sudo to run again:"
 yellowEcho " sudo ${COMMAND_LINE[@]}"
 echo
 continue
 fi
 fi
 } || { 
 redEcho "[$((count++))] Fail to jstack Busy(${pcpu}%) thread(${threadId}/0x${threadId0x}) stack of java process(${pid}) under user(${user})."
 echo
 rm ${jstackFile}
 continue
 }
 }
 blueEcho "[$((count++))] Busy(${pcpu}%) thread(${threadId}/0x${threadId0x}) stack of java process(${pid}) under user(${user}):"
 sed "/nid=0x${threadId0x} /,/^$/p" -n ${jstackFile}
 done
 }
 
 
 ps -Leo pid,lwp,user,comm,pcpu --no-headers | { 
 [ -z "${pid}" ] &&
 awk '$4=="java"{print $0}' ||
 awk -v "pid=${pid}" '$1==pid,$4=="java"{print $0}'
 } | sort -k5 -r -n | head --lines "${count}" | printStackOfThread

##  ##

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70]: /images/20221023/88d6f002960948969d6fc61982ffce83.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 1]: /images/20221023/9ef912949af949af97f269a2037a99a2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 2]: /images/20221023/41caa2370435444cbf2eac1514885763.png
[20210313174104705.png]: /images/20221023/c1da0013c91f4ab391ab946df23516ab.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 3]: /images/20221023/1aaee4df8e8b48dabd55d7736fd0689b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 4]: /images/20221023/4fb2a0d2899f4e859c27ab4317c2dbd1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 5]: /images/20221023/4d5da89880aa4044a6ccfce3814dc756.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQzMTkzMzg2_size_16_color_FFFFFF_t_70 6]: /images/20221023/6d09aa6ec5854361afdcd53f10ebb134.png
[20210313174218939.png]: /images/20221023/4ce1d416615a45b9b249a67afb4ae85f.png