Weblogic读取blob堵塞的原因

雨点打透心脏的1/2处 2022-06-02 01:43 75阅读 0赞

很久之前的一个问题今天终于得到了答案。  
  之前是有这么一个stuck请求，它上了一个锁导致后来的数据库链接都在等待这个锁，上次是没有找到原因，因为只是读取一条数据的一个blob字段，这个字段并不大。  
"\[STUCK\] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f6108009800 nid=0x46bd runnable \[0x00007f60d1ad0000\]  
   java.lang.Thread.State: RUNNABLE  
at java.net.SocketInputStream.socketRead0(Native Method)  
at java.net.SocketInputStream.read(SocketInputStream.java:129)  
at oracle.net.ns.Packet.receive(Packet.java:293)  
at oracle.net.ns.DataPacket.receive(DataPacket.java:104)  
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)  
at oracle.net.ns.NetInputStream.read(NetInputStream.java:260)  
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:105)  
at oracle.jdbc.driver.T4CMAREngine.getNBytes(T4CMAREngine.java:1504)  
at oracle.jdbc.driver.T4C8TTILobd.unmarshalLobData(T4C8TTILobd.java:450)  
at oracle.jdbc.driver.T4C8TTILob.readLOBD(T4C8TTILob.java:767)  
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:358)  
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)  
at oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:146)  
at oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:2346)  
\- locked <0x0000000703ccd180> (a oracle.jdbc.driver.T4CConnection)  
at oracle.sql.BLOB.getBytes(BLOB.java:331)  
at oracle.sql.BLOB.getBytes(BLOB.java:217)  
at oracle.jdbc.driver.T4CBlobAccessor.getBytes(T4CBlobAccessor.java:464)  
at oracle.jdbc.driver.OracleResultSetImpl.getBytes(OracleResultSetImpl.java:708)  
\- locked <0x0000000703ccd180> (a oracle.jdbc.driver.T4CConnection)  
at oracle.jdbc.driver.OracleResultSet.getBytes(OracleResultSet.java:398)  
at weblogic.jdbc.wrapper.ResultSet\_oracle\_jdbc\_driver\_OracleResultSetImpl.getBytes(Unknown Source)  
at com.gg.bpms.engine.deploy.model.DefinitionModelVOConvert.voConvert(DefinitionModelVOConvert.java:80)  
.................................................................  
  
  
**今天有一个服务发生了告警，这是server日志的报的错，是读取一个外系统的功能，由于外系统的功能出了问题，所以一直都连不上，stuck也是相关的错误。**  
<BEA-000000> <2017/12/29 01:05:05:740 CST \[ERROR\] CommonUtils - Tried all: '1' addresses, but could not connect over HTTP to server: '10.10.181.132', port: '80' <java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: '10.10.181.132', port: '80'>java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: '10.10.181.132', port: '80'  
at weblogic.net.http.HttpClient.openServer(HttpClient.java:333)  
at weblogic.net.http.HttpClient.openServer(HttpClient.java:425)  
at weblogic.net.http.HttpClient.New(HttpClient.java:252)  
at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:189)  
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:374)  
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)  
at org.jasig.cas.client.javafilter.util.CommonUtils.getResponseFromServer(CommonUtils.java:392)  
  
<\[STUCK\] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "645" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@7719c697\[  
GET /web/gg/Login.jsp?verifyCode=123 HTTP/1.1  
User-Agent: Jodd HTTP  
\]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:  
java.net.SocketInputStream.socketRead0(Native Method)  
java.net.SocketInputStream.read(SocketInputStream.java:129)  
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)  
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)  
java.io.BufferedInputStream.read(BufferedInputStream.java:317)  
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:224)  
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:148)  
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)  
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:377)  
weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)  
org.jasig.cas.client.javafilter.util.CommonUtils.getResponseFromServer(CommonUtils.java:392)  
  
**下面是这个服务jstack的结果：**  
"\[STUCK\] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f99e4005000 nid=0x400f runnable \[0x00007f99947c1000\]  
   java.lang.Thread.State: RUNNABLE  
at java.net.SocketInputStream.socketRead0(Native Method)  
at java.net.SocketInputStream.read(SocketInputStream.java:129)  
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)  
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)  
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)  
\- locked <0x0000000716c42dd8> (a java.io.BufferedInputStream)  
at weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:224)  
at weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:148)  
at weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:468)  
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:377)  
\- locked <0x0000000716c43048> (a weblogic.net.http.SOAPHttpURLConnection)  
at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:37)  
at org.jasig.cas.client.javafilter.util.CommonUtils.getResponseFromServer(CommonUtils.java:392)  
  
可以看到两类错误上面的堆栈都是一样，所以第一个问题就是当时网络出了问题，导致连接一直hang住。