ORA-00603 ORA-27504 ORA-27300 ORA-27301 ORA-27302

客官°小女子只卖身不卖艺 2021-09-30 02:20 469阅读 0赞

\--节点1 告警日志

Thu Jun 20 03:02:18 2019  
Thread 1 advanced to log sequence 2092 (LGWR switch)  
Current log\# 11 seq\# 2092 mem\# 0: +DATA/test/onlinelog/redo11\_1.log  
Current log\# 11 seq\# 2092 mem\# 1: +DATA/test/onlinelog/redo11\_2.log  
Thu Jun 20 03:02:18 2019  
LNS: Standby redo logfile selected for thread 1 sequence 2092 for destination LOG\_ARCHIVE\_DEST\_2  
Thu Jun 20 03:02:20 2019  
Archived Log entry 5703 added for thread 1 sequence 2091 ID 0x9b8db1ed dest 1:  
Thu Jun 20 03:03:53 2019  
skgxpvfynet: mtype: 61 process 316751 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)  
Errors in file /u01/app/oracle/diag/rdbms/test/test1/trace/test1\_ora\_316751.trc (incident=560025):  
**ORA-00603: ORACLE server session terminated by fatal error**  
**ORA-27504: IPC error creating OSD context**  
**ORA-27300: OS system dependent operation:sendmsg failed with status: 105**  
**ORA-27301: OS failure message: No buffer space available**  
**ORA-27302: failure occurred at: sskgxpsnd2**  
Incident details in: /u01/app/oracle/diag/rdbms/test/test1/incident/incdir\_560025/test1\_ora\_316751\_i560025.trc  
Thu Jun 20 03:03:54 2019  
Dumping diagnostic data in directory=\[cdmp\_20190620030354\], requested by (instance=1, osid=316751), summary=\[incident=560025\].  
opiodr aborting process unknown ospid (316751) as a result of ORA-603  
Thu Jun 20 03:04:03 2019  
Sweep \[inc\]\[560025\]: completed  
Sweep \[inc2\]\[560025\]: completed  
Thu Jun 20 03:06:15 2019  
Thread 1 advanced to log sequence 2093 (LGWR switch)  
Current log\# 5 seq\# 2093 mem\# 0: +DATA/test/onlinelog/redo05\_1.log  
Current log\# 5 seq\# 2093 mem\# 1: +DATA/test/onlinelog/redo05\_2.log  
Thu Jun 20 03:06:16 2019

\---trc 文件--trc文件，初步看应该是服务器 网卡mtu值相关问题  
\*\*\* 2019-06-20 03:03:53.395  
\*\*\* CLIENT ID:() 2019-06-20 03:03:53.395  
\*\*\* SERVICE NAME:() 2019-06-20 03:03:53.395  
\*\*\* MODULE NAME:() 2019-06-20 03:03:53.395  
\*\*\* ACTION NAME:() 2019-06-20 03:03:53.395  
  
SKGXP:\[7f5ab0ef57c0.0\]\{0\}: SKGXPVFYNET: Socket self-test could not verify successful transmission of 32768 bytes (mtype 61).  
SKGXP:\[7f5ab0ef57c0.1\]\{0\}: The network is required to support UDP protocol sends of this size. Socket is bound to 169.254.16.188.  
SKGXP:\[7f5ab0ef57c0.2\]\{0\}: phase 'send', 0 tries, 100 loops, 32354 ms (last)  
struct ksxpp \* ksxppg\_ \[0xc11fde0, 0x7f5aadaf5588) = 0x7f5aadaf5580  
Dump of memory from 0x00007F5AADAF5580 to 0x00007F5AADAF6AB0

\---数据库状态  
select INST\_ID,INSTANCE\_NAME,STARTUP\_TIME from gv$instance;

\--添加到CRT  
set lines 1000\\r alter session set nls\_date\_format='yyyymmdd hh24:mi:ss';\\r set time on \\r select INST\_ID,INSTANCE\_NAME,STARTUP\_TIME from gv$instance;

09:42:29 SYS@test1(test1)> set lines 1000  
09:43:30 SYS@test1(test1)> alter session set nls\_date\_format='yyyymmdd hh24:mi:ss';

Session altered.

09:43:31 SYS@test1(test1)> set time on  
09:43:31 SYS@test1(test1)> select INST\_ID,INSTANCE\_NAME,STARTUP\_TIME from gv$instance;

INST\_ID INSTANCE\_NAME STARTUP\_TIME  
\---------- ---------------- -----------------  
1 test1 20190509 15:30:08  
2 test2 20190507 21:37:04

\---节点2 日志  
Thu Jun 20 03:03:54 2019  
Dumping diagnostic data in directory=\[cdmp\_20190620030354\], requested by (instance=1, osid=316751), summary=\[incident=560025\].  
Thu Jun 20 03:10:10 2019  
Thread 2 advanced to log sequence 912 (LGWR switch)  
Current log\# 6 seq\# 912 mem\# 0: +DATA/test/onlinelog/redo06\_1.log  
Current log\# 6 seq\# 912 mem\# 1: +DATA/test/onlinelog/redo06\_2.log

\---参考官方文档

**2041723.1**

CAUSE  
This happens due to less space available for network buffer reservation.

SOLUTION  
1. On servers with High Physical Memory, the parameter vm.min\_free\_kbytes should be set in the order of 0.4% of total Physical Memory. This helps in keeping a larger range of defragmented memory pages available for network buffers reducing the probability of a low-buffer-space conditions.

\*\*\* For example, on a server which is having 256GB RAM, the parameter vm.min\_free\_kbytes should be set to 1073742 \*\*\*

On NUMA Enabled Systems, the value of vm.min\_free\_kbytes should be multiplied by the number of NUMA nodes since the value is to be split across all the nodes.

On NUMA Enabled Systems, the value of vm.min\_free\_kbytes = n \* 0.4% of total Physical Memory. Here 'n' is the number of NUMA nodes.

2. Additionally, the MTU value should be modified as below

\#ifconfig lo mtu 16436

To make the change persistent over reboot add the following line in the file /etc/sysconfig/network-scripts/ifcfg-lo :

MTU=16436  
Save the file and restart the network service to load the changes

\#service network restart

Note : While making the changes in CRS nodes, if network is restarted while CRS is up, it can hung CRS. So cluster services should be stopped prior to the network restart.

\---野鸡博客  
http://ju.outofmemory.cn/entry/76102

\---实际操作，两个节点执行

\[root@test1 bin\]\# ifconfig lo mtu 16436  
\[root@test1 bin\]\#

\[root@test1 bin\]\# vi /etc/sysconfig/network-scripts/ifcfg-lo  
DEVICE=lo  
IPADDR=127.0.0.1  
NETMASK=255.0.0.0  
NETWORK=127.0.0.0  
\# If you're having problems with gated making 127.0.0.0/8 a martian,  
\# you can change this to something else (255.255.255.255, for example)  
BROADCAST=127.255.255.255  
ONBOOT=yes  
NAME=loopback  
MTU=16436

\---重启网络服务  
\# systemctl stop network  
\# systemctl start network

二、设定 vm.min\_free\_kbytes 参数为物理内存的0.4%

For example, on a server which is having 256GB RAM, the parameter vm.min\_free\_kbytes should be set to 1073742 \*\*\*

\---此次修改主机内存 512G ，所以 这个值是 1073742\*2=2147484

vm.min\_free\_kbytes=2147484

\[oracle@test1 ~\]$ cat /proc/sys/vm/min\_free\_kbytes  
65536  
\[oracle@test1 ~\]$

调整MIN\_FREE\_KBYTES的目的是保持物理内存有足够的空闲空间，防止突发性的换页。

**vi /etc/sysctl.conf**  
**vm.min\_free\_kbytes=2147484**

\--使生效  
sysctl -p

转载于:https://www.cnblogs.com/ss-33/p/11057478.html