Oracle 11gR2 RAC orarootagent Memory Leak Causes Swap Space Exhaustion
Reference link (blog.itpub.net)
11.2.0.1 RAC 2nodes ASM AIX6.1 orarootagent.bin swap memory memory leak
oracle 11.2.0.1aix 6.1
rac (2nodes) + asm
Symptom: RAC OS is unresponsive; checking the alert logs for CRS, ASM, and DB shows no obvious anomalies.
Signs: The orarootagent.bin process consumed nearly 40GB of swap memory, almost exhausting the entire swap partition.
Metalink reference
ORAAGENT or ORAROOTAGENT High Resource (CPU, Memory etc) Usage [ID 1062676.1]
Bug 9855476 - aix:cssd/ohasd may coredump, or orarootagent/gipcd may leak memory [ID 9855476.8]
The following describes the cause of the memory leak. Observation shows that on each node, the orarootagent.bin process increases by approximately 220MB daily.
Description
gipcd.bin or orarootagent.bin process may leak process memory.
Huge memory consumption by these processes may cause other processes
such as cssd.bin or ohasd.bin to be unable to allocate memory and crash.
Rediscovery Notes
If the following are true, you might be hitting this problem.
- Platform is IBM AIX on POWER Systems (64-bit)
- Process memory is steadily increasing in oraagent.bin process
or gipcd.bin process. There is a report that oraagent.bin
process increased about 4KB to 8KB memory every 15 seconds.
Solution: Apply patches to upgrade to >11.2.0.2, or periodically kill the orarootagent.bin process.
It's recommended to apply latest patch set or PSU. But if patches are unavailable or can not be applied, the workaround is to kill agent process during off peak hour. There should be no impact as agent will be respawn automatically.
To kill agent process with pid 7242: kill -9 7242
Locate the orarootagent.bin process and kill it. The process will automatically restart without affecting RAC operation.
ps -ef|grep orarootagentkill -9 2163012
ps -ef|grep 12845336[@more@]