Analysis of AIX System Hang, HMC and SSH Login Failures
2017-04-07 08:08 Listen to full text
Fault Description
AIX operating system hangs; unable to log in via HMC or SSH.
System Environment
AIX 5300-11-07
Fault Analysis
Since the system was already hung and terminal access was unavailable, AIX forcedump data was collected on-site. The system has 116 GB of memory. The dump was collected successfully, and the system resumed normal operation after reboot.
The dump data shows a total of 36 logical CPUs, with processes running on each CPU as follows:
The dla command was used to check for deadlocks. Analysis revealed 220 deadlocks, detailed as follows:
Further analysis of the deadlock cause indicates that all deadlocks were waiting for thread TID A7233.
Investigation confirms that thread TID A7233 belongs to IBM.CSMAgent. Thus, 220 threads are waiting for IBM.CSMAgent.
Analysis reveals that IBM.CSMAgent runs on CPU0 and is waiting for the xmtopas thread, which also runs on CPU0. Since xmtopas has a priority of 18 and IBM.CSMAgent has a priority of 3A (hexadecimal), IBM.CSMAgent waits for the higher-priority xmtopas to complete before responding.
Further analysis shows xmtopas was malfunctioning at the time, exiting abnormally with signal 11 due to memory boundary violation. Consequently, xmtopas entered an infinite loop, blocking subsequent IBM.CSMAgent execution and pending other downstream processes, ultimately causing the system to hang.
Fault Recommendations
For this issue, since the xmtopas process is not currently used in the system (it is only invoked when executing topas -C to collect CPU information from other LPARs on a given LPAR), disabling the xmtopas service can mitigate the problem.
Author Column: http://www.aixchina.net/Column/detail/id/7
This Series of Articles:
Analysis and Resolution of varyonvg Errors (LVM-related)
Causes and Solutions for High Utilization of Reserved Memory (Unallocatable to Partitions) on P780
AIX System Crash Related to Memory Parameter numperm_global
Oracle Database Restart in Such Scenarios: Four Solutions Available
What Causes Automatic Startup After Abnormal AIX Shutdown?
AIX Operating System Crash (Related to Memory Parameter kernel_heap_psize)
AIX System CPU Usage at 100%: Root Cause Analysis
Case Study: AIX Network Performance Issues
Images localized into English for illustration; labels may vary by product version.