Analysis of Causes for AIX System CPU Usage at 100%

 Feb 8, 2017 07:26 Listen to full text

Yesterday, we shared 'Fault Case Analysis: AIX Network Performance Issues' (click to review), which is a common case in daily AIX system operations. In daily AIX system operations, we often encounter various faults: What if the system has problems again? How should we analyze a machine dump when it crashes? After fault handling is completed, are there any experiences to share?

Although there are many issues, they all follow the same principles. These faults can basically be categorized into abnormal crashes, performance issues, parameter settings, and system bugs. In this era of standardized approaches, each type of fault has a fixed analysis method.

The author of this case—Cui Zengshun—has summarized representative classic fault cases handled over recent years, covering fault analysis approaches in memory, network, CPU, LVM, and dump analysis directions, sharing a series of fault case analyses and resolution suggestions. Including:

Case 1: AIX Network Performance Issues

Case 2: High CPU Usage in AIX Operating System

Case 3: AIX Operating System Crash (Related to Memory Parameter kernel_heap_psize)

Case 4: AIX Operating System Crash (Related to Memory Parameter numperm_global)

Case 5: AIX Operating System Crash (Caused by netstat BUG)

Case 6: High Utilization Rate of Reserved Memory on P780 (Cannot be allocated for partition use)

Case 7: Oracle Database Restart (Related to Memory Parameter numperm_global)

Case 8: varyonvg Error (LVM Related)

Case 9: AIX Operating System Hang

The author's sharing aims to encourage further contributions. We hope everyone will share more and discuss faults encountered during daily operations and their solutions together at the online event 'AIX System Daily Operations Fault Analysis and Handling Online Technical Exchange' on February 17. Welcome to participate.

Time: Feb 17, 9:00-18:00

Participation Method: Click 'Read Original Text' to register online

(There are rewards for participating in the event~)

Today's sharing—

Fault Case Analysis: High CPU Usage in AIX Operating System

Fault Description

CPU at 100%, need to analyze the cause.

Fault Analysis

By checking relevant system data, it was found that the cause of high CPU usage in the system is a large number of topasrec processes.

root 24117384 1 0 May 14 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=11:27:14,May14,2013 -o /etc/perf/
root 24314016 1 0 May 06 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=01:52:54,May06,2013 -o /etc/perf/
root 24444950 1 0 May 10 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=07:29:35,May10,2013 -o /etc/perf/
root 24510670 1 0 May 09 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=03:00:46,May09,2013 -o /etc/perf/
root 24707102 1 0 May 07 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=13:11:10,May07,2013 -o /etc/perf/
root 24772628 1 0 May 07 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=21:46:45,May07,2013 -o /etc/perf/
root 24838176 1 0 May 12 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=10:27:28,May12,2013 -o /etc/perf/
root 24903888 1 0 May 06 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=06:47:43,May06,2013 -o /etc/perf/
root 24969418 1 0 May 06 - 0:00 /usr/bin/topasrec -L -O type=azizo -ystart_time=19:29:37,May06,2013 -o /etc/perf/

View the process stack information:

0xd01407b0 malloc_y(0x80, 0x0, 0xf23800c8, 0xf24d2920, 0x0, 0x0, 0x0, 0x2ff191c8) + 0x590
0xd012a908 malloc_common@AF103_86(??) + 0x28
0xd08a5e1c uprv_malloc@AF10_7(??) + 0x5c
0xd08cd2e4 ures_openDirect(??, ??, ??) + 0x44
0xd06afe48 icu::openOlsonResource(consticu::UnicodeString&,UResourceBundle&,UErrorCode&)(??, ??, ??) + 0x28
0xd06afa98 icu::TimeZone::createSystemTimeZone(consticu::UnicodeString&)(??) + 0x58
0xd06af954 icu::TimeZone::createTimeZone(consticu::UnicodeString&)(??) + 0x14
0xd07eb764 _createTimeZone(constwchar_t*,int,UErrorCode*)(??, ??, ??) + 0x164
0xd07ebb48 ucal_open(??, ??, ??, ??, ??) + 0x108
0xd01bfe4c localtime_tz_ost_r(??, ??) + 0xac
0xd01c03d4 localtime_tz_r(??, ??, ??) + 0x394
0xd01c194c ctime(??) + 0xc
0x100077e4 write_to_log(??) + 0x84
0x1000bf9c must_exit(??) + 0x3c
0xd01407b0 malloc_y(0x80, 0x0, 0xf23800c8, 0xf24d2920, 0x0, 0x0, 0x0, 0x2ff1aa48) + 0x590
0xd012a908 malloc_common@AF103_86(??) + 0x28
0xd08a5e1c uprv_malloc@AF10_7(??) + 0x5c
0xd08cd2e4 ures_openDirect(??, ??, ??) + 0x44
0xd06afe48 icu::openOlsonResource(consticu::UnicodeString&,UResourceBundle&,UErrorCode&)(??, ??, ??) + 0x28
0xd06afa98 icu::TimeZone::createSystemTimeZone(consticu::UnicodeString&)(??) + 0x58
0xd06af954 icu::TimeZone::createTimeZone(consticu::UnicodeString&)(??) + 0x14
0xd07eb764 _createTimeZone(constwchar_t*,int,UErrorCode*)(??, ??, ??) + 0x164
0xd07ebb48 ucal_open(??, ??, ??, ??, ??) + 0x108
0xd01bfe4c localtime_tz_ost_r(??, ??) + 0xac
0xd01c03d4 localtime_tz_r(??, ??, ??) + 0x394
0xd01c194c ctime(??) + 0xc
0x100077e4 write_to_log(??) + 0x84
0x1000bf9c must_exit(??) + 0x3c
0x10014b88 get_stats(??) + 0x88
0xd01407b0 malloc_y(0x80, 0x0, 0xf23800c8, 0xf24d2920, 0x0, 0x0, 0x0, 0x2ff1c718) + 0x590
0xd012a908 malloc_common@AF103_86(??) + 0x28
0xd08a5e1c uprv_malloc@AF10_7(??) + 0x5c
0xd08cd2e4 ures_openDirect(??, ??, ??) + 0x44
0xd06afe48 icu::openOlsonResource(consticu::UnicodeString&,UResourceBundle&,UErrorCode&)(??, ??, ??) + 0x28
0xd06afa98 icu::TimeZone::createSystemTimeZone(consticu::UnicodeString&)(??) + 0x58
0xd06af954 icu::TimeZone::createTimeZone(consticu::UnicodeString&)(??) + 0x14
0xd07eb764 _createTimeZone(constwchar_t*,int,UErrorCode*)(??, ??, ??) + 0x164
0xd07ebb48 ucal_open(??, ??, ??, ??, ??) + 0x108
0xd01bfe4c localtime_tz_ost_r(??, ??) + 0xac
0xd01c03d4 localtime_tz_r(??, ??, ??) + 0x394
0xd01c194c ctime(??) + 0xc
0x100077e4 write_to_log(??) + 0x84
0x1000bf9c must_exit(??) + 0x3c

Subsequent Measures and Suggestions

Analysis shows this is caused by an AIX bug:

IV24599: TOPASREC AZIZO PROCESS HANGS BECAUSE OF SIGALRM IN-TERRUPT APPLIES TO AIX 6100-06

http://www-01.ibm.com/support/docview.wss?uid=isg1IV24599

Current AIX version is 6100-06-06; this patch requires upgrading the system to 6100-06-11.

Two measures can be taken to address this issue:

1. Upgrade the operating system version to 6100-06-11

2. Disable xmdaily to prevent automatic startup of topasrec

# vi /etc/inittab

Modify xmdaily: change '2:once' to '2:off'

# telinit q

Apply the new inittab file

# vi /etc/inetd.conf

Comment out the line corresponding to xmquery

# refresh –s inetd

Apply the new inetd.conf file