AIX HACMP Tuning Case Study
2018-02-28 07:25 Listen to full text
System Status
1. The host rebooted; file systems and hardware were checked and found normal.
2. System load was high; CPU usage reached 100% and I/O load was very heavy.
I. Review Error Log Reports
6D19271E 0827152608 I O topsvcs Topology Services daemon stopped
AA8AB241 0827152608 T O OPERATOR OPERATOR NOTIFICATION
BA431EB7 0827152608 P S SRC SOFTWARE PROGRAM ERROR
BC3BE5A3 0827152608 P S SRC SOFTWARE PROGRAM ERROR
12081DC6 0827152608 P S haemd SOFTWARE PROGRAM ERROR
BC3BE5A3 0827152608 P S SRC SOFTWARE PROGRAM ERROR
AA8AB241 0827152608 T O clstrmgrES OPERATOR NOTIFICATION
9DEC29E1 0827152608 P O grpsvcs Group Services daemon exit to merge domain
Detailed review of error messages revealed that the system crashed at 15:26 due to HACMP issues.
II. Problem Cause
Nodes in the cluster experienced extreme performance issues, such as massive I/O transfers, excessive error records, and insufficient memory. This caused the cluster manager (clstrmgrES) to not receive sufficient CPU processing time, leading to the Deadman Switch being reset within its allocated time window. This issue occurs when an application runs with higher priority than the cluster manager.
III. Resolution Steps
The term "Deadman Switch" refers to a kernel extension component that causes system crashes and dumps when the switch fails to be reset promptly under specific cluster conditions. If the Deadman Switch exceeds a certain time limit, it will trigger a crash on a suspended node. This process causes other nodes in the cluster to take over resources from the suspended node. To resolve this issue, several related performance problems must be addressed:
Tune performance optimization parameters in HACMP:
※ Adjust system I/O pacing
※ Increase the frequency of information synchronization (syncd)
※ Increase the amount of memory used by the communication subsystem
※ Change the error detection rate
1. Adjust system I/O pacing steps
Use I/O pacing to adjust the system so that resource allocation is more reasonable during heavy write operations. Activating I/O Pacing for HACMP clusters is necessary, especially when there may be a large number of disk block write operations in the cluster.
Modify I/O Pacing settings according to the following steps:
# smitty hacmpCluster Configuration
Advanced Performance Tuning Parameters
Change/Show I/O Pacing
Modify high/low threshold values in hacmp:
HIGH water mark for pending write I/Os per file 33
LOW water mark for pending write I/Os per file 24
2. Increase syncd execution frequency
Increase the syncd execution frequency, changing the default interval from once every 60 seconds to once every 10 seconds. This forces an increase in I/O refresh rates and reduces the likelihood of triggering the Deadman Switch due to heavy I/O traffic.
Modify syncd execution frequency settings according to the following steps:
# smitty hacmpCluster Configuration
Advanced Performance Tuning Parameters
Change/Show syncd frequency
Modify the 'syncd frequency in seconds' field; the recommended value is 10 seconds. Available values range from 0 to 32767.
3. Increase available memory for the communication subsystem
If running the command # netstat –m reveals that mbuf requests are denied, or if running # errpt shows LOW_MBUFS errors, then the network parameter 'thewall' value should be increased. The default value for Thewall is 25% of system real memory. It can be increased to 50% of system real memory.
Modify thewall value settings according to the following steps:
# vi /etc/rc.netAdd the following to the end of this file:
no -o thewall= 1024
4. Modify error detection rate
If activating I/O Pacing or increasing Syncd execution frequency does not resolve the issue of the Deadman Switch failing to reset, then modify the error detection rate and set it to Slow. This extends the time required before a suspended node invokes the Deadman Switch, as well as the time required for a takeover node to detect the node failure and acquire the suspended node's resources.
Note:
Before completing the above steps, I/O Pacing must be activated first. This is because modifying this setting adjusts the volume of I/O data transmission.
IV. Handling Results
After optimizing HACMP parameters, the system has been running normally.