IBM 7133 Fault Location and Troubleshooting

Chapter 3: Fault Localization and Troubleshooting

Based on problems encountered in actual commercial systems, we have summarized the following common faults along with their localization methods and solutions.

3.1 Hardware Faults

Hardware faults are of many types and have varying impacts on the system. Here, hardware is classified into two categories based on the severity of the impact on the system: fatal hardware faults and functional-only hardware faults:

Hardware whose damage causes a fatal impact on the system (causing the machine to hang or fail to boot) includes:

Motherboard, CPU, I/O Cabinet (containing local disks, optical drives, PCI slots, etc.) or CEC Cabinet (containing CPU/MEMORY, etc.), cabling between the I/O Cabinet and CEC Cabinet, power modules, fans, local hard disks, memory damage, etc.

Note: I/O Cabinets and CEC Cabinets are typically found only in high-end midrange servers, such as the M80; low-end models combine them into a single unit.

Damage to these devices will prevent the system from completing self-tests, booting, and starting up. Error messages will appear on the LCD display. The cause of the error can be identified by cross-referencing the error code displayed on the LCD with the Service Guide. If such hardware damage occurs while the system is operational, the system will be suspended or crash.

Hardware whose damage has only a functional impact on the system (the machine will not crash and can start normally) includes:

Network interface cards, local hard disks with bad blocks, graphics cards, SSA cards, and other peripheral devices.

Damage to these devices affects only specific functions, such as network functionality, display functionality, or access to the disk array. In the case of local hard disks with bad blocks, it must be determined whether the bad blocks contain critical system files. If they do not contain critical system files, system functionality remains unaffected, but immediate replacement of the hard disk is still recommended.

Fault Localization and Troubleshooting:

The above hardware fault information can be utilized using:

Error codes on the LCD display or:

errpt –dH  View the results

Determine which hardware component has failed based on the error code. For commercial systems, which are dual-system configurations, if the damaged machine is the primary server, switch this server to become the standby server, then repair the faulty machine and restore the system.

3.2 Disk Array Faults

Faults caused by the disk array are currently the most frequent and most severe. According to incomplete statistics, these faults account for over 70% of total faults. Specifically, factors that may cause disk array faults include:

Disk array hard disks, 7133 cabinets, SSA cards on the host, SSA cables connecting the 7133 to the host, disk drive positions and SSA cable wiring methods, voltage used by the disk cabinet and surrounding magnetic fields, and microcode for the disk array/disks/SSA cards.

All of these can cause anomalies in the 7133.

Issues with the 7133 disk array are the most complex. Generally, there are physical damage causes and environmental causes; these are the primary causes. Secondary causes include incorrect wiring, improper disk insertion positions, and failure to check system alarms in time, which can cause system interruptions. Based on our experience, regardless of which hardware fault causes the 7133 fault, the system will generate an alarm. If the problem can be found and measures taken promptly, the occurrence of faults can generally be prevented.

Fault Localization:

7133 hardware faults can also be diagnosed using:

errpt –dH  View the results

Accompanying error codes include:

B4C00618 0115140004 P H ssa0 RESOURCE UNAVAILABLE

FE9E9357 0401082304 P H ssa0 DISK OPERATION ERROR

FE9E9357 1205000803 P H pdisk3 DISK OPERATION ERROR

03913B94 1122031103 U H LVDD HARDWARE DISK BLOCK RELOCATION ACHIEVED

613E5F38 1121125103 P H LVDD I/O ERROR DETECTED BY LVM

625E6B9A 0401090004 P H ssa0 ADAPTER DETECTED OPEN SERIAL LINK

26CA120B 0206081104 P H ssa0 CACHE STORAGE CARD

All error codes indicate an anomaly in the 7133. Red text indicates a definite hardware fault requiring immediate inspection and action; otherwise, the disk array will soon become inaccessible. Regarding blue text:

625E6B9A 0401090004 P H ssa0 ADAPTER DETECTED OPEN SERIAL LINK

It indicates that the SSA loop is open. An open loop not only degrades I/O performance but also increases risk; if the other loop also encounters issues, the disk array will become inaccessible.

There are generally two scenarios for an open loop: 1) If 625E6B9A errors occur frequently, such as several times a day, it is highly likely that a hardware fault exists. Although this will not cause disk array access failure, the cause must be identified and resolved immediately. Refer to the description below for troubleshooting methods. 2) If 625E6B9A an error occurs only occasionally, the situation should be handled on a case-by-case basis. It could be a false alarm due to heavy read/write activity. Proceed with troubleshooting using the method described below. If no specific cause is found, continue monitoring.

26CA120B 0206081104 P H ssa0 CACHE STORAGE CARD

This error generally occurs when: the SSA card has write cache enabled and FastWrite is turned on, and the rechargeable nickel-cadmium battery used for write cache power supply on the SSA card has reached or approached its safe lifespan. This type of error will affect I/O write performance. Furthermore, since the primary and standby servers need to synchronize the write cache on the SSA cards when FastWrite is enabled, it may even impact primary-standby synchronization. Refer to the following section for specific solutions.

Troubleshooting:

For issues indicated by red text errors, further diagnosis can generally be performed using the diag command:

#diag -> Task Selection -> SSA Service Aids -> Link Verification 

Check whether any disk drives in the loop show ??? status or if any hard disk status is not 'good'.

Or use:

#smitty ssaraid -> List All Defined SSA RAID Arrays

to check if the RAID disk status is 'Good'. If the status is 'degrade' or any other state, it indicates a problem with the RAID disk.

At this point, performing individual disk certification is not recommended. Instead, immediately notify IBM to prepare identical model and size hard disks (at least two) to be sent onsite for further diagnosis and bad disk replacement.

625E6B9A 0401090004 P H ssa0 ADAPTER DETECTED OPEN SERIAL LINK

troubleshooting method:

1. Ensure all empty slots in the 7133 are populated with dummy drives. (Dummy drive: A placeholder shell, essentially a plastic model representing an SSA hard disk, used when the 16 slots of the disk array are not fully occupied by SSA hard disks.)

2. Check the Loop status: diag --> Task Selection --> SSA Service Aids --> Link Verification. Under a normal Adapter Port, the two columns of numbers should be continuous and uninterrupted, and the Status should be 'good'. If the Physical column shows ??????? symbols or the Status is not 'good', it indicates an existing disk or link fault. Immediate action is required to perform further checks to determine if a disk needs replacement. The method to check a single disk for issues is as follows:

diag--->Task Selection-->SSA Service Aids-->Certify Disk

Select the suspected faulty disk for inspection.

3. When a hardware fault exists in the 7133, it can be observed via the status lights:

When a single hard disk fails or is not in use, the hard disk status light on its panel will be off

The yellow array status light will be on

Or the indicator light for the port connected to the SSA line will also turn off

If no issues are found using the above three methods and the system still reports an Open Serial Link error, continue monitoring.

26CA120B 0206081104 P H ssa0 CACHE STORAGE CARD

Handling method:

Background Information:

The SSA card configured for the 7133 magnetic array connected to IBM small computers generally includes a rechargeable battery. This battery protects information in the fast write cache on the SSA card during sudden power failures. The safe lifespan of this battery is typically 22,000 hours, approximately two and a half years. Therefore, when the fast write mode is active, this battery usually needs to be replaced after about two and a half years.

Problem Manifestation:

Battery protection on the SSA card is implemented via a counter on the card. For every hour of operation, the counter increments by one. When the count approaches or exceeds 22,000, the system generates hardware alarm 26CA120B:

26CA120B 0206120904 P H ssa1 CACHE STORAGE CARD

You can check the status of the SSA card using the following command: (append the logical device name of the card after -a; here it is assumed to be ssa0)

ssa_fw_status -a ssa0 -p (Check how long the battery has been operating, in hours)

ssa_fw_status -a ssa0 -l (Check the safe working life of the battery, in hours)

ssa_fw_status -a ssa0 -c (Check whether the fast write function on the SSA card is activated)

Processing Steps: (Listed in order of preference; select only one method)

1) Replace the battery

During system idle time, replace the batteries on the primary and standby SSA cards using the following procedure:

Stop the standby server → Replace the standby server's SSA card battery → Start the standby server (dual-server service) → Perform primary/standby switchover →

Stop the original primary server → Replace the original primary server's SSA card battery → Start the original primary server (dual-server service)

2) If the system generates a 26CA120B battery alarm, use ssa_fw_status -a ssa0 -c to check whether Fast Write on the active SSA card (typically ssa0) is in an inactive state. If so, execute the following commands as root:

ssa_format -l ssa0 -b

errclear 0

/usr/lib/errstop

/usr/lib/errdemon

Battery replacement can be deferred until a new battery is available, but disk array read/write performance may be affected during this period.

3) If the battery cannot be replaced in the short term and the Fast Write function on the primary card of the host remains Active, manually disable the fast write function is recommended.

1) First, stop the dual-server cluster

2) On Node 1, modify the hdisk attribute:

smitty dev->ssa disks->ssa logical disks->change /show characters of ... -> [choose hdisk2] -> fast write [no]

3) On Node 1, activate the volume group

varyonvg zxinvg

4) On Node 1, deactivate the volume group

varyoffvg zxinvg

5) Execute on Node 2:

smitty dev->ssa disks->ssa logical disks->change /show characters of ... -> [choose hdisk2] -> fast write [no]

Check whether fast write has been changed to no (if step 2 was performed on Node 1, no further modification is needed on Node 2)

6) Execute on Node 2:

rmdev -dl hdisk2

	   cfgmgr -v

7) Execute on Node 2:

lspv (Verify whether hdisk2 has been detected)

8) Then execute:

varyonvg zxinvg

varyoffvg zxinvg

9) Restart the dual-server cluster

For errors occurring:

B4C00618 0115140004 P H ssa0 RESOURCE UNAVAILABLE

Without accompanying other red-highlighted errors, troubleshoot using the following methods:

It is possible that 'enable user of hot spare' is enabled in hdisk, but no hot spare disk is actually configured.

This can be resolved as follows:

Check the RAID status:

#smitty ssaraid                                                           

list all defined SSA RAID arrays : all are in status good (ssa0 )

Check whether a hot spare is configured:

list/identify SSA Physical disks-->List hot spares : none

Modify the attributes of each hdisk configured for each SSA card:

: set "enable use of hot spare" to "no"

Then execute:

# /usr/lib/errstop

# cp /var/adm/ras/errlog   /var/adm/ras/errlog.bak

#/usr/lib/errdemon

#errclear 0

For the connection method of physical hard disks, refer to the following explanation: (If the old site has not configured a hot spare, calculate the wiring method based on the number of disks excluding hot spares)

Describe the following situations separately:

1. When the number of data disks is less than or equal to 6, it is recommended to configure one hot spare.
2. When the number of data disks is greater than or equal to 8 and less than or equal to 10, it is recommended to configure two hot spares.
3. When the number of data disks is greater than or equal to 12 and less than or equal to 14, it is recommended to configure two hot spares.
4. Wiring and configuration rules for configuring one hot spare when the number of data disks is less than or equal to 6:
5. Host A1 and A2 connect to positions 1 and 8 of the disk array; standby machine A1 and A2 connect to positions 4 and 5 of the disk array.
6. Ensure that hard disks are inserted in the front row at positions 1, 4, 5, and 8. Other disks can be inserted into any available slots. Note that data disks on both sides of 1-4 and 5-8 must be inserted in an axially symmetric relationship (e.g., 1 and 8 are axially symmetric, 4 and 5 are axially symmetric).
7. Any position without a hard disk must have a dummy disk installed.
8. When performing RAID 0+1, considering performance issues, please mirror the disks at positions 1-4 and 5-8 in an axially symmetric manner one-to-one. An example is as follows:

IBM's sorting order for pdisks differs from their actual physical positions. Therefore, you must first use the command `lsdev -C | grep pdisk` to identify and record the correspondence, such as:

pdisk0 Available 11-08-1641-01-P SSA160 Physical Disk Drive

pdisk1 Available 11-08-1641-05-P SSA160 Physical Disk Drive

pdisk2 Available 11-08-1641-04-P SSA160 Physical Disk Drive

pdisk3 Available 11-08-1641-08-P SSA160 Physical Disk Drive

Look for the character string '-01-' in the middle of the third column, which indicates the physical hard disk position. The above correspondence represents

The physical slots corresponding to pdisk0-pdisk3 are positions 1, 5, 4, and 8 in the front row, respectively. Therefore, when using `smitty ssaraid` to perform RAID 0+1, the disks at positions 1 and 8 (pdisk0/pdisk3) must be mirrored, and the disks at positions 4 and 5 (pdisk2/pdisk1) must be mirrored. Thus, when selecting Primary Disks/Secondary Disks, do the following:

smitty ssaraid-> Add an SSA RAID Array -> ...

Primary Disks [Select: pdisk0 pdisk2] -- Disks in slots 1 and 4 as primary disks

Secondary Disks [Select: pdisk3 pdisk1] -- Disks in slots 8 and 5 as secondary disks

This requirement is only for performance considerations and does not add other features.

1. Hot spare disks can be inserted into any empty slot among the first eight positions in the front row. There are no special position requirements. The creation method is as follows: Use `lsdev -C | grep pdisk` to find its pdisk number, e.g., pdisk5, and then create it as a hot spare using the following method.

smitty ssaraid -> Change Use of Multiple SSA Physical Disks -> Select the ssa card (the ssa card connected to the disk array, default is ssa0)

-> Select the pdisk to be used as a hot spare, e.g., pdisk6 -> New Use -> Select 'hot spare' and press Enter to create.

1. Wiring and configuration rules for configuring two hot spares when the number of data disks is greater than or equal to 8 and less than or equal to 10:
2. Host A1 and A2 connect to positions 1 and 12 of the disk array; standby machine A1 and A2 connect to positions 8 and 9 of the disk array.
3. Ensure that positions 1 through 8 in the front row of the disk array are fully populated with data disks. Data disks and hot spare positions in slots 9-12 can be placed arbitrarily.
4. Any position without a hard disk must have a dummy disk installed.
5. When performing RAID 0+1, considering performance issues, please mirror the disks at positions 1-4 and 5-8 in an axially symmetric manner one-to-one.

If there are data disks in positions 9-12, mirror them correspondingly. Use the remaining two disks as hot spares. An example is as follows:

IBM's sorting order for pdisks differs from their actual physical positions. Therefore, you must first use the command `lsdev -C | grep pdisk` to identify and record the correspondence, such as:

pdisk0 Available 11-08-1641-01-P SSA160 Physical Disk Drive

pdisk1 Available 11-08-1641-05-P SSA160 Physical Disk Drive

pdisk2 Available 11-08-1641-04-P SSA160 Physical Disk Drive

pdisk3 Available 11-08-1641-08-P SSA160 Physical Disk Drive

pdisk4 Available 11-08-1641-02-P SSA160 Physical Disk Drive

pdisk5 Available 11-08-1641-03-P SSA160 Physical Disk Drive

pdisk6 Available 11-08-1641-07-P SSA160 Physical Disk Drive

pdisk7 Available 11-08-1641-06-P SSA160 Physical Disk Drive

pdisk8 Available 11-08-1641-09-P SSA160 Physical Disk Drive

pdisk9 Available 11-08-1641-11-P SSA160 Physical Disk Drive

pdisk10 Available 11-08-1641-10-P SSA160 Physical Disk Drive

pdisk11 Available 11-08-1641-12-P SSA160 Physical Disk Drive

Look for the character string '-01-' in the middle of the third column, which indicates the physical hard disk position. The above correspondence represents

The physical slots corresponding to pdisk0-pdisk11 are positions 1, 5, 4, 8, 2, 3, 7, 6, 9, 11, 10, and 12, respectively. Therefore, when using `smitty ssaraid` to perform RAID 0+1, the first eight disks should be mirrored according to the axially symmetric correspondence of 1-4 and 5-8. If positions 9 (pdisk8) and 12 (pdisk11) are selected for mirroring within 9-12

Thus, when selecting Primary Disks/Secondary Disks, do the following:

smitty ssaraid-> Add an SSA RAID Array -> ...

Primary Disks [Select: pdisk0 pdisk4 pdisk5 pdisk2 pdisk8] Physical slots 1-4, 9

Secondary Disks [Select: pdisk3 pdisk6 pdisk7 pdisk1 pdisk11] Physical slots 8-5, 12

This requirement is only for performance considerations and does not add other features.

1. The remaining physical positions 10 (pdisk10) and 11 (pdisk9) should be configured as hot spares, using the same method as step e).
2. Wiring and configuration rules for configuring two hot spares when the number of data disks is greater than or equal to 12 and less than or equal to 14:
3. Host A1 and A2 connect to positions 1 and 16 of the disk array; standby machine A1 and A2 connect to positions 8 and 9 of the disk array.
4. Leave positions 3 and 14 open for hot spares. If there are 12 data disks, leave positions 2 and 15 open for dummy disks, and insert data disks in all other positions. If there are 14 data disks, insert data disks in all remaining 14 positions.
5. When performing RAID 0+1, considering performance issues, please mirror the disks at positions 1-8 and 9-16 in an axially symmetric manner one-to-one. Hot spares also satisfy the axial symmetry relationship. An example is as follows:

IBM's sorting order for pdisks differs from their actual physical positions. Therefore, you must first use the command `lsdev -C | grep pdisk` to identify and record the correspondence, such as:

pdisk0 Available 11-08-1641-01-P SSA160 Physical Disk Drive

pdisk1 Available 11-08-1641-05-P SSA160 Physical Disk Drive

pdisk2 Available 11-08-1641-04-P SSA160 Physical Disk Drive

pdisk3 Available 11-08-1641-08-P SSA160 Physical Disk Drive

pdisk4 Available 11-08-1641-02-P SSA160 Physical Disk Drive

pdisk5 Available 11-08-1641-03-P SSA160 Physical Disk Drive

pdisk6 Available 11-08-1641-07-P SSA160 Physical Disk Drive

pdisk7 Available 11-08-1641-06-P SSA160 Physical Disk Drive

pdisk8 Available 11-08-1641-09-P SSA160 Physical Disk Drive

pdisk9 Available 11-08-1641-11-P SSA160 Physical Disk Drive

pdisk10 Available 11-08-1641-10-P SSA160 Physical Disk Drive

pdisk11 Available 11-08-1641-12-P SSA160 Physical Disk Drive

pdisk12 Available 11-08-1641-16-P SSA160 Physical Disk Drive

pdisk13 Available 11-08-1641-13-P SSA160 Physical Disk Drive

pdisk14 Available 11-08-1641-15-P SSA160 Physical Disk Drive

pdisk15 Available 11-08-1641-14-P SSA160 Physical Disk Drive

Look for the character string '-01-' in the middle of the third column, which indicates the physical hard disk position. The above correspondence represents

The physical slots corresponding to pdisk0-pdisk15 are positions 1, 5, 4, 8, 2, 3, 7, 6, 9, 11, 10, 12, 16, 13, 15, and 14, respectively. Therefore, when using `smitty ssaraid` to perform RAID 0+1, mirrors are created using the axially symmetric method for 1-8 and 9-16. Thus, when selecting Primary Disks/Secondary Disks, do the following:

smitty ssaraid-> Add an SSA RAID Array -> ...

Primary Disks [Select: pdisk0 pdisk4 pdisk2 pdisk1 pdisk7 pdisk6 pdisk3]

Physical slots 1-8, excluding slot 3

Secondary Disks [Select: pdisk12 pdisk14 pdisk13 pdisk11 pdisk9 pdisk10 pdisk8]

Physical slots 16-9, excluding slot 14

This requirement is for performance considerations only and does not add other features.

The remaining physical locations 3 (pdisk5) / 14 (pdisk15) are configured as hot spares using the same method as step e)

3.3 Network Faults

Since our applications have a strong dependency on the network, any complete network outage or transient interruption will have a significant impact on the system. Network faults generally fall into two categories: hardware faults (such as NIC failures, switch failures, or router failures) and software faults (such as IP packet attacks or network congestion within the network).

Hardware Faults:

1. Network Interface Card (NIC)

For NIC faults, since commercial systems typically use an IBM HA dual-system architecture, and each machine is configured with at least two NICs, if a single NIC or network cable fails, the HA software will take measures to switch the Service IP.

NIC Fault Localization Method: Use errpt -dH to view errors indicating NIC service interruption, then use diag to diagnose the NIC.

NIC Fault Troubleshooting Method: If the diagnosis indicates a NIC problem, shut down the system and replace it. (If it is a host, manually switch to the standby node first before proceeding.)

2. Switch

Our networks generally adopt a dual-network dual-plane structure, so if a switch in one network plane fails, network services will not be interrupted. However, the following point is of great importance: The direct connection between the primary and standby switches must remain operational; otherwise, if a NIC failover occurs on the IBM server host or SIU host, it will cause a link break between the IBM server host and the SIU host, resulting in a complete business blockage.

There is another scenario: in some places, for network security, different devices connected to the switch are assigned to different VLANs, while the connection between the primary and standby switches is changed to use two ports and placed in a channel group. A situation encountered was:

1) When the main NIC of the IBM server migrated from default VLAN 0 to VLAN 6, the network experienced a transient interruption of 15 to 30 seconds.

2) The channel group between the primary and standby switches was not functioning correctly, and the link was down.

When scenario 1) occurred, the IBM dual-system performed a primary/standby NIC switchover, but due to issue 2), the IBM host could not establish a link with the SIU.

Problem Troubleshooting and Recommendations:

1. It is recommended that during initial deployment, perform a fault test where one plane of the dual-network dual-plane switch loses power.
2. It is essential to perform network inter-communication tests between devices connected to the primary and standby switches to ensure connectivity.
3. Avoid making configuration changes on the switch during the day. If changes are necessary, they should be performed at night with a strict plan.

Software Faults:

1. Network Congestion

Since the system operates in a closed network, the likelihood of network congestion is relatively low. However, if the network topology is complex, this may still occur. On the host, symptoms include severe packet loss when pinging the host, intermittent connectivity between the host and SIU, packet loss, and abnormal device functionality.

Problem Troubleshooting and Recommendations:

1. Try to isolate NT/2000 machines from the network.
2. If the situation does not improve, it is recommended to initiate the SIU emergency process.
3. Under the premise of restoring calls, use network tools to capture packets, identify the attack source, and gradually restore devices to the network.

2. Failover Failure

A situation has been encountered on-site where failovers between primary and standby nodes consistently failed. Upon inspection, it was found that binding the floating IP to the standby node's main NIC always failed. Further investigation revealed that the failure was due to the inability to bind the MAX address to the standby node's main NIC. During IBM dual-system configuration, the Service IP must be configured with a fixed MAC address, following the rule of taking the MAC address of the host's main NIC and changing the last two digits to fixed numbers (required to be different from the original main NIC address, e.g., customized to 89). However, this rule did not work in the field.

Problem Resolution: Finally, modify the Service Adapter configuration in the Ethernet configuration within the HA topology diagram by removing the MAC address configuration (set to empty), synchronize the dual-system, and then perform the failover. Everything worked correctly.

Since network faults can lead to the failure of remote login, system maintenance becomes impossible. Therefore, it is recommended that every set of IBM servers be equipped with a maintenance console.

3.4 OS Faults

AIX is a relatively stable operating system; faults are generally caused by human factors:

1. Not applying OS patches as required, such as applying patch 09 to version 433 causing memory leaks.
2. Applications or databases consuming too much memory or having memory leaks, leading to exhaustion of physical memory and paging space, causing the system to hang.
3. Accidental deletion of important directories or files by humans, such as: /dev, /usr, /bin, /sbin, /etc, etc.

Troubleshooting:

1. Check if the operating system patches meet requirements (433 requires version 10 or above, 5.1 requires version 5 or above)

2. Check memory and paging space usage (use 'lsps -a' to verify usage is less than 20%)

3. Check for errors when executing shell commands; check errpt for related error messages

3.5 HA Failure

For failures occurring during HA installation, such as synchronous topology map failure, check the following items to ensure they meet requirements:

1. For HA version 4.4.1, apply patch 15 or higher; for HA version 4.5, apply patch 9 or higher

2. Network-related configuration files must be set according to the installation manual

3. Network IP configuration must be correct and physical links must be operational

4. All versions 4.4.1 and 4.4.0 must use the Standard Edition

5. Do not install both Standard Edition and ES Edition on the same machine simultaneously

6. When applying patches, use the 'smitty update_all' method. Do not select 'apply all patches', because Standard Edition and ES Edition patches are often bundled together, which will cause version inconsistency.

If this is a new deployment, ensure dual-system failover testing is performed and confirmed successful.

If the following operations were performed during subsequent usage, future dual-system switching will fail:

1. Modifications were made on the primary host regarding storage array RAID or shared VG information, such as adding new RAID disks or modifying shared VG configurations

2. New file systems (FS) or new logical volumes (LV) were added within the shared VG

3. The above configurations or modifications were performed only on the primary host, without timely synchronization of shared VG information to the standby host

4. Although the standby host imported the new shared VG information, the shared VG attribute was not modified to prevent automatic startup at system boot

5. The permissions of the standby host's shared file systems and raw devices were not modified to database-operable permissions while the shared VG was active and the shared file systems were mounted.

The following describes the normal HA operation procedure and handling for abnormal situations, for reference:

Normal HA Operation Procedure

1. a. Before starting HA, ensure the following:
2. Physical connections of dual-system components are correct
3. Operating system is running normally
4. HA software installation and configuration are correct, and topology structure and resource group synchronization have completed successfully
5. No HA processes such as clstrmgr, clsmuxpd, or clinfo are currently running. If any are running, perform HA shutdown first
6. No application processes or devices defined in any HA resource groups are currently active (e.g., shared disk VG is not mounted on any node, or database and application processes belonging to resource groups are not started)
7. b. The HA startup process is triggered by clstrmgr after it is activated, which calls relevant scripts in the background. Therefore, whether the startup has completed cannot be determined solely by checking if the cluster process is active. The best method is to monitor the hacmp.out log record:
8. # tail -f /tmp/hacmp.out
9. Startup completion indicator: hacmp.out records 'HACMP Event Summary' (any action completion marker)
10. It is recommended to start the primary and standby nodes sequentially; do not start HA on both nodes simultaneously
11. c. Any event operation must wait until the current event is fully completed before executing the next event
12. d. While HA combined with server redundant components can completely eliminate single points of failure in a dual-system environment, it does not guarantee normal system operation during multiple-point failures. Multiple-point failures include scenarios where both systems experience multiple errors related to HA components simultaneously, or an error on one system triggers an HA event script switch, followed by another error requiring a second resource switch. In such cases, specific situations must be evaluated manually to guide the system back to normal operation
13. e. When shutting down HA, ensure HA is completely shut down before performing subsequent operations. Completion of shutdown can be verified by checking:
14. # lssrc -g cluster
15. No processes are running
16. # tail -f /tmp/hacmp.out
17. hacmp.out records 'HACMP Event Summary'
18. # lsvg -o
19. Shared VG is not mounted on any node server
20. f. By default, any event must complete within 360 seconds; otherwise, HA will automatically trigger the config_too_long event
21. g. If an anomaly or error occurs while executing any event script, HA will hang and unable to complete subsequent event scripts. After 360 seconds, HA will automatically execute the config_too_long script, and hacmp.out will record the execution of this event

Emergency Handling for Abnormal Situations

22. When any event script encounters an anomaly preventing normal HA operation, follow these steps:
23. a. Close all HA-related application processes, such as manually stopping Intelligent Network processes and manually stopping the database.
24. b. Unmount all file systems on shared VGs.
25. c. Execute varyoffvg to take the shared VG offline from the server.
26. d. Execute smitty clstop and select Shutdown mode as "forced" to shut down HA.
27. e. Use the command
28. # ps -ef
29. # lsvg -o
30. # lssrc -g cluster
31. # netstat -i
32. to verify whether the above operations have been completed successfully. If some related processes are not closed, you can use kill to terminate them (note that this cannot guarantee normal system operation; a system crash may occur).
33. f. Check whether the physical connections of the dual-system are intact.
34. g. After closing all HA-related processes using the commands above, restart HA via smitty clstart and re-verify if HA is functioning normally.
35. h. If HA still does not function properly after performing all the steps above, shut down and reboot the system.

3.6 Other Faults

Other faults include:

1. Inability to normally use system commands such as sar, svmon, or topas, resulting in the inability to obtain statistics for CPU, MEM, etc.

Handling method:

Check only the following two packages:

lslpp –l |grep "perfagent.tools"

lslpp –l |grep "bos.acct"

If they are missing, install

Additionally, note that files under the /usr/lib/sa directory must have execute permissions;

/usr/bin/svmon must have u+s permissions.

2. Time synchronization failure caused by Daylight Saving Time (DST) settings.

Our commercial systems must not be configured for Daylight Saving Time. If DST is enabled, it will cause issues with the time synchronization function.

You can check the timezone by running echo $TZ. If the timezone ends with DT, it indicates that DST is in use.

If DST is not desired, you can change it by navigating to smit->system environment->change show date and time->change time zone using system defined values. In the popped-up USE DAYTIME SAVING dialog box, select No. After selecting the appropriate timezone, reboot the machine.

3. Asynchronous I/O not configured.

If asynchronous I/O is not enabled, IO performance will be severely impacted. The check method is as follows:

lsdev -Cc aio |grep "aio0 Available"  If output is returned, it has already been configured

Otherwise, execute:

mkdev -l aio0

chdev -P -l aio0 -a autoconfig='available'

to perform the configuration.

For storage arrays with poor IO access status, asynchronous I/O parameters can be further adjusted according to the following principles:

smity aio-> Change / Show Characteristics of Asynchronous I/O

MINIMUM number of servers -- Set to: Number of physical data disks in the storage array * 5

MAXIMUM number of servers -- Set to: Number of physical data disks in the storage array * 10 (cap at 80 if exceeding)