IBM Power Mini-system Maintenance and Troubleshooting Training Manual (Advanced)

IBM P Series Small Systems

Daily Maintenance / Fault Localization / Troubleshooting

Training Manual / Operation Guide

Chapter 1: Data Center Environment and Physical Inspection

I. Data Center Environmental Requirements

Temperature and Humidity

Optimal Operating Temperature: 20-25°C

Extreme Operating Temperature: 10-40°C

Humidity: 8-80% (at 23°C)

If not operating at optimal temperature, please improve the data center environment.

The data center must also be kept clean.

The data center should remain clean; excessive dust can easily cause resource read/write errors and damage disks or read/write heads in disk drives.

II. Power Requirements

Voltage: Voltage must be stable; voltage spikes can damage equipment.

Voltage Range: 220V +/- 10%, i.e., 200-240V, 50-60Hz

Power Capacity: Depends on machine type and system configuration.

Power Cable: Standard neutral, ground, and live three-phase power; voltage between neutral and ground must not exceed 3.0V.

Power Connection: Connect to the main computer power cable using an air switch or other equipment meeting current requirements. To ensure reliable operation of the computer system, use a stabilized power supply and UPS, and it is recommended to have a generator set available. For redundant power input, two separate input lines are recommended.

III. Hardware Inspection

Check that the installation of servers, disk arrays, power cables, 7133 connections, and host connections comply with requirements.

Server Status Check:

When the server is in startup and normal working status, there should be no information displayed on the LCD screen on the front panel.

1. When information containing numbers and letters appears on the LCD screen, it indicates a hardware alarm. The cause of the alarm can be found by querying the Service Guide for the relevant model. In severe cases, immediately notify an IBM technical expert for troubleshooting.

7133 Status Check:

There are status lights for the 7133 cabinet (next to the power light) and status lights for each hard drive (a row of small lights corresponding one-to-one with each hard drive position) on the front panel of the disk array.

1. When the cabinet status light turns orange-yellow, it indicates a hardware alarm. At this time, check the disk cabinet's power, cabling, hard drives, etc. If there is a hardware failure, replace and correct it immediately. If the specific problem cannot be identified, contact the relevant expert for further diagnosis.

2. When hard drives are working normally, the corresponding hard drive lights will be green. If there is no read/write activity, the green light stays on continuously; if there is read/write activity, the green light flashes irregularly. When a hard drive fails or an SSA loop issue occurs, the hard drive status light will turn off or flash: flashing regularly and continuously at a frequency of 1~3 seconds.

Chapter 2: Daily System Maintenance Procedures

2.1 System Startup

The normal sequence for system startup is as follows:

First, power on external peripherals (disk arrays, tape libraries, etc.).

After all external peripherals complete their self-test, power on the host normally. After the host is powered on, the system performs a self-test. Only after "ok" is displayed on the LCD screen can you press the white POWER key to start up.

Start HACMP on the host. After startup, use the command tail -f /tmp/hacmp.out to check the startup status. Do not proceed to the next step before HACMP has fully started.

Check the server's network address and routing table (use commands such as netstat -i, netstat -rn), check file systems and logical volumes (use commands such as mount, lsvg -o).

Check whether all applications are functioning normally.

2.2 System Shutdown

Stop HACMP (smitty clstop).

Check the HACMP status, verify the server's network address and routing table (use commands such as netstat -i, netstat -rn), check file systems and logical volumes (use commands such as mount, lsvg -o). Do not proceed to the next step before HACMP has fully stopped.

Shut down the host (shutdown –F). To restart the system, use: shutdown -Fr

If necessary, press the white button on the front of the disk array to shut down the disk array.

2.3 View System Error Logs

During system operation, some system errors are recorded in the errlog, and some of these errors are also displayed on the terminal. The following commands can be used to check error logs:

#errpt|more	View all system records

IDENTIFIER TIMESTAM P T C RESOURCE_NAME DESCRIPTION

E85C5C4C 0426104399 P S CFGLFT SOFTWARE PROGRAM ERROR

2BFA76F6 0426104099 T S SYSPROC SYSTEM SHUTDOWN BY USER

9D4CF6E7 0426104399 T O errdemon ERROR LOGGING TURNED ON

1E2AC07E 0426103999 T O errdemon ERROR LOGGING TURNED OFF

1E5EER4T 0423132999 T O clstrmgr OPERATOR NOTIFICATION

Where

IDENTIFIER is the error number, which is often used when checking detailed information.

TIME STAMP is the time tag, recording the time the error occurred. Its format is: MMDDHHMMYY.

T represents Type, recording the error type.

P represents Permanent Error; attention is required.

T represents Temporary Error.

C represents Class, recording the error category, such as

H : Hardware

S : Software

O : Errloger command messages

U : undetermined

RESOURCE_NAME is the source of the error.

DESCRIPTION is the error description.

#errpt -aj <IDENTIFIER>	View detailed system records

Where IDENTIFIER is the error number, for example #errpt -aj 0426104399

#errpt -dH 				View all system hardware error records

2.4 System and Data Backup

Effective and timely system backup is a very important part of system management. When system failures occur, especially when file systems are severely damaged or hard drives fail, system backups are often needed to restore the system. System backups should be performed in the following situations:

1. New Installation. After hardware and system software installation is complete, perform a system backup.

2. Software Changes. Perform a system backup when system software or application software has been modified.

3. Periodic Backup. Perform periodic backups of the system; it is best to do so every two or three months.

Introduction to System Backup Operations:

Enter smitty mksysb at the command line to see the following screen:

Press the F4 key to see the following screen:

Use the up/down keys to select the backup target device (select /dev/rmt0 tape drive),

Press Enter to see the following screen:

Press Enter again to start the backup. Continue until you see the following screen, indicating that the backup is complete:

Backup Precautions:

System backup does not require stopping business operations; business can continue.

It is recommended that customers perform regular system backups (using the smitty mksysb command). Customers can also use SMIT to back up specific VGs or FS of the system as needed.

smitty mksysb only backs up file systems mounted within rootvg; other file systems or data are not backed up, so data backup must be completed separately (it is recommended that customers use TAR format).

If conditions permit, it is best to have more than one backup tape to prevent tape damage.

During the system backup process, you may sometimes receive a prompt that some files in the /tmp directory cannot be backed up, as shown below:

Creating list of files to back up...

Backing up 34025 files..............................

1694 of 34025 files (4%)..............................

2733 of 34025 files (8%).........backup: 0511-449 An error occurred accessing ./

tmp/sh34736.1: A file or directory in the path name does not exist.

backup: 0511-449 An error occurred accessing ./tmp/sh34736.2: A file or director

y in the path name does not exist.

backup: 0511-449 An error occurred accessing ./tmp/sh34736.3: A file or director

y in the path name does not exist.

.....................

15458 of 34025 files (45%)..............................

31920 of 34025 files (93%)...........

0512-003 mksysb may not have been able to archive some files.

The messages displayed on the Standard Error contained additional

information.

This is normal behavior; the backup was successful.

2.5 System Recovery

When a serious system failure occurs such that general maintenance measures cannot restore the original system in the short term, after consulting with the customer, the system can be restored from the most recent system backup tape to fully restore the system environment to that at the time of the last backup. Then, the daily data backup can be restored into the system. At this point, the system can resume normal operation. Thereafter, the customer should conduct a comprehensive analysis and review of the entire event with an IBM engineer to identify the cause of the failure and take corresponding measures to prevent similar incidents from occurring again.

2.6 DUMP

When a software failure causes the system to go down during operation, the machine's LCD display will show 888 102 xxx 0c0 (xxx may be 700 or other values). AIX often records the system status at that time; this is called DUMP.

After a DUMP is generated, insert the tape into the tape drive and use the command # snap -a -o /dev/rmt0 to copy the DUMP file to the tape device /dev/rmt0. Note the block size of the tape drive, the date the DUMP was generated, and the machine model and serial number. Additionally, use # errpt -a >/tmp/err.log to save the error log to /tmp/err.log, and copy /tmp/err.log and /tmp/hacmp.* to a floppy disk or tape. Hand over the tape and floppy disk to the IBM engineer.

2.7 Items for Daily Server Status Checks and Related Commands

1. Run the lsdev command with various parameters; the status of all listed devices should be Available.

#lsdev –C –H –S a	    		List available devices in the system.

#lsdev –Cc processor			List all CPUs in the system.

#lsdev –Cc memory			List all memory in the system.

#lsdev –Cc disk				List all disks in the system.

#lsdev -Cc adapter | grep ent		List all network adapters in the system

#lsdev -Cc adapter | grep scsi		List all SCSI adapters in the system.

#lsdev -Cc adapter | grep ssa		List all SSA adapters in the system.

2 lspv command

#lspv			Show available PVs in the system.

#lspv hdiskn		Show detailed information about hdiskn.

#lsdev –Cc pdisk  Show detailed information about disks in the disk array.

SSA disk detection: Before working on the host, you can power on the disk array first. After the host has fully booted, log in to the host and run the following command: "lsdev –Cc pdisk". You should be able to see all SSA disks, and their status should be Available.

3 lsattr command

# lsattr –E –l mem0	List the size of mem0; this system has 4GB of memory.

4 lsvg command

#lsvg			List all vgs in the system.

#lsvg rootvg		List detailed information about rootvg.

#lsvg –o         List active vgs

5 oslevel command

#oslevel		Show the operating system version.

6 netstat command

#netstat –in	Show each network adapter's configuration and check whether its IP is configured.

7 Run the hardware diagnostic program to check all hardware inside the host by executing the # diag command. If the result shows "No trouble found", it indicates that all components are functioning normally.

8 Use the #diag command (select: Task Selection -> SSA Service Aids) to verify the SSA disk link connection. This can be detected via Link Verification in the SSA tool. If necessary, use Certify Disk to run the disk diagnostic section for surface analysis testing from 1% to 100%. The test results show that all sectors of the host's built-in disks read and write normally.

9 lsps –a Check the usage of PAGING SPACE. If the usage rate exceeds 70%, measures must be taken.

10 lsvg –o | lsvg –il | grep –i stale Check for any stale LVs. If there is output after running this command, measures must be taken.

11 Check if there are error reports sent to the root user (mail).

12 Check dual-machine status: lssrc –g cluster Check whether the three HA worker processes are activated; /usr/sbin/cluster/clstat –a Check if the dual-machine status is up, and check the hacmp.out log for any abnormal information.

13 Use vmstat, topas, sar commands to check system performance, including CPU, memory, IO, and determine if there are performance bottlenecks.

14 Check if the CDE interface can be entered smoothly. If not, check the /etc/hosts file for any incorrect entries.

15 Use smitty ssaraid to check if the RAID disk status is Good. If it shows degrade or any other status, it indicates a problem with the RAID disks.

16 Use sysdumpdev –l to check if the system DUMP settings are normal.

17 Use instfix –ik | grep ML to check if the current operating system patch version (PTF) meets the requirements for stable operation. Generally, OS 433 patches need to be at level 10 or higher, and OS 5.1 patches need to be at level 5 or higher.

18 Use df –kP to check disk space utilization. Please ensure immediate reporting if the following filesystems exceed 80% utilization:

/ /var /usr /home /tmp /zxindata/zxinbak. For other filesystems, report immediately if utilization exceeds 95%.

Alternatively, navigate to each filesystem and use the combined command: find . –size +2048 –o ctime 1 –exec ls –l {} \; to find files larger than 1M or modified within the last day.

2.8 Performance Monitoring and Tuning

Observe using the vmstat 1 command.

kthr memory page faults cpu

----- ----------- ------------------------ ------------ -----------

r b avm fre re pi po fr sr cy in sy cs us sy id wa

Command explanation: The output of the vmstat command reflects the overall system operation status, including CPU, memory, virtual pages, system processes, and system call status.

Check if the CPU is a bottleneck by separately checking the four CPU values and the two kthr values.

Check if the MEM is a bottleneck by separately checking the two Memory values and the six Page values.

Observe using the sar –mu –P ALL command.

Command explanation: sar can be used to collect data reflecting system operation status. Here, it is mainly used to view CPU operation status, whether CPU load balancing exists, and if there is uneven allocation.

Observe using the ps gv | more command.

PID TTY STAT TIME PGIN SIZE RSS LIM TSIZ TRS %CPU %MEM COMMAND

0 - A 4:20 7 12 14516 xx 0 14504 0.0 5.0 swapper

1 - A 1:33 103 78952 79044 32768 25 36 0.0 29.0 /etc/init

Command explanation: ps can be used to view the current status of processes. By combining parameters here, you can observe the time, CPU, and memory consumption of currently running processes. Specifically, %CPU indicates the CPU resources occupied by the process, and %MEM indicates the memory resources occupied by the process. Mainly check for zombie processes marked as <defunc> consuming system resources, as well as the system consumption of the informix database oninit process.

Perform a write operation verification on the disk array using the dd command on the machine, while simultaneously observing the disk using iostat 1 –d hdiskX.

Disks: % tm_act Kbps tps Kb_read Kb_wrtn

Command explanation: iostat can be used to view the system I/O input/output status. Here, it mainly checks the read/write count per second of the disks on the array, and simultaneously estimates the disk array read/write speed HHhHhh.

.

Observe using the netstat –a command.

Active Internet connections (including servers)

Proto Recv-Q Send-Q Local Address Foreign Address (state)

Command explanation: netstat can be used to view the system network status. Here, it mainly checks the network client connection status and overhead. The parameter to check is state. If there are dead connections with state fin_wait, this consumes system network resources, leading to decreased network performance and forming a bottleneck.

2.9 Safety Work Rules

a. Perform regular system backups; the system disk is recommended to be mirrored.

b. When there are system changes or when installing operating system patches, a system backup must be performed.

c. When hard disk failures are detected on 7133 and shark, remind the customer to pay attention to the daily data backup. When replacing hard disks, confirm that the daily data backup has been completed.

d. When replacing sensitive electronic components, anti-static measures must be taken.

e. When plugging or unplugging peripheral devices, power off the peripheral devices.

f. When upgrading host microcode, please pay attention to the integrity of the microcode.

g. When deleting files, please verify that the current path is correct.

h. When decompressing files, please verify the parameters and paths.

Chapter 3: Fault Localization and Troubleshooting

Based on problems encountered in actual commercial systems, we have summarized the following common faults along with their localization methods and solutions.

3.1 Hardware Faults

There are many types of hardware faults with varying impacts on the system. Here, they are classified into two categories based on the severity of their impact on the system: fatal hardware faults and functional-only hardware faults:

Hardware whose damage causes a fatal impact on the system (causing the machine to crash or fail to boot) includes:

Motherboard, CPU, I/O Cabinet (containing local disks, optical drives, PCI slots, etc.) or CEC Cabinet (containing CPU/MEMORY, etc.), wiring between the I/O cabinet and CEC cabinet, power modules, fans, local hard drives, memory damage, etc.

Note: I/O cabinets and CEC cabinets are generally found only in high-end mini-computers, such as the M80; low-end models have them combined.

Damage to these devices will prevent the system from completing self-test, booting, and starting up. Error messages will appear on the LCD display. You can check the cause of the error by referencing the Service Guide using the error code displayed on the LCD. If these hardware failures occur while the system is operating, the system will be suspended or crash.

Hardware whose damage only affects specific functions (the machine will not crash and can boot normally) includes:

Network cards, local hard drives with bad blocks, graphics cards, SSA cards, and other peripheral devices.

Damage to these devices only affects specific functions, such as network functionality, display functionality, access to disk arrays, etc. In the case of local hard drives with bad blocks, you must check whether the bad blocks contain important system files. If they do not contain important system files, system functionality remains unaffected, but it is still recommended to replace the hard drive immediately.

Fault Localization and Troubleshooting:

The above hardware fault information can be identified using:

Error codes on the LCD display, or:

errpt –dH  View the results

Determine which hardware has failed based on the error code. For commercial systems, since they are dual-machine systems, if the damaged machine is the primary server, switch this server to standby mode, then repair the faulty machine to restore the system.

3.2 Disk Array Faults

Faults caused by disk arrays are currently the most frequent and harmful. According to incomplete statistics, they account for over 70% of total faults. Specifically, potential causes of disk array faults include:

Disk array hard drives, 7133 cabinets, SSA cards on the host, SSA cables connecting the 7133 to the host, hard drive placement and SSA cable connection methods, voltage used by the disk cabinet and surrounding magnetic fields, microcode for disk arrays/hard drives/SSA cards, etc.

All of these may cause abnormalities in the 7133.

Problems with the 7133 disk array are the most complex, typically involving physical damage as the primary cause and environmental factors as secondary causes, such as improper cabling, incorrect disk insertion positions, failure to monitor system alerts promptly, etc., leading to system interruptions. Based on our experience, regardless of what hardware fault causes a 7133 failure, the system will generate an alert. If the issue is detected and addressed in time, the occurrence of a full fault can generally be prevented.

Fault Localization:

7133 hardware faults can also be identified using:

errpt –dH  View the results

Accompanying error codes include:

B4C00618 0115140004 P H ssa0 RESOURCE UNAVAILABLE

FE9E9357 0401082304 P H ssa0 DISK OPERATION ERROR

FE9E9357 1205000803 P H pdisk3 DISK OPERATION ERROR

03913B94 1122031103 U H LVDD HARDWARE DISK BLOCK RELOCATION ACHIEVED

613E5F38 1121125103 P H LVDD I/O ERROR DETECTED BY LVM

625E6B9A 0401090004 P H ssa0 ADAPTER DETECTED OPEN SERIAL LINK

26CA120B 0206081104 P H ssa0 CACHE STORAGE CARD

All error codes indicate an anomaly in the 7133. The red portion indicates a definite hardware fault requiring immediate inspection and action; otherwise, the disk array will soon become inaccessible. Regarding the blue portion:

625E6B9A 0401090004 P H ssa0 ADAPTER DETECTED OPEN SERIAL LINK

It indicates that the SSA loop has opened. An open loop not only affects IO performance but also increases risk. If another loop also encounters issues, the disk array will become inaccessible.

Open loops generally occur in two scenarios: 1) If 625E6B9A errors are reported frequently, such as several times a day, it is highly likely that a hardware fault exists. Although this will not cause disk array access failure, the cause must be identified and resolved immediately. Refer to the description below for troubleshooting methods. 2) If 625E6B9A errors are reported occasionally, each situation must be handled specifically. It could be a false alarm due to heavy read/write activity. Follow the method described below for troubleshooting. If no specific cause is found, continue monitoring.

26CA120B 0206081104 P H ssa0 CACHE STORAGE CARD

This error generally occurs when: the SSA card has write cache enabled and FastWrite is turned on, and the rechargeable nickel-cadmium battery used for write cache power supply on the SSA card has reached or approached its safe lifespan. Such errors will affect IO write performance. Furthermore, since the primary and standby servers must synchronize the write cache on the SSA card when FastWrite is enabled, it may even impact primary-standby synchronization. Specific solutions can be referenced in the text below.

Troubleshooting:

For issues indicated by the red portion of the error, the diag command can generally be used for further diagnosis:

#diag -> Task Selection -> SSA Service Aids -> Link Verification 

Check if there are ??? drive letters in the loop or if any hard drive status is not good.

Or use:

#smitty ssaraid -> List All Defined SSA RAID Arrays

to check if the state of RAID disks in the disk array is Good. If the status is degrade or any other state, it indicates a problem with the RAID disk.

At this point, it is not recommended to perform individual disk certification. Instead, immediately notify IBM to prepare identical model and size hard drives (at least two units) to arrive at the site for further diagnosis and replacement of the faulty disk.

625E6B9A 0401090004 P H ssa0 ADAPTER DETECTED OPEN SERIAL LINK

troubleshooting method:

1. Insert dummy disks into all empty slots in the 7133. (Dummy disk: A placeholder shell, equivalent to a plastic model of an SSA hard drive installed in the disk array. It is used only when the 16 slots of the disk array are not fully populated with SSA hard drives.)

2. Check the Loop status: diag --> Task Selection --> SSA Service Aids --> Link Verification. Under normal Adapter Port, the two columns of numbers should be continuous and uninterrupted, and the Status should be good. If the Physical column shows ??????? symbols or the Status is not good, it indicates that a hard drive or link fault already exists. Immediate action must be taken to perform further checks to determine if a hard drive needs replacement. The method to check if a single disk is problematic is as follows:

diag--->Task Selection-->SSA Service Aids-->Certify Disk

Select the suspected faulty hard disk for inspection

3. If there is a hardware fault in the 7133, it can be observed from the status lights:

When a single hard disk fails or is not in use, its hard disk status light on the panel will not light up.

The array's status light (yellow) will illuminate.

Or the indicator light for the port connected to the SSA cable will also turn off.

If no issues are found using the above three methods, but the system still reports an Open Serial Link error, continued tracking is recommended.

26CA120B 0206081104 P H ssa0 CACHE STORAGE CARD

Handling procedure:

Background information:

IBM Power systems with 7133 storage arrays typically configure an SSA card that includes a backup battery. This battery protects information in the fast write cache on the SSA card during sudden power failures. The safe lifespan of this battery is generally 22,000 hours, approximately two and a half years. Therefore, when the fast write mode is enabled, this battery generally needs to be replaced after about two and a half years.

Problem manifestation:

Battery protection on the SSA card is implemented via a counter on the card. For every hour of operation, the counter increments by one. When this count approaches or exceeds 22,000, the system generates a 26CA120B hardware alarm:

26CA120B 0206120904 P H ssa1 CACHE STORAGE CARD

The following command can be used to check the status of the SSA card: (-a followed by the logical device name of the card; here assumed to be ssa0)

ssa_fw_status -a ssa0 -p (Check the time the battery has been active, in hours)

ssa_fw_status -a ssa0 -l (Check the safe working life of the battery, in hours)

ssa_fw_status -a ssa0 -c (Check if the fast write function on the SSA card is activated)

Handling steps: (Listed in order of preference; select one method only)

1) Replace the battery

During system idle periods, replace the SSA card batteries on both the primary and standby systems. Methods include:

Stop standby system → Replace standby SSA card battery → Start standby system (dual-system service) → Failover between primary and standby →

Stop original primary system → Replace original primary SSA card battery → Start original primary system (dual-system service)

2) If the system generates a 26CA120B battery alarm, use ssa_fw_status -a ssa0 -c to check whether Fast Write on the active SSA card (typically ssa0) is in the inactive state. If so, execute the following commands as root:

ssa_format -l ssa0 -b

errclear 0

/usr/lib/errstop

/usr/lib/errdemon

Replacing the battery can be deferred until a replacement is available, but this may impact storage array read/write performance during that period.

3) If the battery cannot be replaced in the short term and the Fast Write function on the primary card of the main system remains Active, it is recommended to manually disable the fast write function.

1) First, stop the dual-system service.

2) On Node 1, modify the hdisk attribute:

smitty dev->ssa disks->ssa logical disks->change /show characters of ... -> [choose hdisk2] -> fast write [no]

3) On Node 1, activate the volume group.

varyonvg zxinvg

4) On Node 1, deactivate the volume group.

varyoffvg zxinvg

5) Execute on Node 2:

smitty dev->ssa disks->ssa logical disks->change /show characters of ... -> [choose hdisk2] -> fast write [no]

Check whether fast write has been changed to no (as long as Node 1 has done it, Node 2 does not need to modify it again).

6) Execute on Node 2:

rmdev -dl hdisk2

	   cfgmgr -v

7) Execute on Node 2:

lspv (Verify whether hdisk2 has been detected).

8) Then execute:

varyonvg zxinvg

varyoffvg zxinvg

9) Restart the dual-system service.

For errors appearing as:

B4C00618 0115140004 P H ssa0 RESOURCE UNAVAILABLE

Without accompanying other red-highlighted errors, troubleshoot using the following methods:

It may be that 'enable user of hot spare' is enabled in hdisk, but no hot spare disk is actually configured.

This can be resolved using the following method:

Check the RAID status:

#smitty ssaraid                                                           

list all defined SSA RAID arrays : all are in status good (ssa0 )

Check whether a hot spare is configured:

list/identify SSA Physical disks-->List hot spares : none

Modify the attributes of hdisk configured for each SSA card:

: set "enable use of hot spare" to "no"

Then execute:

# /usr/lib/errstop

# cp /var/adm/ras/errlog   /var/adm/ras/errlog.bak

#/usr/lib/errdemon

#errclear 0

Refer to the following description for physical hard disk connections: (If the old site does not have a hot spare configured, calculate the wiring method according to the number of disks when no hot spare is configured)

Describe the following scenarios:

1. When the number of data disks is less than or equal to 6, it is recommended to configure one hot spare
2. When the number of data disks is greater than or equal to 8 and less than or equal to 10, it is recommended to configure two hot spares
3. When the number of data disks is greater than or equal to 12 and less than or equal to 14, it is recommended to configure two hot spares
4. Wiring and configuration rules for configuring one hot spare when the number of data disks is less than or equal to 6:
5. Host A1 and A2 connect to magnetic array positions 1 and 8; Standby host A1 and A2 connect to magnetic array positions 4 and 5
6. Ensure that hard disks are inserted in the front row positions 1\4\5\8. Other disks can be inserted in any empty slots. Note that data disks on both sides of 1-4 and 5-8 must be inserted in an axially symmetric relationship (e.g., 1 and 8 are axially symmetric, 4 and 5 are axially symmetric)
7. Dummy disks must be inserted in positions where no hard disks are installed
8. When performing RAID 0+1, considering performance issues, mirror the hard disks in positions 1-4 and 5-8 in a one-to-one axial symmetric manner. Example as follows:

IBM's sorting of pdisks differs from their actual physical positions. Therefore, you must first use the lsdev -C | grep pdisk command to find and record their correspondence, such as:

pdisk0 Available 11-08-1641-01-P SSA160 Physical Disk Drive

pdisk1 Available 11-08-1641-05-P SSA160 Physical Disk Drive

pdisk2 Available 11-08-1641-04-P SSA160 Physical Disk Drive

pdisk3 Available 11-08-1641-08-P SSA160 Physical Disk Drive

Observe the third column; the middle characters containing '-01-' indicate the physical hard disk position. The above correspondence indicates

The physical slots corresponding to pdisk0-pdisk3 are positions 1\5\4\8 in the front row. Therefore, when using smitty ssaraid to perform RAID 0+1, the disks at positions 1\8 (pdisk0/pdisk3) must be mirrored, and the disks at positions 4\5 (pdisk2/pdisk1) must be mirrored. Thus, select Primary Disks/Secondary Disks as follows:

smitty ssaraid-> Add an SSA RAID Array -> ...

Primary Disks [Select: pdisk0 pdisk2] -- Disks in slots 1\4 as primary disks

Secondary Disks [Select: pdisk3 pdisk1] -- Disks in slots 8\5 as secondary disks

This requirement is only for performance considerations and does not add other features.

1. Hot spare disks can be inserted into any empty slot among the front row 8 positions. There are no special position requirements. The creation method is as follows: Use the lsdev -C | grep pdisk command to find its pdisk number, e.g., pdisk5, then use the following method to create it as a hot spare disk.

smitty ssaraid -> Change Use of Multiple SSA Physical Disks -> Select the SSA card (the SSA card connected to the magnetic array, default is ssa0)

-> Select the pdisk to be used as hot spare, e.g., pdisk6 -> New Use -> Select 'hot spare' and press Enter to create

1. Wiring and configuration rules for configuring two hot spares when the number of data disks is greater than or equal to 8 and less than or equal to 10:
2. Host A1 and A2 connect to magnetic array positions 1 and 12; Standby host A1 and A2 connect to magnetic array positions 8 and 9
3. Ensure that positions 1-8 in the front row of the magnetic array are filled with data disks. Positions 9-12 for data disks and hot spare disks can be arranged arbitrarily.
4. Dummy disks must be inserted in positions where no hard disks are installed
5. When performing RAID 0+1, considering performance issues, mirror the hard disks in positions 1-4 and 5-8 in a one-to-one axial symmetric manner,

For positions 9-12 containing data disks, mirror them correspondingly. The remaining two disks serve as hot spares. Example as follows:

IBM's sorting of pdisks differs from their actual physical positions. Therefore, you must first use the lsdev -C | grep pdisk command to find and record their correspondence, such as:

pdisk0 Available 11-08-1641-01-P SSA160 Physical Disk Drive

pdisk1 Available 11-08-1641-05-P SSA160 Physical Disk Drive

pdisk2 Available 11-08-1641-04-P SSA160 Physical Disk Drive

pdisk3 Available 11-08-1641-08-P SSA160 Physical Disk Drive

pdisk4 Available 11-08-1641-02-P SSA160 Physical Disk Drive

pdisk5 Available 11-08-1641-03-P SSA160 Physical Disk Drive

pdisk6 Available 11-08-1641-07-P SSA160 Physical Disk Drive

pdisk7 Available 11-08-1641-06-P SSA160 Physical Disk Drive

pdisk8 Available 11-08-1641-09-P SSA160 Physical Disk Drive

pdisk9 Available 11-08-1641-11-P SSA160 Physical Disk Drive

pdisk10 Available 11-08-1641-10-P SSA160 Physical Disk Drive

pdisk11 Available 11-08-1641-12-P SSA160 Physical Disk Drive

Observe the third column; the middle characters containing '-01-' indicate the physical hard disk position. The above correspondence indicates

The physical slots corresponding to pdisk0-pdisk11 are positions 1\5\4\8\2\3\7\6\9\11\10\12 respectively. Therefore, when using smitty ssaraid to perform RAID 0+1, the first eight disks should be mirrored in an axial symmetric manner according to positions 1-4 and 5-8. If positions 9 (pdisk8)/12 (pdisk11) are selected for mirroring in the 9-12 range

Thus, select Primary Disks/Secondary Disks as follows:

smitty ssaraid-> Add an SSA RAID Array -> ...

Primary Disks [Select: pdisk0 pdisk4 pdisk5 pdisk2 pdisk8] Physical slots 1-4, 9

Secondary Disks [Select: pdisk3 pdisk6 pdisk7 pdisk1 pdisk11] Physical slots 8-5, 12

This requirement is only for performance considerations and does not add other features.

1. Create hot spares from the remaining physical positions 10 (pdisk10) and 11 (pdisk9), using the same method as step e)
2. Wiring and configuration rules for configuring two hot spares when the number of data disks is greater than or equal to 12 and less than or equal to 14:
3. Host A1 and A2 connect to magnetic array positions 1 and 16; Standby host A1 and A2 connect to magnetic array positions 8 and 9
4. Leave positions 3 and 14 open for hot spares. If there are 12 data disks, leave positions 2 and 15 open for dummy disks, and insert data disks in all other positions. If there are 14 data disks, insert data disks in all remaining 14 positions
5. When performing RAID 0+1, considering performance issues, mirror the hard disks in positions 1-8 and 9-16 in a one-to-one axial symmetric manner. Hot spares also satisfy the axial symmetric relationship. Example as follows:

IBM's sorting of pdisks differs from their actual physical positions. Therefore, you must first use the lsdev -C | grep pdisk command to find and record their correspondence, such as:

pdisk0 Available 11-08-1641-01-P SSA160 Physical Disk Drive

pdisk1 Available 11-08-1641-05-P SSA160 Physical Disk Drive

pdisk2 Available 11-08-1641-04-P SSA160 Physical Disk Drive

pdisk3 Available 11-08-1641-08-P SSA160 Physical Disk Drive

pdisk4 Available 11-08-1641-02-P SSA160 Physical Disk Drive

pdisk5 Available 11-08-1641-03-P SSA160 Physical Disk Drive

pdisk6 Available 11-08-1641-07-P SSA160 Physical Disk Drive

pdisk7 Available 11-08-1641-06-P SSA160 Physical Disk Drive

pdisk8 Available 11-08-1641-09-P SSA160 Physical Disk Drive

pdisk9 Available 11-08-1641-11-P SSA160 Physical Disk Drive

pdisk10 Available 11-08-1641-10-P SSA160 Physical Disk Drive

pdisk11 Available 11-08-1641-12-P SSA160 Physical Disk Drive

pdisk12 Available 11-08-1641-16-P SSA160 Physical Disk Drive

pdisk13 Available 11-08-1641-13-P SSA160 Physical Disk Drive

pdisk14 Available 11-08-1641-15-P SSA160 Physical Disk Drive

pdisk15 Available 11-08-1641-14-P SSA160 Physical Disk Drive

If the third column contains '-01-' in the middle character, it indicates the physical hard disk location. The correspondence above represents

the physical slots corresponding to pdisk0-pdisk15 are positions 1\5\4\8\2\3\7\6\9\11\10\12\16\13\15\14 respectively. Therefore, when using smitty ssaraid to create RAID0+1, creating mirrors with axis symmetry for 1-8 and 9-16 means selecting Primary Disks/Secondary Disks as follows:

smitty ssaraid-> Add an SSA RAID Array -> ...

Primary Disks [Select: pdisk0 pdisk4 pdisk2 pdisk1 pdisk7 pdisk6 pdisk3]

Physical slots 1-8, excluding position 3

Secondary Disks [Select: pdisk12 pdisk14 pdisk13 pdisk11 pdisk9 pdisk10 pdisk8]

Physical slots 16-9, excluding position 14

This requirement is based on performance considerations only and does not add other features.

The remaining physical locations 3 (pdisk5) and 14 (pdisk15) disks are configured as hot spares, following the same method as step e)

3.3 Network Faults

Since our application has a strong dependency on the network, any complete network outage or transient interruption will have a significant impact on the system. Network faults generally fall into two categories: hardware faults (such as NIC failures, switch failures, or router failures) and software faults (such as IP packet attacks or network congestion in the network).

Hardware Faults:

1. Network Interface Card (NIC)

For NIC faults, since commercial systems adopt IBM's HA dual-machine system, and each machine is configured with at least two NICs, when a single NIC or network cable fails, the HA software will take measures to implement Service IP failover.

NIC fault localization method: Use errpt –dH to view errors indicating NIC service interruption, then use diag to diagnose the NIC.

NIC fault troubleshooting method: If the diagnosis reveals a problem with the NIC, shut down the system and replace it. (If it is the primary host, manually switch to the standby machine first before operating.)

2. Switch

Our networks generally adopt a dual-network, dual-plane structure. Therefore, if a switch in one network plane fails, network services will not be interrupted. However, the following point deserves attention: The direct link between the primary and standby switches must remain operational. Otherwise, if a network interface card (NIC) failover occurs on either the IBM server host or the SIU host, it will cause a disconnection between the IBM server host and the SIU host, resulting in a complete service outage.

There is another scenario: in some places, for network security, different devices connected to the switch are assigned to different VLANs. Simultaneously, the connection between the primary and standby switches is changed to use two ports connected and grouped into a channel group. A situation encountered was:

1) When the IBM server's main NIC migrated from the default VLAN0 to VLAN6, the network experienced a transient interruption of 15 to 30 seconds.

2) The channel group operating between the primary and standby switches is malfunctioning, and the link is down.

When scenario 1) occurred, the IBM dual-machine system performed a primary/standby NIC failover. However, due to issue 2), the IBM host could not establish a link with the SIU.

Troubleshooting and Recommendations:

1. It is recommended that during initial setup, a power-loss failure test be conducted on a switch in one plane of the dual-network, dual-plane configuration.
2. It is essential to perform network inter-access tests between devices connecting the primary and standby switches to ensure connectivity.
3. Avoid making configuration changes on the switch during the day. If modifications are necessary, they should be performed at night with a strict plan.

Software Faults:

1. Network Congestion

Since the system operates within a closed network, the likelihood of network congestion is relatively low. However, if the network topology is complex, this may still occur. On the host, this manifests as severe packet loss when pinging the host, intermittent connectivity between the host and SIU, packet loss, and abnormal device functionality.

Troubleshooting and Recommendations:

1. Try to isolate NT/2000 machines from the network.
2. If the situation does not improve, it is recommended to initiate the SIU emergency process.
3. Under the premise of restoring calls, use network tools to capture packets, identify the attack source, and gradually restore devices to the network.

2. Failover Failure

A situation was encountered on-site where failovers between the primary and standby machines consistently failed. Upon inspection, it was found that binding the floating IP to the standby machine's main NIC always failed. Further investigation revealed the root cause: failing to bind the MAX address to the standby machine's main NIC. During IBM dual-machine configuration, the Service IP must be configured with a fixed MAC address. The rule is to take the MAC address of the primary host's main NIC and change the last two digits to fixed numbers (required to be different from the original main NIC address, e.g., customized as 89). However, this rule did not work in the field.

Resolution: Finally, the Service Adapter configuration in the Ethernet settings of the HA topology diagram was modified by removing the MAC address configuration (setting it to empty), synchronizing the dual machines, and performing the failover again. Everything worked correctly.

Since network faults can lead to the failure of remote logins, preventing system maintenance. Therefore, it is recommended that every set of IBM servers be equipped with a maintenance console.

3.4 OS Faults

AIX is a relatively stable operating system; faults usually arise from human factors:

1. Not applying OS patches as required, such as applying patch 09 to version 433, causing memory leaks.
2. Applications or databases consume too much memory or have memory leaks, causing physical memory and paging space to be exhausted, resulting in system hang.
3. Important directories or files were manually deleted, such as: /dev, /usr, /bin, /sbin, /etc, etc.

Troubleshooting:

1. Check if the operating system patches meet requirements (433 requires patch level 10 or higher, 5.1 requires patch level 5 or higher).

2. Check memory and paging space usage (use lsps -a; usage must be less than 20%).

3. Check for errors when executing shell commands and verify errpt for related error reports.

3.5 HA Failure

For failures occurring during HA installation, such as synchronous topology diagram failure, check the following items to ensure they meet requirements:

1. For HA version 4.4.1, patch level 15 or higher is required; for HA version 4.5, patch level 9 or higher is required.

2. Network-related configuration files must be set according to the installation manual.

3. Network IP configuration must be correct and ensure physical links are operational.

4. All versions 4.4.1 and 4.4.0 must use the Standard Edition.

5. Standard Edition or ES Edition cannot be installed simultaneously on the same machine.

6. When applying patches, use the smitty update_all method; do not select 'apply all patches', because patches for Standard Edition and ES Edition are often bundled together, which would cause version inconsistency.

If it is a new deployment, you must ensure that a dual-system failover test is performed and confirmed successful.

If the following operations are performed during subsequent use, subsequent dual-system failovers will fail:

1. Modifications were made on the primary host regarding disk array RAID or shared VG information, such as adding new RAID disks or modifying shared VG configurations.

2. New File Systems (FS) or new Logical Volumes (LV) were added within the shared VG.

3. The above configurations or modifications were performed only on the primary host without timely synchronization of shared VG information to the standby host.

4. Although the standby host imported the new shared VG information, the shared VG attribute was not modified to prevent automatic startup at system boot.

5. The permissions of the shared file systems and raw devices on the standby host were not modified to database-operable permissions while the shared VG was active or the shared file systems were mounted.

The following are normal HA operation procedures and handling methods for abnormal situations, for reference:

HA Normal Operation Procedure

1. a. Before starting HA, ensure the following:
2. Physical connections of the dual-system structure components are correct and error-free.
3. Operating system is running normally.
4. HA software installation and configuration are correct, and topology structures and resource groups have synchronized successfully.
5. No HA processes such as clstrmgr, clsmuxpd, or clinfo are currently running. If any are running, perform HA shutdown first.
6. No application processes or devices defined in any HA resource group are currently active (e.g., shared disk VG is not mounted on any node; database and application processes belonging to the resource group are not started).
7. b. The HA startup process involves clstrmgr being activated, which then calls and executes relevant scripts in the background. Therefore, whether the startup process has completed cannot be determined solely by checking if the cluster process is active. The best method is to track the hacmp.out log record:
8. # tail -f /tmp/hacmp.out
9. Startup completion indicator: hacmp.out records HACMP Event Summary (any action completion marker).
10. It is recommended to start the primary and standby nodes sequentially; do not start HA on both primary and standby machines simultaneously.
11. c. Any event operation must wait until the current event is fully executed before proceeding to the next event.
12. d. HA combined with server redundant components can completely eliminate single points of failure in dual-system environments. However, when multiple simultaneous failures occur, normal system operation is not guaranteed. Multiple failures include multiple errors related to HA components occurring simultaneously on both systems, or an error on one system triggering an HA startup event script switch, followed by another error requiring a second resource switch. In such cases, specific situations must be evaluated and manual intervention may be required to guide the system back to normal operation.
13. e. When shutting down HA, ensure HA is completely shut down before performing subsequent operations. Whether the shutdown action is complete can be verified by checking:
14. # lssrc -g cluster
15. No processes are running.
16. # tail -f /tmp/hacmp.out
17. hacmp.out records HACMP Event Summary.
18. # lsvg -o
19. Shared VG is not mounted on any node server.
20. f. Any event must defaultly complete within 360 seconds; otherwise, HA will automatically trigger the config_too_long event.
21. g. If an anomaly or error occurs while executing any event script, HA will hang and unable to complete subsequent event scripts. After 360 seconds, HA will automatically trigger the config_too_long script, and hacmp.out will record the execution of this event.

Emergency Handling of Abnormal Conditions

22. If an event script fails to run normally during execution, causing HA to malfunction, follow these steps:
23. a. Stop all application processes related to HA manually, such as stopping intelligent network processes and database services.
24. b. Unmount all file systems on shared VGs.
25. c. Execute varyoffvg to take the shared VG offline from the server.
26. d. Execute smitty clstop and select Shutdown mode as "forced" to shut down HA.
27. e. Use the command
28. # ps -ef
29. # lsvg -o
30. # lssrc -g cluster
31. # netstat -i
32. to verify whether the above operations were completed successfully. If some related processes are still not stopped, you can terminate them using kill (note: this does not guarantee normal system operation and may cause a system crash).
33. f. Check whether the physical connections of the dual-system setup are intact.
34. g. After closing all HA-related processes via the above commands, restart HA using smitty clstart and re-verify that HA is functioning correctly.
35. h. If HA still fails to function properly after performing all the above steps, shut down and reboot the system.

3.6 Other Faults

Other faults include:

1. Inability to use system commands such as sar, svmon, topas, etc., resulting in failure to obtain statistics for CPU, MEM, etc.

Remediation method:

Check only the following two packages:

lslpp –l |grep "perfagent.tools"

lslpp –l |grep "bos.acct"

If they are missing, install them.

Also note that files in the /usr/lib/sa directory must have execute permissions;

/usr/bin/svmon must have u+s permissions.

2. Time synchronization issues caused by daylight saving time settings.

Our commercial systems must not be configured for daylight saving time. If configured, it will cause problems with the time synchronization function.

You can check the time zone using echo $TZ; if the time zone ends with DT, daylight saving time is enabled.

If you do not wish to use it, change it via smit -> system environment -> change show date and time -> change time zone using system defined values. In the displayed USE DAYTIME SAVING dialog box, select No, choose the appropriate time zone, and then reboot the machine.

3. Asynchronous I/O not configured.

If asynchronous I/O is not enabled, it will severely impact I/O performance. The verification method is as follows:

lsdev -Cc aio |grep "aio0 Available"  If output is returned, it has already been configured

Otherwise, execute:

mkdev -l aio0

chdev -P -l aio0 -a autoconfig='available'

to perform the configuration.

For storage array access with poor I/O conditions, you can further adjust the asynchronous I/O parameters according to the following principles:

smity aio-> Change / Show Characteristics of Asynchronous I/O

MINIMUM number of servers -- Set to: Number of physical data disks in the storage array * 5

MAXIMUM number of servers -- Set to: Number of physical data disks in the storage array * 10 (cap at 80 if exceeding)