IBM DS / MD Storage: Piece Failed Handling Steps

1. What is a Piece?

Multiple virtual disks are created in an MD storage disk group. The blocks occupied by each virtual disk on every hard drive are identical (same Offset and Length). We refer to these blocks on each hard drive as a 'Piece'.

2. View the status of all MD storage drives in the StorageArrayprofile.txt log file

Depending on the configuration, a single hard drive can contain multiple Pieces. A physical hard drive may show an Optimal status, but not all Pieces within it are necessarily Optimal; this must be confirmed via the stateCapturedata.txt log file.

3. How to handle multiple Piece failures in MD storage (illustrated with a specific example)

Symptoms: The status of a certain VD in an MD system is Fail, while other VDs are normal, and all hard drive statuses are OPTIMAL.

Main processing steps:

Operation	Command
MDSM Cancel Hot Spare
View failed piece SSID	`vdmShowRAIDVolList`
View piece failure time and piece#	`vdmShowRVOosPieces SSID`
Bring pieces online in chronological order	`vdmRevivePiece <ssid>, <piece>`
Restart the peer controller	`altCtlReset 2`

View SSID of the failed piece

Through the statecapturedata.txt log file, search for the keyword 'piece' to see how many Pieces of VD5 have failed.

Remote telnet / serial port, enter command vdmShowRAIDVolList, or search for this command in statecapturedata.txt to view the SSID information as follows:

Total RAIDVolumes: 16
                                  Curr  Prim
RVAddr     Ssid   State       Pcs Owner Owner VG# PI ExclOp
===========================================================
0x04278878 000000 RV_OPTIMAL   22 This  Alt     1  0 NONE
0x0427d9d8 000001 RV_FAILED    22 Alt   Alt     1  0 MEDIA SCAN
0x0427ebb8 000002 RV_FAILED    22 Alt   Alt     1  0 NONE
0x0427e2c8 000003 RV_FAILED    22 This  This    1  0 NONE
0x042bfbd0 000004 RV_OPTIMAL   22 This  This    1  0 NONE
0x042c0950 000005 RV_DEGRADED  22 Alt   Alt     1  0 NONE
0x042d3fa0 000006 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x04349a68 000007 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x0434c138 000008 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x0434e808 000009 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x042d4890 000010 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x0434a358 000011 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x0434ca28 000012 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x04376078 000013 RV_FAILED    22 Alt   Alt     1  0 NONE
0x04376968 000014 RV_OPTIMAL   22 Alt   Alt     1  0 NONE
0x04377258 000015 RV_FAILED    22 Alt   Alt     1  0 NONE

Check the offline order of Pieces

Use the Majorevent LOG tool, or use the serial port / Telnet command vdmShowRVOosPieces <ssid> to check the offline order of Pieces.

IBM DS / MD Storage Piece Failed Handling Steps

Bring pieces online in chronological order

After confirming the hard drive offline order, SSID number, and Piece, you can bring pieces online using command vdmRevivePiece <ssid>, <piece> to force them online. Bring all failed pieces online except the last one (the earliest to go offline).

How to view the <piece> number: Command vdmShowRVOosPieces <ssid> shows 0rd:0x000X, where this 'X' is in hexadecimal.

Refer to the table below to convert the hexadecimal X to decimal N; N-1 is the <piece> number we need to input.

Hexadecimal	1	2	3	4	5	6	7	8	9	a
Decimal	1	2	3	4	5	6	7	8	9	10
Hexadecimal	b	c	d	e	f	10	11	12	13	14
Decimal	11	12	13	14	15	16	17	18	19	20

Therefore, in the image above, 0rd:0x0014 corresponds to piece 19, and the online command is vdmRevivePiece 3,19.

Restart the peer controller

altCtlReset 2

The RAID status becomes degraded; ask the user to verify data integrity (e.g., by performing a backup).

Under normal circumstances, after the controller restarts, the other FAILED piece will automatically trigger a rebuild. Once the rebuild completes, reconfigure the hot spare drive.

Images localized into English for illustration; labels may vary by product version.