Hitachi GAD (Global-Active Device) Daily Maintenance Manual

Global-Active Device (GAD) Daily Fault Maintenance Manual

Hitachi Data Systems

CS&S North China

Note/Explanation

GAD Status Explanation

For some faults, it is impossible to predict which storage unit will be in a normal state after the fault occurs.

For example, when an arbitration disk (primary/secondary) fails, it is impossible to determine which storage unit (primary/secondary) takes over business I/O. Therefore, a correct method to restore the GAD status must be selected only after careful judgment.

Usually, the "L/M" mode in the pair status is used to determine which volume is currently accepting I/O.

The "L/M" mode indicates that the volume is available when the GAD Pair status is normal (normal state during regular operation).

The "L/L" mode indicates that the volume is available when a fault occurs.

The "B/B" mode indicates that the volume is unavailable when a fault occurs.

In the following example, because the P-VOL status is "L/L", the P-VOL accepts I/O at this time, and the S-VOL cannot be used:

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

In the following example, because the S-VOL status is "L/L", the S-VOL accepts I/O at this time, and the P-VOL cannot be used:

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

When an arbitration disk (primary/secondary) fault occurs, even after the arbitration disk recovers to the normal state, you must wait another 5 minutes before the Pair synchronization command can execute normally.

Common Pair Status Explanations

1. "PAIR" indicates that the primary/secondary storage replicated volumes are in a normal synchronized state.
2. "PSUS/SSUS" indicates that the primary/secondary storage replicated volumes are in a suspended state (this suspension is due to manual intervention, not a fault).
3. "PSUE" indicates that the primary/secondary storage replicated volumes are in a suspended state (this suspension is caused by a fault, not manual intervention, and is an abnormal state).
4. "COPY" indicates that the primary/secondary storage volumes are executing data synchronization (the replication may be the initial copy when creating a Pair or an incremental copy after fault recovery).
5. "SMPL" indicates that no Pair relationship has been established between the primary/secondary storage volumes.

CCI Tool Explanation

The CCI tool controls the replication relationship between primary and secondary storage by sending commands to the storage.

CCI Deployment: Currently, one CCI server is deployed at Site A (Primary Center) and one at Site B (Secondary Center). Note that CCI manages the replication direction between primary and secondary storage.

Roles in CCI:

In CCI, the primary storage volume is represented as "P-VOL".

In CCI, the secondary storage volume is represented as "S-VOL".

CCI Instance Explanation:

In CCI, one instance number must be assigned for each primary/secondary storage volume.

Replication from Site A to Site B:

Production Volume Instance Number: horcm100

Backup Volume Instance Number: horcm200

Replication from Site B to Site A

Production Volume Instance Number: horcm10

Backup Volume Instance Number: horcm20

Site A CCI (IP: x.x.x.x, administrator/ <password>):

Manages replication from Site A VSPG1K (SN:xxxxx) to Site B VSPG1K (SN:xxxxx).

At this time, the roles of the volumes in both storage centers are:

Site A VSPG1K (SN:xxxxx) volume is P-VOL (Production Volume)

Site B VSPG1K (SN:xxxxx) volume is S-VOL (Backup Volume)

Site B CCI (IP: x.x.x.x, administrator/ <password>):

Manage replication from Site B VSPG1K (SN:xxxxx) to Site A VSPG1K (SN:xxxxx).

At this time, the roles of the volumes in both storage centers are:

Site B VSPG1K (SN:xxxxx) volume is P-VOL (Production Volume)

Site A VSPG1K (SN:xxxxx) volume is S-VOL (Backup Volume)

GAD Pair Status Check Method

Taking replication from Site A VSPG1K (SN:xxxxx) to Site B VSPG1K (SN:xxxxx) as an example:

Log in to Site A CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairdisplay -g TDB_A1 -fxce –CLI -IH100

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TDB_A1 000200-001200(L) (CL1-A-2, 1, 4)350532 0200.P-VOL PAIR NEVER , 100 1200 - - 0 2 - - - L/M

TDB_A1 001200-000200(R) (CL3-A-2, 0, 1)350557 1200.S-VOL PAIR NEVER , 100 0200 - - 0 2 - - - L/M

Taking replication from Site B VSPG1K (SN:xxxxx) to Site A VSPG1K (SN:xxxxx) as an example:

Log in to Site B CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PAIR NEVER , 100 1100 - - 0 2 - - - L/M

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL PAIR NEVER , 100 0100 - - 0 2 - - - L/M

GAD Pair Manual Suspend Method

Taking replication from Site A VSPG1K (SN:xxxxx) to Site B VSPG1K (SN:xxxxx) as an example:

Log in to Site A CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairsplit -g TDB_A1 -IH100

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TDB_A1 000200-001200(L) (CL1-A-2, 1, 4)350532 0200.P-VOL PSUS NEVER , 98 1200 - - 0 2 - - - L/L

TDB_A1 001200-000200(R) (CL3-A-2, 0, 1)350557 1200.S-VOL SSUS NEVER , 100 0200 - - 0 2 - - - B/B

Taking replication from Site B VSPG1K (SN:xxxxx) to Site A VSPG1K (SN:xxxxx) as an example:

Log in to Site B CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairsplit -g TKY_A1 -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PSUS NEVER , 98 1100 - - 0 2 - - - L/L

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL SSUS NEVER , 100 0100 - - 0 2 - - - B/B

Common Pair Faults and Recovery Methods

Fault 1: After the Pair is suspended (caused by a fault, at this time the Production Volume P-VOL status is "L/L")

Production Storage Volume (P-VOL) status is "L/L"

Backup Storage Volume (S-VOL) status is "B/B"

Taking replication from Site B VSPG1K (SN:xxxxx) to Site A VSPG1K (SN:xxxxx) as an example:

Log in to Site B CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PSUE NEVER , 100 1100 - - 0 2 - - - L/L

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL PAIR NEVER , 100 0100 - - 0 2 - - - B/B

At this time, since the Production Volume (P-VOL) status is "L/L", current business I/O uses the P-VOL. The normal pariresync synchronization method can be used to recover the Pair status:

Recovery Steps:

1. Confirm that the primary/secondary storage status is normal
2. Confirm that the primary/secondary storage arbitration device status is normal
3. Confirm that the remote link status between primary/secondary storage is normal
4. Execute the Pair synchronization command pairresync to recover Pair synchronization

C:\Windows>pairresync -g TKY_A1 -fxce -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL COPY NEVER , 92 1100 - - 0 2 - - - L/L

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL COPY NEVER , 100 0100 - - 0 2 - - - B/B

1. Wait until synchronization is complete and the status reaches "PAIR". Then confirm that both P-VOL/S-VOL statuses have changed to "L/M", indicating the GAD Pair has returned to normal. Displayed as follows:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PAIR NEVER , 100 1100 - - 0 2 - - - L/M

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL PAIR NEVER , 100 0100 - - 0 2 - - - L/M

Fault 2: After the Pair is suspended (caused by a fault, at this time the Backup Volume S-VOL status is "L/L")

Production Storage Volume (P-VOL) status is "B/B"

Backup Storage Volume (S-VOL) status is "L/L"

Taking replication from Site B VSPG1K (SN:xxxxx) to Site A VSPG1K (SN:xxxxx) as an example:

Log in to Site B CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PSUE NEVER , 100 1100 - - 0 2 - - - B/B

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL PAIR NEVER , 100 0100 - - 0 2 - - - L/L

At this time, since the backup volume (S-VOL) status is "L/L", current business I/O uses the S-VOL. Use the pairresync --swaps synchronization method to restore the Pair status:

Recovery steps:

1. Confirm that the primary and secondary storage systems are functioning normally.
2. Confirm that the arbitration device status for primary and secondary storage is normal.
3. Confirm that the remote link status between primary and secondary storage is normal.
4. Execute the Pair synchronization command pairresync -swaps to restore Pair synchronization.

C:\Windows>pairresync -g TKY_A1 -fxce -swaps -IH20 (This command must be executed on the backup storage CCI; at this time, log in to Site A CCI to execute the command)

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.S-VOL COPY NEVER , 92 1100 - - 0 2 - - - L/L

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.P-VOL COPY NEVER , 100 0100 - - 0 2 - - - B/B

1. Wait until synchronization completes and reaches the "PAIR" status, then confirm that both P-VOL/S-VOL statuses change to "L/M" as shown below:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI –IH20

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.S-VOL PAIR NEVER , 100 1100 - - 0 2 - - - L/M

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.P-VOL PAIR NEVER , 100 0100 - - 0 2 - - - L/M

1. Then execute the Pair synchronization command pairresync -swaps to restore Pair synchronization.

C:\Windows>pairresync -g TKY_A1 -fxce -swaps -IH10 (This command must be executed on the primary storage CCI; at this time, log in to Site B CCI to execute the command)

1. Check the Pair status and role relationship at Site B CCI to verify correctness, as shown below:

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PAIR NEVER , 100 1100 - - 0 2 - - - L/M

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL PAIR NEVER , 100 0100 - - 0 2 - - - L/M

Fault 3: When a Pair is suspended (caused by a fault; at this time, production volume P-VOL/backup volume S-VOL status is "B/B")

Production storage volume (P-VOL) status is "B/B"

Backup storage volume (S-VOL) status is "B/B"

Example: Replication from Site B VSPG1K (SN:xxxxx) to Site A VSPG1K (SN:xxxxx):

Log in to Site B CCI server IP: x.x.x.x, administrator/ <password>)

Execute the following command:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PSUE NEVER , 100 1100 - - 0 2 - - - B/B

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL SSUE NEVER , 100 0100 - - 0 2 - - - B/B

At this time, since both production volume P-VOL and backup volume S-VOL statuses are "B/B", GAD has completely failed. To restore normal operation, the GAD Pair relationship must be completely removed and the Pair rebuilt.

Recovery steps:

1. Confirm that the primary and secondary storage systems are functioning normally.
2. Confirm that the arbitration device status for primary and secondary storage is normal.
3. Confirm that the remote link status between primary and secondary storage is normal.
4. Completely remove the Pair relationship and execute the following command:

C:\Windows>pairsplit –S -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL SMPL NEVER , - 1100 - - 0 2 - - - --

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL SMPL NEVER , - 0100 - - 0 2 - - - ---

1. Rebuild the Pair relationship and execute the following command:

C:\Windows>paircreate –vl –f never –jq 1 -g TKY_A1 -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL COPY NEVER , - 1100 - - 0 2 - - - --

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL COPY NEVER , - 0100 - - 0 2 - - - ---

1. Verify that the Pair status is restored:

C:\Windows>pairdisplay -g TKY_A1 -fxce –CLI -IH10

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

TKY_A1 000100-001100(L) (CL1-A-2, 1, 4)350557 0100.P-VOL PAIR NEVER , 100 1100 - - 0 2 - - - L/M

TKY_A1 001100-000100(R) (CL3-A-2, 0, 1)350532 1100.S-VOL PAIR NEVER , 100 0100 - - 0 2 - - - L/M

Several common GAD failure scenarios

Secondary storage failure

1. Failure status (Primary volume P-VOL status "L/L", Secondary volume S-VOL status "B/B")

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - B/B

1. Recovery steps:

Restore the arbitration disk (secondary) status to normal

Restore the Remote Path status between primary and secondary storage to normal.

Use the pairresync command to restore the synchronized pair (note that at this time the P-VOL status is "L/L"; simply execute the synchronization command normally).

C:\Windows>pairresync -g SC-XLAB1E -IH48

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL COPY NEVER , 9 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL COPY NEVER , 100 7000 - - 0 2 - - - B/B

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

Primary storage failure

1. Failure status

At this time, the secondary volume S-VOL status is "L/L"; at this time, the S-VOL receives I/O:

C:\Windows>pairdisplay -g SC-XLAB1E -fxce –l -IH49

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL3-A-2, 0, 1)350232 7001.S-VOL SSWS NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(L) (CL3-A-2, 0, 0)350232 7000.S-VOL SSWS NEVER , 97 7000 - - 0 2 - - - L/L

1. Recovery steps
2. Restore the arbitration disk (secondary) status to normal
3. Restore the Remote Path status between primary and secondary storage to normal.
4. Execute pairdisplay on the CCI of the primary storage to check the pair status as follows:

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL SSWS NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL SSWS NEVER , 9 7000 - - 0 2 - - - L/L

Use the pairresync command to restore the synchronized pair. Note that at this time the S-VOL status is "L/L"; execute the swaps synchronization command.

C:\Windows>pairresync -g SC-XLAB1E -swaps -IH49 ------ (This command must be executed on the backup storage CCI)

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH49

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.P-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

C:\Windows>pairresync -g SC-XLAB1E -swaps -IH48 ------ (This command must be executed on the primary storage CCI)

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

Primary storage arbitration disk failure

Note:

For some failures, it is impossible to predict which storage will be in a normal state after the failure occurs.

For example, when an arbitration disk (primary/standby) fails, it is impossible to determine which storage has taken over business IO at that time. Therefore, careful judgment is required before selecting the correct method to restore the GAD state.

Typically, the "L/M" mode in the pair status is used to determine which volume is currently accepting IO.

"L/M" mode indicates that the volume is receiving IO.

"B/B" mode indicates that the volume is not receiving IO.

1. Failure status:
2. At this time, the P-VOL status is "L/L", and the S-VOL status is "B/B".

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

1. Recovery steps:
2. Restore the link from the primary storage to the arbitration storage.
3. If the arbitration disk status is blocked, format the arbitration disk within the arbitration storage first, and also format it in the primary storage.
4. Confirm that the primary storage arbitration disk status is normal.
5. Check the pair status using Pairdisplay output; P-VOL should be "L/L".

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

1. Execute the normal synchronization command pairresync in the CCI of the primary storage:

C:\Windows>pairresync -g SC-XLAB1E -IH48

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL COPY NEVER , 9 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL COPY NEVER , 100 7000 - - 0 2 - - - B/B

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

Standby storage arbitration disk failure

Note:

For some failures, it is impossible to predict which storage will be in a normal state after the failure occurs.

For example, when an arbitration disk (primary/standby) fails, it is impossible to determine which storage has taken over business IO at that time. Therefore, careful judgment is required before selecting the correct method to restore the GAD state.

Typically, the "L/M" mode in the pair status is used to determine which volume is currently accepting IO.

"L/M" mode indicates that the volume is receiving IO.

"B/B" mode indicates that the volume is not receiving IO.

1. Failure status:
2. At this time, the P-VOL status is "L/L", and the S-VOL status is "B/B".

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

1. Recovery steps:
2. Restore the link from the primary storage to the arbitration storage.
3. If the arbitration disk status is blocked, format the arbitration disk within the arbitration storage first, and also format it in the primary storage.
4. Confirm that the primary storage arbitration disk status is normal.
5. Check the pair status using Pairdisplay output; P-VOL should be "L/L".

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

1. Execute the normal synchronization command pairresync in the CCI of the primary storage:

C:\Windows>pairresync -g SC-XLAB1E -IH48

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL COPY NEVER , 9 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL COPY NEVER , 100 7000 - - 0 2 - - - B/B

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

All primary/standby arbitration disks

Note:

For some failures, it is impossible to predict which storage will be in a normal state after the failure occurs.

For example, when an arbitration disk (primary/standby) fails, it is impossible to determine which storage has taken over business IO at that time. Therefore, careful judgment is required before selecting the correct method to restore the GAD state.

Typically, the "L/M" mode in the pair status is used to determine which volume is currently accepting IO.

"L/M" mode indicates that the volume is receiving IO.

"B/B" mode indicates that the volume does not receive IO

1. Fault status:
2. At this time, the P-VOL status is "L/L", and the S-VOL status is "B/B"

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

1. Recovery steps:
2. Restore the link from the primary storage to the arbitration storage
3. If the arbitration disk status is blocked, it must first be formatted within the arbitration storage, and then formatted in the primary storage.
4. Confirm that the arbitration disk status on the primary storage is normal
5. Check pair status using Pairdisplay output; P-VOL should be "L/L"

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PSUE NEVER , 100 7001 - - 0 2 - - - L/L

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PSUE NEVER , 100 7001 - - 0 2 - - - B/B

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PSUE NEVER , 9 7000 - - 0 2 - - - L/L

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PSUE NEVER , 100 7000 - - 0 2 - - - B/B

1. Execute the normal synchronization command pairresync in the CCI of the primary storage:

C:\Windows>pairresync -g SC-XLAB1E -IH48

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL COPY NEVER , 9 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL COPY NEVER , 100 7000 - - 0 2 - - - B/B

C:\Windows>pairdisplay -g SC-XLAB1E -fxce -IH48

Group PairVol(L/R) (Port#,TID, LU),Seq#,LDEV#.P/S,Status,Fence, %,P-LDEV# M CTG JID AP EM E-Seq# E-LDEV# R/W

SC-XLAB1E 007001-007001(L) (CL1-A-2, 1, 4)350216 7001.P-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007001-007001(R) (CL3-A-2, 0, 1)350232 7001.S-VOL PAIR NEVER , 100 7001 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(L) (CL1-A-2, 1, 3)350216 7000.P-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

SC-XLAB1E 007000-007000(R) (CL3-A-2, 0, 0)350232 7000.S-VOL PAIR NEVER , 100 7000 - - 0 2 - - - L/M

Simultaneous failure of primary storage and its arbitration disk

1. If both P-VOL and S-VOL are in "B/B" status, the only solution is to delete the GAD pair and recreate the GAD pair
2. Delete GAD pairs
3. This must be done in Storage Navigator (only way to use the force option)
4. Storage Navigator can be accessed using the emergency link or by right clicking on the array in HCS and selecting Remote Connections
5. Select the GAD pairs that are reporting P-Vols and S-Vols blocked
6. Select More Options and then Delete Pairs
7. Note: by default the Delete Mode is set to Force. In the confirmation screen to delete pairs nothing should need to be changed.
8. Click Finish