HPE 3PAR Detailed Hard Drive Replacement Guide (servicemag)
1. Understand the servicemag start/resume working principle
2. How to repair disks in Degraded state
3. Steps to replace NORMAL disks
4. Summary of situations encountered during disk replacement
1. Understand the servicemag start/resume working principle
First, we must understand that replacing a disk involves executing the servicemag command to replace problematic disks. For F/T/V class systems, engineers must manually execute servicemag start –log to migrate data from the problematic disk to spare/freespace. After replacing the disk, execute servicemag resume to migrate the data back to the new disk. For SS7000, the system automatically executes servicemag start to migrate data from the problematic disk to spare/freespace. After the engineer replaces the disk, the system automatically runs servicemag resume to migrate the data back to the new disk.
Next, let us discuss the working principle of servicemag: In fact, the servicemag start or servicemag resume commands are scripts composed of a series of sub-commands.
Let us first discuss the servicemag start process:
a. First, the storage marks the ldalloc of this disk as off, so that host IO cannot access this problematic disk. The storage executes 'setpd ldalloc off <pdid>'.
b. Migrate all data on this disk to free/spare space. At this time, the storage executes movepd<PDID>.
c. Then stop the disk rotation. The command executed is 'controlpd spindown <disk WWPN>'.
d. Finally, remove this disk from the LOOP. The command executed is 'controlmag offloop <cage id> <mag>'.
Next, let us discuss the servicemag resume process:
a. After replacing the disk, we must first load the disk into the LOOP environment. The command executed is
‘controlmag onloop <cage id> <magazine>’
b. Start the disk rotation. The command executed is: 'controlpd spindup <DISK WWPN>
c. At this time, migrate the previously moved data back to this new disk. The command executed is:
‘ moverelocpd <NEW PDID>’ showpdch –from oldpdid
d. Finally, allow HOST IO to continue accessing this newly replaced disk. The command to execute is:
’ setpd ldalloc on <NEW PDID>’
The above is the working principle of servicemag for disk replacement.
2. How to repair disks in Degraded state
Understanding the principle after disk replacement, we need to know when to replace disks and under what conditions replacement is necessary.
We know that when more than 6 chunklets on a physical disk are marked as bad, the physical disk will show a Failed state. Only then do we need to replace the disk. If it is only in 'degraded' state, we do not need to replace it.
So, how should we handle a degraded state? In most cases, the disk status changes to degraded only when the disk has a single loop, rather than due to physical disk damage.
1. First, execute 'showpd –degraded –failed' output to check which disks are in degraded or failed state.
2. Then execute 'showpd–i' to find the WWPN of the problematic disk.
3. At this time, execute: 'controlpd clearerr <drive wwn>'
4. Wait 60 seconds, then execute 'controlmag offloop –disk <pos> cage<num><mag num>'
5. Wait another 60 seconds, then execute 'controlmag onloop –disk <pos> cage<num><magnum>'
6. Next, wait 60 seconds, then clear the error information of the problematic disk again: 'controlpd clearerr <drive wwn>'
7. If the problem persists, escalate it for further analysis.
Suggestion: If encountering a disk in failed state, you can also try following the operation steps for degraded disks. If it remains in failed state, replace the disk.
Case follows:
cli% controlmag offloop -disk 3 cage13 0
cli% controlmag on loop -disk 3 cage13 0
cli% controlpd clearerr 2210000A33006FB6
3. Steps to replace NORMAL disks
Some customers discover many SCSI IO error messages at the operating system level and strongly request disk replacement. However, at this time, our disk status is all NORMAL. In principle, we do not replace them. But if the customer insists on replacing to avoid potential risks, we cannot operate according to the servicemag start/resume commands. Keep this in mind; if executed, it will report an error and fail to proceed.
The following steps are for replacing NORMAL disks under a customer V-Class:
a. Set the ldalloc of the disk to be replaced to off
V400 cli% setpd ldalloc off 28
V400 cli% showpd -c 28
------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used - -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK FailFree Uninit Unavail Fail OK Fail Free Uninit Fail
28 3:9:0 FC degraded 272 56 0 0 0 216 0 0 0 0 0 0
------------------------------------------------------------------------------------------
1total 272 56 0 0 0 216 0 0 0 0 0 0
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
... ...
28 3:9:0 FC degraded 272 56 0 0 0 216 0 0 0 0 0 0
29 3:9:1 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
30 3:9:2 FC normal 272 57 0 181 0 0 0 0 0 34 0 0
31 3:9:3 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
-------------------------------------------------------------------------------------------
32total 8704 1869 0 5565 0 216 0 0 0 1054 0 0
b. Migrate the chunklets of the disk to be replaced to Spare Space
V400 cli% movepdtospare -nowait 28
Are you sure you want to move the pd to spare?
select q=quit y=yes n=no: y
Move -State-- ----------------------Detailed_State-----------------------
28:0-18:218 normal ch_move_pending,normal
28:1-29:231 degradedch_move_pending,remote_disks_used
28:2-24:220 normal ch_move_pending,normal
......
28:267-6:235 normal ch_move_pending,normal
28:268-6:236 normal ch_move_pending,normal
28:269-6:237 normal ch_move_pending,normal
28:270-6:238 normal ch_move_pending,normal
28:271-6:239 normal ch_move_pending,normal
c. We use showpdch <pdid> to check if the chunklets of the disk to be replaced have been migrated out
V400 cli% showpdch 28
Pdid Chnk LdName LdCh State Usage Media Sp Cl From To
28 0 tp-15-sa-0.0 0 normal relsrc valid N N ---18:218
28 1 tp-15-sd-0.0 72 normal relsrc valid N N ---29:231
... ...
28 269 log0.0 31normal relsrc valid N N --- 6:237
-------------------------------------------------------------------
Total chunklets: 54
d. Check disk status:
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 2 0 32 0 0
... ...
27 3:0:3 FC normal 272 59 0 179 0 0 0 6 0 28 0 0
28 3:9:0 FC degraded 272 52 0 0 0 220 0 0 0 0 0 0
29 3:9:1 FC normal 272 58 0 180 0 0 0 6 0 28 0 0
30 3:9:2 FC normal 272 57 0 181 0 0 0 3 0 31 0 0
31 3:9:3 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
-------------------------------------------------------------------------------------------
32total 8704 1865 0 5565 0 220 0 56 0 998 0 0
e. At this time, we execute the servicemag start command
V400 cli% servicemag start -log -pdid 28
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag start -log -pdid 28
... servicing disks in mag: 3 9
... normal disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
.................... WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
... not normal disks:
The servicemag start operation will continue in thebackground.
V400 cli% servicemag status
Cage 3, magazine 9:
The magazine is being brought offline due to a servicemagstart.
The last status update was at Sun Feb 15 11:25:34 2015.
Unable to provide a relocation estimate
servicemag start -log -pdid 28 -- is in Progress
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK Fail FreeUninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 3 0 31 0 0
... ...
27 3:0:3 FC normal 272 59 0 179 0 0 0 6 0 28 0 0
28 3:9:0 FC normal 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1 FC normal 272 58 0 0 0 208 0 6 0 0 0 0
30 3:9:2 FC normal 272 57 0 0 0 212 0 3 0 0 0 0
31 3:9:3 FC normal 272 58 0 0 0 214 0 0 0 0 0 0
-----------------------------------------------------------------------------------------
32total 8704 1813 0 5024 0 906 0 100 0 861 0 0
f. Check servicemag status :
V400 cli% servicemag status
Cage 3, magazine 9:
The magazine is being brought offline due to a servicemagstart.
The last status update was at Sun Feb 15 11:25:34 2015.
Unable to provide a relocation estimate
servicemag start -log -pdid 28 -- is in Progress
V400 cli% servicemag status -d
Cage 3, magazine 9:
The magazine is being brought offline due to a servicemagstart.
The last status update was at Sun Feb 15 11:25:34 2015.
Unable to provide a relocation estimate
The cumulative output so far is:
servicemag start -log -pdid 28
... servicing disks in mag: 3 9
... normal disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
.................... WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
... not normal disks:
... normal disk WWN [2000B452537B9D40] Id [28] and notlogging it
... vacant disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
... relocating chunklets from degraded raid sets to sparespace
g. Verify via showpdch <pdid> that all data has been migrated off the disk to be replaced:
V400 cli% showpdch 28
No chunklet information available.
h. servicemag start completes normally:
V400 cli% servicemag status -d
Cage 3, magazine 9:
The magazine was successfully brought offline by a servicemagstart command.
The command completed Sun Feb 15 11:32:34 2015.
The output of the servicemag start was:
servicemag start -log -pdid 28
... servicing disks in mag: 3 9
... normal disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
.................... WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
... not normal disks:
... normal disk WWN [2000B452537B9D40] Id [28] and notlogging it
... vacant disks: WWN[2000B452537B9D40] Id [28] diskpos [0]
... relocating chunklets from degraded raid sets to sparespace
... logging chunklets from pd WWN [2000B452537B9DC4] Id [29]
... logging chunklets from pd WWN [2000B452537B99C1] Id [30]
... logging chunklets from pd WWN [2000B452537B9988] Id [31]
... retrying logging chunklets...
... spinning down disk WWN [2000B452537B9D40] Id [28]
... spinning down disk WWN [2000B452537B9DC4] Id [29]
... spinning down disk WWN [2000B452537B99C1] Id [30]
... spinning down disk WWN [2000B452537B9988] Id [31]
... bypassing mag 3 9
... bypassed mag 3 9
servicemag start -log -pdid 28 --Succeeded
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
0 0:0:0 FC normal 272 58 0 180 0 0 0 3 0 31 0 0
... ...
28 3:9:0? FC degraded 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1? FC degraded 272 0 45 0 0 221 0 0 6 0 0 0
30 3:9:2? FC degraded 272 0 41 0 0 228 0 0 3 0 0 0
31 3:9:3? FC degraded 272 0 43 0 0 229 0 0 0 0 0 0
-------------------------------------------------------------------------------------------
32total 8704 1640 129 5024 0 950 0 91 9 861 0 0
i. Replace PDID28 disk: Ensure the four status LEDs for the hard drives in the magazine containing the disk to be replaced are green.
j. After replacing the disk, a new drive appears but without a PDID:
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
--- 3:9:0 FC degraded 272 0 0 0 0 0 0 0 0 0 0 0
28 3:9:0 FC degraded 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1 FC degraded 272 0 41 0 0 225 0 0 6 0 0 0
30 3:9:2 FC degraded 272 0 38 0 0 231 0 0 3 0 0 0
31 3:9:3 FC degraded 272 0 40 0 0 232 0 0 0 0 0 0
--------------------------------------------------------------------------------------------
33total 8976 1640 119 5024 0 960 0 101 9 851 0 0
k. You must manually add this drive:
V400 cli% showpd -i
Id CagePos State ----Node_WWN---- --MFR-------Model------ -Serial- -FW_Rev- Protocol MediaType
--- 3:9:0 degraded 5000CCA017A49568HITACHI HVIPC0300GBFC15K JXXXHZ3N 3P02 FC Magnetic
... ...
28 3:9:0 degraded 2000B452537B9D40 SEAGATESEGLE0300GBFC15K 6SJ415EY 3P00 FC Magnetic
29 3:9:1 degraded 2000B452537B9DC4 SEAGATESEGLE0300GBFC15K 6SJ41A0N 3P00 FC Magnetic
30 3:9:2 degraded 2000B452537B99C1 SEAGATESEGLE0300GBFC15K 6SJ419D1 3P00 FC Magnetic
31 3:9:3 degraded 2000B452537B9988 SEAGATESEGLE0300GBFC15K 6SJ4198V 3P00 FC Magnetic
---------------------------------------------------------------------------------------------------
33total
V400 cli% admitpd 5000CCA017A49568
Warning: The following disks are marked as being part ofanother system.
Admitting these disks to this system will cause any existingdata on them
to be erased.
-Disk_WWN- System_IDSystem_Name
5000CCA017A49568 5322 0
Are you sure?
select y=yes n=no: y
l. Verify the newly added drive has a new PDID: (In this case, the new PDID is 32)
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
-Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OKFail Free Uninit Unavail Fail OK Fail Free Uninit Fail
28 3:9:0 FC degraded 272 0 0 0 0 272 0 0 0 0 0 0
29 3:9:1 FC degraded 272 0 41 0 0 225 0 0 6 0 0 0
30 3:9:2 FC degraded 272 0 38 0 0 231 0 0 3 0 0 0
31 3:9:3 FC degraded 272 0 40 0 0 232 0 0 0 0 0 0
32 3:9:0 FC degraded 272 0 0 0 272 0 0 0 0 0 0 0
-------------------------------------------------------------------------------------------
33total 8976 1640 119 5024 272 960 0 101 9 851 0 0
m. Ensure all disks are recognized:
V400 cli% showpd
---Size(MB)--------Ports----
Id CagePos Type RPM State Total Free A B Capacity(GB)
... ...
28 3:9:0 FC 15 degraded 278528 0 ----- ----- 300
29 3:9:1 FC 15normal 278528 00:6:4 1:6:4* 300
30 3:9:2 FC 15normal 278528 0 0:6:4*1:6:4 300
31 3:9:3 FC 15normal 278528 00:6:4 1:6:4* 300
32 3:9:0 FC 15 degraded 278528 278528 0:6:4*1:6:4 300
-----------------------------------------------------------------------
33total 91914245423104
n. Execute servicemag resume at this point to migrate data back to the new disk:
V400 cli% servicemag resume 3 9
Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag resume 3 9
... mag 3 9 already onlooped
... firmware is current on pd WWN [2000B452537B9DC4] Id [29]
... firmware is current on pd WWN [2000B452537B99C1] Id [30]
... firmware is current on pd WWN [2000B452537B9988] Id [31]
... firmware is current on pd WWN [5000CCA017A49568] Id [32]
... firmware is current on pd WWN [2000B452537B9D40] Id [28]
... checking for valid disks...
... disks in mag : 3 9
... normal disks: WWN[2000B452537B9DC4] Id [29] diskpos [1]
.................... WWN[2000B452537B99C1] Id [30] diskpos [2]
.................... WWN[2000B452537B9988] Id [31] diskpos [3]
.................... WWN[5000CCA017A49568] Id [32] diskpos [0]
... not normal disks: WWN [2000B452537B9D40] Id[28]
... verifying spare space for disks 29 and 29
... verifying spare space for disks 30 and 30
... verifying spare space for disks 31 and 31
... verifying spare space for disks 28 and 32
... playback chunklets from pd WWN [2000B452537B9DC4] Id[29]
... playback chunklets from pd WWN [2000B452537B99C1] Id[30]
... playback chunklets from pd WWN [2000B452537B9988] Id[31]
... playback chunklets from pd WWN [5000CCA017A49568] Id[32]
The servicemag resume operation will continue in thebackground.
o. Confirm there are no active servicemag processes, indicating successful data migration:
V400 cli% servicemag status -d
No servicemag operations logged.
V400 cli% showpd -c
-------- Normal Chunklets -------- ---- Spare Chunklets ----
- Used -- -------- Unused -------- - Used - ---- Unused ----
Id CagePos Type State Total OK Fail FreeUninit Unavail Fail OK Fail Free Uninit Fail
29 3:9:1 FC normal 272 58 0 180 0 0 0 0 0 32 2 0
30 3:9:2 FC normal 272 57 0 181 0 0 0 0 0 34 0 0
31 3:9:3 FC normal 272 58 0 180 0 0 0 0 0 34 0 0
32 3:9:0 FC normal 272 56 0 138 44 0 0 0 0 0 34 0
-----------------------------------------------------------------------------------------
32total 8704 1869 0 5703 44 0 0 0 0 1043 45 0
4. Summary of potential issues during the disk replacement process
The above summarizes the working principle of disk replacement and the procedures for repairing degraded and normal state disks. Below we list methods to handle failures during the disk replacement process: In fact, if servicemag start/resume fails, we execute the sub-commands within it separately to resolve the issue.
1. How to handle failure after executing servicemag resume:
cli% servicemag status -d
Cage 1, magazine 12:
A servicemag resume command failed on thismagazine.
The command completed at WedApr 9 05:37:33 2014.
The output of the servicemag resume was:servicemag resume 1 12
... mag 1 12 already onlooped
... firmware is current on pd WWN[xxxxxxxxxxxxxxxx] Id [yy]
... firmware is current on pd WWN[xxxxxxxxxxxxxxxx] Id [zz]
... checking for valid disks...
... disks in mag :1 12
... normal disks: WWN [xxxxxxxxxxxxxxxx] Id [yy] diskpos [0]
... not normal disks: WWN [xxxxxxxxxxxxxxxx] Id [zz]
... verifying spare space for disksyy and yy
... playback chunklets from pd WWN[xxxxxxxxxxxxxxxx] Id [yy]
... All chunklets played back /relocated.
... cleared logging mode for cage 1mag 12
... relocating chunklets from sparespace
Failed
--
Could not find a destination forchunklet aa:bbb
All chunklets from the target driveyy are in use
Please also check for unclean ormedia failed chunklets
Use the -partial flag to move asmany of the rest as possible
servicemag resume 1 12 -- Failed
At this point, servicemag resume has failed.
We need to handle this using the following commands:
a. setpd ldalloc on<PD_ID> // Will start the re-initialization of thedrive.
b. showpd -c<PD_ID> // The engineer must wait until the value in the 'uninit' column is 0 before proceeding to the next step
c. servicemag unmark<CAGE_ID><BAY_ID> // Cancel the current servicemag resume operation using the servicemag unmark command; answer 'y' when prompted
d. servicemag clearstatus <CAGE_ID><BAY_ID>
e. servicemag resume <CAGE_ID><BAY_ID> Execute the servicemag resume operation again to restore data to the newly replaced disk.
Another scenario is that some logical disks get stuck during the servicemag resume process and cannot be migrated back.
root@1629459-0 Mon Jun 3011:46:50:~# showpdch 82
PdidChnk LdNameLdCh State Usage Media Sp Cl From To
82 234media-fail.82.234 0 normal ld failed N N -------------------------------------------------------------------------
Total chunklets: 1
root@1629459-0 Mon Jun 3011:47:53:~# removeld media-fail.82.234
Removing ld media-fail.82.234
select q=quit y=yes n=no: y
root@1629459-0 Mon Jun 3011:49:27:~#
In this case, delete the failed media LD and then continue executing the servicemag resume operation.
For the scenarios summarized above regarding disk replacement, please ensure you record all your operational steps in OPT before upgrading, so L2 engineers are aware of what operations you performed.