EMC DMX / VMAX Maintenance Study Notes
VMAX Study Notes
VMAX Series
CN-Serial VMAX1
CK-Serial VMAX2
Component Explanation
engine: Engine; more engines mean higher performance
BE: Module responsible for disk communication
FE: Module responsible for host communication
Management Module: Manages communication
SIB: SIB cards present in two cabinets for data flow transmission
Disk Models
VMAX 600GB disks have an extremely high failure rate, often causing 6 or 7 disks to fail at once. If such disks are recommended against use, maintenance is a major pitfall.
Disks not recommended for DMX: 101-000-111/112 (101-000-326, 101-000-052)
Disks incompatible with VMAX: 005049031/005049033/005049032/
Disks compatible with VMAX: 005051000 /005048950/005051002/005051004
Replacing spare parts in environmetal
Replacing power modules on VMAX carries risk; removing one power module may cause a DAE drop.
If the Director shows DD, 90% of the time replacement is not needed; verification must be performed via environmetal check.
If all Directors are lost, replacement of the Director is required.
Remove the Director from the front. Note that new spare Directors do not come with memory; you must use the original memory.
Diagnosis Process
1. Execute '9b,a' within inline in symmwin
2. Run health check2. If bad disks exist, they will appear under bad FRUs, where 'suspected' indicates suspected failures.
Click the row containing the error and click 'replace' on the right side to start the replacement script.
Host Management Software Solution Enable
symdisk -sid xxx -failed list to check for bad disks
symevent -sid xxx -all list to view all errors, including power and battery issues
DMX Study Notes
Remote Management
Use EMC Remote to connect to the console remotely.
You can disable the firewall by unchecking the private option, then connect to the network port behind the SP console using an external network cable.
Host Management Software Solution Enable (SE)
symdisk -sid xxx -failed list to check for bad disks
symevent -sid xxx -all list to view all errors, including power and battery issues
For DMAX host LUN provisioning, use symmask; for VMAX hosts, use symaccess.
Command to view FC ports on the host
symcfg -sid xxx list -dir all -port to view FA WWNs (DMX3/4)
symmask -sid xxx list login dmx (DMX3/4)
symaccess -sid xxx list login (VMAX)
PowerPath Commands
powermt display
powermt display dev=allpowermt watch
powermt display every=1Analysis Process
Log in to symmwin
Execute '9B,A' in inline mode
Then run health check from the menu: procedures -> procedure Wizards -> RTS/CEs serivice -> Key to seccessCheck spare part number via environmental --> VPD
In the procedures menu, open environmetal tools and run the run health check button. You can view power supply, fan, card and SPS LED status, but not disks. To check disks, use health check.Bin modification approach
In symmwin, go to the File menu: first run IMPL form ..., then save bin file. Use the bin modification tool to modify it, then import the bin configuration through the symmwin script procedure Wizards.
Maintenance precautions
1. DMX and VMAX require frequent proactive checks.
2. Common faults involve Director cards; always inspect the pins. Failure to do so may damage the pins. Replacing the backplane requires a shutdown and is very complex. Replacement requires simultaneous entry from top and bottom.
3. Director1 is installed in slot 0; Director2 is installed in slot 1. DA1 and DA16 are a pair, satisfying the Rule of 17.
3. In the system bay, XCM carries the highest risk. If two XCMs fail or replacement is faulty, the storage becomes unusable. Exercise caution during replacement.
4. When replacing DAE power supplies, be aware that replacing one might cause another power supply to drop offline, resulting in 15 drives below going offline.
5. When replacing the console server, use the previous hard drive and apply necessary corrections.
6. KVM UPS failure triggers a red light. Replace with part 078-000-053. This does not affect functionality; press and hold for a few extra seconds when powering on.
7. Viewing SPS from the rear: A is on the right, B is on the left.
8. In case of loop issues where all drives drop in E7, use inline commands to manually bring them up one by one.
9. Batteries weigh approx. 60 jin (30 kg); exercise safety precautions during replacement.
Performance slowness, link inspection
Have there been any recent changes, such as in the database or topology?
Run health check in SymmWin on the storage systemBrocade Fibre Channel switch optical power check: sfpshow 3 -f | grep Power, followed by porterrshow and then portstatsclear <?> to clear errors and check for accumulation.
Cisco switch optical power check
Check the host Fibre Channel links: powermt display System logs: On AIX, view errpt; on Linux, view /var/adm/syslog/syslog.log
Storage inline check using 9b,a. Port errors will show numerous XX3E errors. Check FC, BKAD, STAT optical power levels.
Product series differences
DMX3 disk channel rate is 2GB; system bay battery is 078-000-054.
DMX4 disk channel rate is 4GB; system bay battery is 078-000-050.
DMX3 and DMX4 storage bay batteries are both 078-000-050/086.
The difference between DMX-1500, 2500, 3500, and 4500 lies in the number of DA cards, maximum supported drive count, memory size, etc. The 1500 supports 1 pair of DA cards; the 2500 supports 2 pairs of DA cards....
The DMX-950 FEBE card integrates FA and DA BA (back adapter) functions, whereas the FA and DA back adapters in DMX3 and DMX4 are independent. If slots 1&2 or 15&16 are both DD (DEAD), the FEBE card is likely failed.
Some spare parts require monitoring,
User account permissions
PSE permission: highest level; fails a bad FRU.
PE permission
CE permission: lowest level; replaces hot-spare drives.
Components
Rear cards (Back Adapter, located at the device rear, containing FA, DA, XCM)
FA: FC adapter (when replacing, select the script option replace director...)
Operations targeting FA: f0,ce,0 (FA enable); f0,ce,0f (FA disable). After disabling, it displays as '='.
DA: disk adapter. DA ports ABCD are arranged from bottom to top, connecting to the LCC. (When replacing, select the script option replace a director...)_)
Front-end cards (Director Adapter, located at the device front)
The component corresponding to the FA's DA on the device front is the Front-end card: Director Adapter. Sometimes the front-end card fails; if no LEDs display on the front, replacement is required. Replacing the front-end card requires strictly following a complex procedure.
XCM: Internal communication card controlling and communicating with all components, functioning as the device Switch. It has 2 redundant units.
DAE PS: After replacing the power supply, plug it back in within 2 minutes; do not insert it backwards.
SPS: When replacing the battery, keep your eyes away from the battery when pressing the switch. Use the menu Add/replace power/SPS.
LCC: Link Control Card; can connect up to 15 drives.
If DD is displayed, in 90% of cases the front-end card does not need replacement; this may be due to an LCC fault or multiple drive failures. In cases of complete failure, the front-end card is usually not visible at all.
Data flow path: Host -> FA -> Memory (read miss, read hit, write hit, write miss) -> DA-LCC -> Drive
Terminology
BCV Synchronization: Synchronization between the source volume and the target volume; for RAID5, the target volume is typically used.
First split the mirror, then create the meta (striped).
SRDF: Remote storage replication functionality.
./syscfg discover and ./symcfg list to check local and remote storage.
INLINE Check
DMX3/4 INLINE commands
Check all: 9b,a
Check drives: A7,nrwd; A7,hots, EB, E7
9b,a checks overall status; explanation of the content displayed below.
0473: Bad drive. Use A7,NRWD and A7,HOTS; after selecting the director, enter eb, e7.
4780: Power supply fault.
282C: Mismatch between memory data and drive data; this is difficult to resolve. Early detection of drive faults can prevent this.
Error colors represent different time periods.
If there are many errors with XX3E or XX2F, the system is unhealthy.
If many red and purple errors appear, attention is required. If the screen is filled with purple and red errors, a fault exists.
Double-click the error; after highlighting the error, press F8 to view the explanation.
Check drives
A7,NRWD, A7,hots: Normal output is...; if a bad drive exists, it shows .x. or x.. In columns containing 'x', the last entry displays the bad drive's location, e.g., C/08.
When a bad drive exists, the subsequent 'invalid' M1 and M2 invalid tracks showing numbers is normal; M3 displaying 0 is normal.
To view bad drives in RAID5, use a7,d5,ab.
In the output of A7,NRWD; A7,hots, you will see DV (Device Volume), such as 1E, 20, 2A, etc. Taking 20 as an example: consider a 146GB drive sliced into 20 segments. These segments are mirrored with corresponding segments from other drives. The 20th segment is mirrored, and the resulting mirrored DV name is 20.
Drive Replacement
In INLINE, select the DA and enter EB,2F,,,0311 to view the error timestamp.
a7,d5,<4b0>, where 4b0 is the content from the last column of EB,2F,,,0311.
a7,d,<4b0>
Commands starting with a7 check if the hot spare has fully taken over: invoke dynamic spare --> invaild track = 0. Then execute replace disk drive (requires the hot spare to have already taken over).
If 0311 executes invoke dynamic spare...
If the failed drive is a hot spare, execute replace spare disk.
Physical view can be used to locate the physical position of drives.
Drives not recommended for DMX: 101-000-111/112 (also 101-000-326, 101-000-052).