Oracle Exadata Daily Maintenance: Emergency Response Plan

Exadata Daily Maintenance Manual

Emergency Response Plan

Exadata Emergency Response Personnel

Fault Scenario One

Scenario Description

A hard drive on the Exadata Appliance Data Server (DB server) has failed.

Handling Steps

Since the Exadata Appliance itself has RAID protection, the hard drive can be replaced directly without shutting down the system. No shutdown operation is required.

Check disk status information via the raid command

/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL | grep "Device Present" -A 8

Use the machine's iLOM port to further diagnose the data server failure and determine the location of the faulty drive.

After collecting all information, log in to support.oracle.com to submit a service request for the Exadata Appliance, requesting maintenance services for the Exadata Appliance.

Oracle support personnel will replace the faulty hard drive based on the relevant information.

Result Verification

After replacement, use the following command to check if the status of the replaced hard drive is normal.

/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL | grep "Device Present" -A 8

Use the following command to check the synchronization status of the data devices

/opt/MegaRAID/MegaCli/MegaCli64 -pdrbld -showprog -physdrv \

[disk_enclosure:slot_number] -a0

Fault Scenario Two

Scenario Description

The Exadata Appliance PDU or InfiniBand switch has a single fault

Handling Steps

Since the equipment adopts a dual-redundancy structure, a failure of a single PDU or InfiniBand switch will not affect the operation of the Exadata Appliance.

Monitor Exadata using EM 13C software; if an alarm occurs for the PDU or InfiniBand switch, collect fault information.

After collecting all information, log in to support.oracle.com to submit a service request for the Exadata Appliance, requesting maintenance services for the Exadata Appliance.

Based on the fault situation, Oracle support personnel will configure or replace the faulty PDU or InfiniBand switch.

Result Verification

Monitor Exadata using EM 13C software; after the fault is resolved, alarms for the PDU or InfiniBand switch will be automatically cleared.

Fault Scenario Three

Scenario Description

A hard drive on the Exadata Appliance Storage Server (Cell Server) has failed.

Handling Steps

Monitor Exadata using EM 13C software. After detecting an alarm for the Exadata Storage Server (Cell Server), use the following command to check the hard drive fault status.

CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status=failed DETAIL

name: 28:5

deviceId: 21

diskType: HardDisk

enclosureDeviceId: 28

errMediaCount: 0

errOtherCount: 0

foreignState: false

luns: 0_5

makeModel: "SEAGATE ST360057SSUN600G"

physicalFirmware: 0705

physicalInterface: sas

physicalSerial: A01BC2

physicalSize: 558.9109999993816G

slotNumber: 5

status: failed

Go to the faulty server, locate the failed hard drive, retrieve a spare hard drive from the warehouse, and prepare to replace the faulty hard drive.

Remove the faulty hard drive, wait for a few minutes, then insert the new hard drive into the corresponding slot.

Use the following command to check and confirm that the hard drive status is Normal.

CellCLI> LIST PHYSICALDISK WHERE name=[enclosureDeviceId:slotNumber] ATTRIBUTES status

Result Verification

Server hard drive alarm cleared, EM13C monitoring software fault cleared

Use relevant commands to check and confirm that the hard drive status is Normal.

Fault Scenario 4

Scenario Description

A flash card failure occurs on an Exadata Machine Storage Server (Cell Server).

Handling Steps

After monitoring the Exadata system with EM 13C software and detecting a fault alert for the Exadata Storage Server (Cell Server), check the disk fault status using the following command:

CellCLI> LIST PHYSICALDISK WHERE diskType=flashdisk AND status=failed DETAIL

Put all grid disks into inactive state

CELLCLI> Alter GRIDDISK ALL INACTIVE

Verify that all grid disks are in inactive state

CELLCLI> LIST GRIDDISK WHERE STATUS !=’inactive’

Stop Cell services

CellCLI> ALTER CELL SHUTDOWN SERVICES ALL

Shut down the faulty storage server

Retrieve a spare flash card from the warehouse to prepare for replacing the faulty flash card.

After replacement, restart the Exadata Storage Server (Cell Server).

Put all grid disks into active state

CELLCLI> ALTER GRIDDISK ALL ACTIVE

Verify that all grid disks are in active state

CELLCLI> LIST GRIDDISK ATTRIBUTES name , asmmodestatus

After restart, the new Flash card will be automatically recognized and used by the system.

Result Verification

Server disk alerts are cleared; EM13C monitoring software fault is resolved.

Fault Scenario 4

Scenario Description

Exadata Machine database query/update anomalies occur, such as slow query/update speeds or freezing.

Handling Steps

Monitor the Exadata system operation using EM13C software to identify data processing bottlenecks.

Collect AWR reports, submit them to the Oracle ACS team for analysis, and quickly resolve the fault.

Result Verification

Exadata Machine query/update operations return to normal.