Inspur Server Log Analysis: RAID Controller, OS, and Black Box Logs

Log Content Description

RAID Card Log Analysis

RAID Card Log Analysis — Common RAID Card Models

Introduction to Common RAID Card Models

RAID Card Log Analysis — RAID Card Log Collection

Description of Common RAID Card Log Collection Scenarios

After collection, the following content is included:storcliAdpAllInfo: RAID card information; storcliPDList: Status of all current hard drives (regardless of whether an array is created)

storcliLdInfo: Current array configuration and status (does not display member drive list and status)

VallShowAll: Current array configuration and status; storcliLdPdInfo: Current array configuration, status, and member drive list and status.

MgEvtLog: Complete event log file; storcliAdpalilog: All log information. LdPdInfo, PdList, and LdInfo only save the configuration information at the moment of log collection; historical configuration information is not retained.

1) AdpAllInfo File Parsing — View RAID Card Information

2) PDList File Parsing -- View RAID Card Physical Hard Drive Information

3) LdInfo File Parsing -- View Array Configuration and Status Under the RAID Card

Note: Number of Spans indicates the number of spans contained in the array. A span can be viewed as a group of hard drives bundled together. When creating an array in WebBIOS, a SPAN is created first. When configuring Raid0, 1, 5, or 6, an array contains only 1 span; when creating Raid10, 50, or 60, an array will contain 2 spans.

4) VallShowAll File Parsing -- View Array Configuration and Status Under the RAID Card

5) LDPDInfo file parsing -- View array configuration status, physical disk status, and information

6) MgEvtLog file parsing – Event logs of the RAID controller During log analysis, our focus is primarily on state change records. Searching for 'state change' can locate status changes and timestamps regarding arrays and hard drives.

The following is a log record showing a hard drive failing and disconnecting, causing the array to degrade to a Degraded state.

PD 13(e0x16/s9) indicates the hard drive slot. PD 13 represents the hard drive with PD number 13, similar to the ID number in Adaptec RAID card logs. The actual slot number can be found in the LdPdInfo or PDList files based on the PD number. e0x16/s9 is an explanation for PD 13, i.e., the actual hard drive slot number; e0 denotes Enclosure 0 (Backplane 0),

s9 denotes Slot 9.

6) MgEvtLog file parsing

VD 01/1 represents Array 1, i.e., the array represented by Virtual Drive 1 in the LdPdInfo or LDInfo files.

In summary, the meaning of the example log is that the Slot 9 hard drive failed and disconnected, causing Array 1 to degrade to a Degraded state. This occurred at 09:10:06 on April 11, 2015. Based on this timestamp, review previous logs to identify the possible cause of the hard drive disconnection.

6) MgEvtLog file parsing Before a hard drive failure occurs, there is generally:

Unexpected sense; check the code table to confirm the error type.

Sense error codes consist of three digits: Sense Key, ASC, and ASCQ. In this example, the sense code is 3/11/00, indicating a Media error, which means the hard drive has media errors.

Common Sense Keys:

Key = 3: Indicates a Medium Error on the hard drive

• Key = 4: Indicates a Hardware Error on the hard drive

• These are generally hard drive failures themselves and require replacement.
• 6/29/00: Indicates Power on Reset, a restart command issued by the RAID controller to the hard drive upon power-on.

Generally not a hard drive issue. If all hard drives show this message, it may be a RAID controller fault. If only specific hard drives repeatedly show this code, it may be a backplane or cable fault.

You can search for errors related to hard drives or RAID controllers. Common ones include the following: Unrecoverable medium error: Hard drive media error.

Error on PD XX: Hard drive errors generally lead to hard drive disconnection. Specific causes can be determined by searching backwards to see if 'Unexpected sense' appeared.

Fatal firmware error: RAID controller firmware issue. For Inspur 2008 RAID controllers, disable OS power-saving options. For LSI 9271 RAID controllers, firmware errors appearing at the beginning of the logs are normal.

SAS Topology error: Link issues between the RAID controller and hard drives, usually caused by a faulty hard drive backplane or Expander card.

7) storcli Adpalilog file parsing

The file contains RAID controller information, physical hard disk information, logical array information, and RAID controller firmware log events.

After collecting PM8060 RAID controller logs, the following contents are included:

Important File Descriptions:

1. Config: Current RAID controller information, array configuration and status, and hard disk status (only configuration information at the moment of log collection is retained; historical configuration information is not retained)

2. Deaddrives: Hard disk error records

3. Device_log: Hard disk error records

4. SmartStats: Hard disk Smart information (this attribute exists only for SATA hard disks)

5. Logs: Complete log file

1) Config file parsing -- View RAID controller information

1) Config file parsing -- View array configuration and status

Note: In the array member slot information, Enclosure and slot represent the RAID controller channel number and hard disk slot number, respectively.

1) Config file parsing -- View hard disk status

Remark: Adaptec RAID cards record the slot position via the hard drive ID number in the logs. This ID number may differ from the hard drive's actual slot number (Slot No.). However, the slot can be confirmed by querying the Config file using the ID number. In the format shown above, 0,0(0:0), the first number outside the parentheses represents the channel number, and the second number represents the ID number.

2) Deaddrives File Analysis -- Hard Drive Failure Records Deaddrives: Hard Drive Failure Records

Records hard drive failure events. The term "failure" here refers to the hard drive being undetected (or considered removed). The example shows a complete failure record; the red box indicates the failure time (September 6, 2013, 17:49:58), and serialNumber denotes the hard drive serial number.

3) Device_log File Analysis -- Hard Drive Anomaly Records

Device_log: Hard Drive Anomaly Counters

A complete anomaly record is as follows:

<driveErrorEntry smartError="false" vendorID="HITACHI " serialNumber="XXXXXXXX" wwn="5000ccaXXXXXXXXX" deviceID="2" productID="HUS15606" numParityErrors="0" linkFailures="0" hwErrors="1" abortedCmds="0" mediumErrors=“3" smartWarning="0" />

• Where:
• ① serialNumber represents the hard drive serial number, and deviceID represents the hard drive ID number.
• ② hwErrors and mediumErrors counters indicate physical hard drive faults.

③ linkFailures and abortedCmds counters indicate transmission link issues, generally related to cables or backplanes.

• Note: Both the Deaddrives and Device_log files retain historical data. Even after a faulty hard drive is replaced, the anomaly records within these files are not cleared. Therefore, when anomalies are found in these files, first check whether the hard drive exists in the Config file using its SN or ID number.

4) SmartStats File Analysis -- View Hard Drive SMART Information

This file is in XML format and can still be opened with text editors such as UltraEdit. Each record begins with the disk's channel number and ID number, containing all SMART information for that hard drive.

5) Logs File Analysis – RAID Controller Event Logs

medium error, this error indicates that the hard drive has media errors, i.e., physical bad sectors. Generally, prior to reporting this error, there will also be other valuable related information. Below is a log screenshot related to medium errors.

This section contains a Sense error code. devt=0x12 indicates the ID number of the faulty hard drive; note that hexadecimal notation is used here. Converted to decimal, the ID number is 18. SenseKey=03 indicates a media error (for more information on Sense error codes, refer to the LSI RAID Card Log Analysis section).

This section provides further explanation for the medium error: ID(0:18:0) indicates the ID number of the faulty hard drive. Unrecovered Read Error indicates this is an unrecoverable read error. The last two lines represent the block address where the error occurred. Finally, locate the hard drive slot based on the reported hard drive ID number.

5) Logs File Analysis

bad strip, this error indicates that the array has bad stripes, which are logical bad blocks. This is equivalent to the array disks encountering a media error and requires deleting the array and recreating it to restore functionality.

container:1 indicates that Array 1 has the issue, corresponding to Logical device 1 in the Config file. Note: Fixing this issue requires deleting the array and recreating it. Be sure to back up data in advance!

CpuFatalError, RAID controller error; the RAID card needs to be replaced to resolve.

5) Logs File Parsing: ECC Error

This error indicates an ECC error in the RAID card cache; the RAID card needs to be replaced.

After collecting logs from the PM8204 RAID card, the following content is included.

Important File Descriptions:

1. Config: Current RAID card information, array configuration and status, hard drive status (only configuration information at the moment of log collection is gathered; historical configuration information is not retained).

2. SmartStats: Hard Drive SMART Information

3. Log: Complete log file, must be collected before rebooting. Low-version firmware cannot save previous logs.

4. ADUReport

PM8204 RAID card adureport: View RAID card information and array information.

PM8204 RAID card adureport: View physical hard drive information and sense key.

3008 SAS Card Log Analysis

3008 Card Log File: sashbalog, containing SAS card information, array information, physical hard drive information, etc.:

3008 Card Log File: sashbalog, containing SAS card information, array information, physical hard drive information, etc.:

RAID Card Log Analysis (Adaptec Case Study)

(1) Customer reported system crash. Collected RAID card logs for analysis and localization. Decompress the logs. Based on the system crash time, calculate backwards (generally, subtract 8 hours from the system time to get the RAID card time). Check the 'logs' file and search for keywords such as 'ecc', 'fatal'. An 'ECC error' message will be found here. Replace the RAID card to resolve the issue.

(2) After a RAID 5 array rebuild completes, partitions are lost. Analyze the cause.

This issue occurred after the RAID rebuild completed. Checking the logs revealed many 'bad strip [17352]: 04:00:27 CT_HashLookup: arrayNum=> 1 startBlkHi:Lo=> 0:fffffff8 alignedStrtBlk=> fffffc00' entries in the logs.

[17353]: 04:00:27 CT_HashLookup: curBse->blkHi:Lo 0:fffffc00 uid e34926ac [17354]: 04:00:27 CheckBst: Bad Stripe Found! container:1 BlkHi:Lo 0:fffffff8 [17355]: 04:00:27 CacheStrategy: IO to a bad stripe!
[17356]: 04:00:27 CacheStrategy: bp 810b9f60 container 1 blkHi:Lo 0:fffffff8 length 1000 [17357]: 04:00:27 returnError: ERROR bp=0x810B9F60 [17358]: 04:00:27 RawIoCallBack() Error - Container: 1, Block: -8, IoError: 5, f: 24004007, f2: 8440
[17359]: 04:01:53 R6_GetRaid6Options: REED_SOLOMON is SUPPORTED (and is the default)
Further inspection revealed bad sectors on multiple disks:

Replace the faulty hard drive and recreate the RAID. The system functions normally.

RAID Card Log Analysis (LSI Case Study)

(1) A customer's NF5460M4 server crashed. Collected RAID card logs for analysis and localization. The following error messages were found in the RAID card logs:

Confirmed RAID card errors above; replace the RAID card and upgrade the RAID card firmware. (2) A customer reported hard drive errors. Collected RAID card logs for analysis and localization. The hard drive error information is shown below; confirmed the faulty hard drive ID is 1d=29, slot 12. Replacing the hard drive resolved the issue.

2) A customer reported multiple hard drive failures. Collected RAID card logs for analysis and localization. The RAID card logs showed backplane anomalies, as follows:

Confirmed anomaly on backplane ID 10 above; replacing the backplane resolved the issue.

Operating System Log Analysis

Windows System — System Log Collection

Files Required:system.evt/application.evtCollection Purpose: Identify causes of abnormal events such as unexpected operating system shutdowns.

Collection Method: It is recommended to use Emcgrab (which can convert evt/evtx format logs to txt format for easier viewing).

Windows System — System Log Analysis Method

Recommended to usenotepad++ View txt format log files:http://notepad-plus-plus.org/zh/

The introduction uses logs collected by Emcgrab as an example. The log files collected by Emcgrab are compressed archives named after the server hostname, with the main logs saved in the HOST directory.

Log Format:

Common Log Codes:

• 6008: The last OS shutdown was unexpected; this information is recorded after the system restarts.
• By searching for this code, you can determine the time points and intervals of unexpected server shutdowns.

10/08/2013 03:47:27 PM 关键 WIN-XXXXXXXXXXX 41 Microsoft-Windows-Kernel- Power 系统在未首先正常关机的情况下重新启动。当系统停止响应、出现故障或意外断电时，会发生此错误。
10/08/2013 03:47:43 PM 错误 WIN-XXXXXXXXXXX 1001 Microsoft-Windows-WER- SystemErro 计算机已经从检测错误后重新启动。检测错误: 0x00000050 (0xfffff8800e278c08, 0x0000000000000000, 0xfffff88006d8083f, 0x0000000000000000)。已将转储的数据保存在: C:\Windows\Minidump\100813-27721-01.dmp。报告ID: 100813-27721-01。10/08/2013 03:47:43 PM 错误 WIN-XXXXXXXXXXX 6008 EventLog 上一次系统的15:38:01 在?2013/?10/?8 上的关闭是意外的。
WIN-XXXXXXXXXXX 6008 EventLog 上一次

Common Log Codes:

• 333: I/O operations initiated from the registry failed to recover. The registry could not read, write, or flush one of the files containing the registry system image.

This issue may occur when non-paged memory or the page buffer pool memory is temporarily insufficient. The system continues to record this log until the computer restarts.

• 2019: Non-paged buffer pool is empty

Since the non-paged buffer pool is empty, the server cannot allocate memory from it.

• 2020: Page buffer pool is empty
• Since the page buffer pool is empty, the server cannot allocate memory from it.

Note: All three errors above are OS/software issues, generally caused by software memory leaks that exhaust system resources, resulting in slow performance or freezing.

Page Buffer Pool (Paged Pool): Data in this section can be saved to the page file (pagefile)—virtual memory.

Non-Paged Buffer Pool (Nonpaged Pool): Memory in this section resides in physical memory and cannot be saved to virtual memory. It is primarily used by the system kernel and hardware drivers. For 32-bit Windows 2003, the maximum allocation is 256MB; for 64-bit, it is 128GB. For 32-bit Windows 2008, the maximum is 2GB; for 64-bit, it is 128GB.

Windows 2008 System Log Records Memory Errors

• If the Windows 2008 system log records memory errors and the server has a hang fault, it is generally due to memory hardware errors, requiring analysis combined with BMC logs.
• It may also be accompanied by a blue screen error 0x00000124.

Refer to the attached case: Microsoft-Windows-WHEA-Logger.TXT

08/01/2013 04:12:48 PM 错误 6008 EventLog 上一次系统的15:59:56 在2013/8/1 上的关闭是意外的。08/01/2013 04:11:40 PM 信息 12 Microsoft-Windows-Kernel-General 操作系统已在系统时间 2013-08-01T08:11:39.610798400Z 08/01/2013 03:59:38 PM 警告 47 Microsoft-Windows-WHEA-Logger 出现已更正的硬件错误。 组件: 内存错误源08/01/2013 03:59:38 PM 警告 47 Microsoft-Windows-WHEA-Logger 出现已更正的硬件错误。 组件: 内存错误源08/01/2013 03:58:38 PM 警告 47 Microsoft-Windows-WHEA-Logger 出现已更正的硬件错误。 组件: 内存错误源08/01/2013 03:58:38 PM 警告 47 Microsoft-Windows-WHEA-Logger 出现已更正的硬件错误。 组件: 内存错误源08/01/2013 03:57:38 PM 警告 47 Microsoft-Windows-WHEA-Logger 出现已更正的硬件错误。 组件: 内存错误源08/01/2013 03:57:38 PM 警告 47 Microsoft-Windows-WHEA-Logger 出现已更正的硬件错误。 组件: 内存错误源
2013-08-01T08:11:39.610798400Z 启动。组件: 内存错误源: 已更正的计算机检查组件: 内存错误源: 已更正的计算机检查组件: 内存错误源: 已更正的计算机检查组件: 内存错误源: 已更正的计算机检查组件: 内存错误源: 已更正的计算机检查组件: 内存错误源: 已更正的计算机检查

Windows System — Dump Files

Open Windows system settings: Recommendation: Disable automatic restart and change 'Write debugging information' to 'Small memory dump'.

Dump File Save Location:

1. When set to Small Memory Dump: C:\Windows\Minidump

2. When set to Kernel Memory Dump: C:\Windows\MEMORY.DMP
Although Emcgrab can also collect a memory_dump file, the parsing is relatively simple, and the dump file still needs to be manually copied.

Dump Analysis Methods

Tool windbg:

After installation, you need to add environment variables by adding a system variable: Variable Name:_NT_SYMBOL_PATHVariable Value:

srv*F:/Symbols*http://msdl.microsoft.com/download/symbols

Common Blue Screen Codes:

• 0x9C/x124, generally indicating hardware failure—

MACHINE_CHECK_EXCEPTION, similar to MCE errors under the Linux system. You need to combine BMC logs to check for memory errors, etc.

• 0x50/0x8E, blue screen caused by Windows 2003 bugs.
• Other Blue Screen Code Queries:http://msdn.microsoft.com/en-us/library/windows/hardware/ff559323(v=vs.85).aspx

Winmsd Logs

Can view major hardware and system information

Linux Logs

Important File Descriptions:

Message: Operating system log Cronlog: Scheduled task logs Dmesg: System boot logs Mcelog: Machine Check Exception logs last_reboot: System reboot records Securelog: Information regarding authorization verification

(1) The last_reboot log records restart time and other information, as follows:

(2) The cron log records the execution logs of some scheduled tasks, as follows:

(3) Securelog generally records authorization, verification, and other log information, as follows:

(4) dmesg records some log information during the Linux system boot process, as follows:

(5) mcelog (machine check exception) records hardware error logs. When hardware errors are detected (memory errors, CPU errors, CPU temperature too high), recording is triggered.

(6) The message log records various event log information of the system, as follows:

(6) Several error cases under message: Abnormal call trace information is as follows:

Raid card driver error:

BMC, Black Box, and Register Log Analysis

BMC SEL Event Logs

BMC Log Analysis:

BMC logs are divided into three levels: Information, Warning, and Critical:

① Normal Information Mainly used to view machine shutdown, crash operations, and time records.

② Warning Logs These are logs that require attention. You need to check the specific issues in the logs, observe if they can be fixed, and replace spare parts promptly if necessary.

③ Critical Logs These are errors that may cause system crashes or other serious issues. Spare parts must be replaced to resolve them.

BMC Sensor Status

Sensor status mainly includes the following status values:

• ok: Indicates the sensor is functioning normally.
• nc: Indicates the sensor has detected a minor alert.
• cr: Indicates the sensor has detected a critical alert.
• nr: Indicates the sensor has detected an urgent alert.
• na: Indicates the sensor has not detected any value or status currently; the device corresponding to this sensor may not be present.

Blackbox Logs

Blackbox Log Analysis Tools and Methods:

1. The analysis of Blackbox.log is consistent with BMC log analysis. Log introduction:

2. Analysis of Blackboxpeci.log requires the use of the Mce_decoder tool or consulting documentation for analysis.

Parsing Tools:(The blackbox logs collected by the Inspur log collection tool have already been parsed.)

Analysis Tools:

Blackbox logs are parsed as follows using parsing tools:

Blackboxpeci Log:

Blackboxpeci.log obtains register values through log parsing. For non-zero abnormal register values in the status field, use mcedecode as follows:

Blackboxpeci Log:

Explanation of Mca_err_src_log Register Parsing:

The parsing result for CPU0 is: 00010100 00011000 0000000000000000. The parsing result for CPU1 is: 10100000 11000000 0000000000000000

Register Logs:

M5 Platform CPU IERR Issue:

The occurrence of CPU IERR on the M5 platform may be caused by multiple factors, including 3Strike-Timeout, Nested MCERR, PCU error, SMI & MCERR racing, internal CPU fuse errors, etc. This is typically not due to a CPU failure itself but may be related to faults in memory, PCIe devices, motherboards, or other components. Location must be determined through analysis of blackbox logs and CPU register values.

M5 Platform CPU IERR Issue: You need to parse the blackboxpeci.log and MSR register values. For common memory-related error reports, the involved banks are as follows; each CPU contains two memory controllers.

M5 Platform CPU IERR Issue:

09/03/2019 16:23:37 CPU0_Proc0 IA32_MC14_STATUS 57 01 00 40 91 00 01 01 00 16 00 dc

Analysis of the CPU0 IMC0 Channel 1 memory error indicates that replacing the memory module is required to resolve the issue.

M4 Platform Memory Correspondence:

Images localized into English for illustration; labels may vary by product version.