Lenovo ThinkSystem FPGA Log Analysis and Fault Localization

This document is intended for Level 2 server hardware support engineers and on-site technical experts. It systematically explains how to deeply analyze onboard FPGA registers via FFDC logs in Lenovo ThinkSystem servers (Purley / Whitley platforms, such as SR650, SR550, SN550, etc.) to precisely locate Motherboard voltage faults (SysBrd Vol Fault / FQXSPPW0063M), Abnormal power loss (Power Fault) and CPU/memory power supply faults.

1. Why is FPGA log analysis required?

In the Lenovo ThinkSystem server architecture, XCC (BMC) is responsible for macro-level out-of-band monitoring and upper-layer alarms, while low-level hardware power-on sequence control, microsecond-level power protection, and hardware fault latching are fully managed by the onboard FPGA / CPLD exclusively.

• XCC Alarm Limitations: When a power supply abnormality occurs, the XCC System Event Log (SEL/CXL) typically reports only one generalized error:

  Sensor SysBrd Vol Fault has transitioned to critical from a less severe state. (FQXSPPW0063M) This error merely indicates "motherboard voltage fault" and cannot specify whether the issue lies with 12V, 5V, 3.3V, or which specific CPU power rail is abnormal. Blindly replacing the motherboard often leads to repair failure.

• FPGA Register Advantages: In the event of power drop, undervoltage, overcurrent, or short circuit on a power rail, microsecond instant, the FPGA will permanently latch the First Capture into hardware registers before forcibly shutting down system power (Power Fault Trip). This data persists until the next AC power cycle reset.

2. Distribution Path of FPGA Logs within FFDC

After extracting the exported FFDC service package from the server (extension name .tzz, actually in tar.gz format), the core FPGA logs are located at the following path:

2.1 Preferred File: Automatically Decoded Log

• File Path: tmp/ffdc_live_dbg/fpga_on_demand.log
• Description: The FFDC collection script (firetool) automatically invokes the register decoding tool to generate a report containing highly readable bit parsing (e.g., directly indicating PVCCIO CPU1 fault).

2.2 Alternative File: Raw Register Dump

When fpga_on_demand.log is not generated or the decoding tool version does not match, the following raw data must be examined:

• File Path: var/volatile/log/fpga_powerfault.log or var/volatile/log/fpga.log
• Description: Records hexadecimal raw bytes (Hex Dump) for each register block (Block 00 ~ Block 63).

2.3 Signal Mapping Database (Cross-Reference)

• File Path: tmp/dm.log
• Description: XCC Data Manager signal pin mapping table, including PWRGD_PVCCIO_CPU0, THERMAL_CPU1_VR_PVCCIN and the physical binding of signals to hardware components.

3. Comprehensive Overview of FPGA Register Architecture (Block Structure)

Inside the FPGA, Blocks (each block typically 16 bytes) as units:

4. Block 04 Core Decoding and Lookup Table Quick Reference (Most Critical Section)

When the system reports SysBrd Vol Fault or powers off immediately upon boot, focus must be on Block 04.

4.1 Raw Data Format Identification

A typical Block 04 data line is as follows:

Block 04 - 7F 00 00 00 02 00 00 00 - 00 01 07 10 80 00 20 00

Offset and function for each byte:

[!TIP] > Under normal conditions, POWER_FAULT0 to POWER_FAULT7(i.e., Byte 1 to Byte 8)Must all be 00. If any single byte is non- 00, it indicates that the power rail corresponding to that byte triggered a protection action.

4.2 POWER_FAULT Detailed Bit Definition Table

Depending on the FPGA firmware version, register definitions have minor adjustments (using SN550_FPGA_PowerFaults1.0 and the SR650 Cyborg platform general definition as examples):

(1) POWER_FAULT4 (Offset 0x044 / Offset 4h) —— CPU core voltage rails

This is the byte where faults most frequently occur; the hexadecimal number must be converted to an 8-bit binary (Bit 7 ~ Bit 0):

• Practical Case Correspondence:
• If the register value is 0x10(binary 0001 0000): Bit 4 set to 1, corresponding to PVCCIN CPU1 fault.
• If the register value is 0x02(binary 0000 0010): Bit 1 set to 1, corresponding to PVCCIO CPU1 fault.

(2) POWER_FAULT5 (Offset 0x043 / Offset 3h) —— Memory and VCCIO

(3) POWER_FAULT6 (Offset 0x042 / Offset 2h) —— Motherboard main voltage and VPP

(4) POWER_FAULT7 (Offset 0x041 / Offset 1h) —— Standby auxiliary and input supply

5. Core determination rule: Is it a motherboard fault or a CPU fault?

This is the most typical dispute scenario faced by second-line technical support. The troubleshooting criteria are as follows:

flowchart TD
    A["SysBrd Vol Fault (FQXSPPW0063M) occurs"] --> B["Extract the FFDC logs and open Block 04"]
    B --> C{"Which byte reports the fault?"}
    
    C -->|"POWER_FAULT7 (0x041)<br>or 12V EFUSE"| D["Likely system-board power fault or peripheral short circuit<br>(backplane, Riser, fan, power backplane)"]
    C -->|"POWER_FAULT6 (0x042)<br>3.3V / 5V"| E["System-board power chip or onboard component short circuit<br>(replace the system board)"]
    C -->|"POWER_FAULT4 / 5<br>(PVCCIN / PVCCIO / VCCSA)"| F["CPU core power rail fault"]
    
    F --> G{"Does the same CPU fault persist after replacing the system board?"}
    G -->|"Yes (same CPU identified before and after replacement)"| H["100% points to CPU damage or poor socket contact<br>(request CPU replacement)"]
    G -->|"No (first service request)"| I["Onsite cross-testing plan"]
    
    I --> J["Minimal boot test with one CPU"]
    I --> K["Swap CPU 1 and CPU 2"]
    
    K --> L{"Does the fault register move?"}
    L -->|"Yes (fault follows the CPU to the other socket)"| M["CPU internal breakdown short circuit confirmed<br>(replace CPU)"]
    L -->|"No (fault stays in the same physical socket)"| N["System-board VRM circuit damage or damaged socket pins<br>(replace the system board)"]

Key heuristic rule

1. Power generation and consumption relationship:

• Motherboard VRM is Power generator;
• CPU internal transistors are Power consumption load.
• When an avalanche breakdown short circuit occurs in the CPU internal core or I/O plane, the VRM attempting to output voltage will immediately trigger overcurrent (OCP) or undervoltage (UVP), resulting in Power Good unable to output.

1. Law of persistent fault after board replacement:

• If a brand new motherboard is replaced but the FPGA fault remains reported on the same CPU (e.g., reporting CPU1 PVCCIN before replacement and CPU1 PVCCIO after replacement), and the power-off source is both 0x80, Motherboard issues can be absolutely ruled out.. Since the probability of two motherboards sharing the same power circuit failing simultaneously is extremely low, the only constant variable is the reused legacy CPU.

1. Precautions for Physical CPU Number Mapping:

• At the FPGA low level: CPU0 Corresponds to physical slot CPU 1 (Boot CPU / Socket 0); CPU1 Corresponds to physical slot CPU 2 (Socket 1).
• When on-site personnel cannot confirm 100% the tool's decoding correspondence, the fastest and most secure operation is Remove the second CPU to boot with a single-CPU minimum configuration or Swap the two CPUs to verify.

6. Appendix: Lenovo ThinkSystem Platform Code Name Quick Reference Table

In Block 00 's 0x000 register, the platform Chip ID can be read to confirm the model: