NF5468M5 Repeated PCIe Faults: Distinguish Reboot and GPU-Load Failures

This is a case history, not a rule that a PCIe bus error identifies one defective FRU. The NF5468M5 had repeated restarts/PCIe events after several replacements. The source eventually distinguishes a reproducible reboot fault from a separate GPU-workload fault.

First incident: GPU-board replacement

Original English July2023SELexcerpt with PCIe bus fatal errors and reboot events

Out-of-band and black-box logs pointed to BDF 174/0/0 and 175/0/0. The source replaced the GPU board:

Original English black-box PCIe logs and topology with root busae highlighted

Second incident: recurrence and CPU1 suspicion

Original English July19PCIeuncorrectable events and July20undeterminedhardware-failure event

The next black-box evidence pointed to BDF 174/0/0 and 176/1/0. Because a GPU board had already been replaced and the source associated the path with CPU1, the engineer replaced CPU1:

Original English black-box buses176and174and associated PCIe topology

Bus numbers 174, 175, 176 correspond to hexadecimal AE, AF, B0, respectively. The topology and full error sequence are needed to interpret these paths; a CPU-related root-port path alone does not prove a faulty CPU.

Third incident: isolate the reboot fault

The source again observed bus 174/176 errors and brought a motherboard, GPU board and cables. It changed the motherboard-to-GPU-board E-port slimline cable, then F/G/H cables, without clearing the issue. CPU stress alone passed, but reboot still triggered PCIe errors. CPU swaps and cross-swapping GPUs between CPU domains did not clear it.

With the GPU module separated, a minimal motherboard configuration with one CPU and one DIMM still reproduced the failure. Cross-replacing CPU/memory did not remove it. Replacing the motherboard then passed repeated minimal-configuration reboots; restoring all components and repeating reboots/CPU Linpack also did not reproduce that reboot fault at that stage.

Fourth incident: a different GPU-load reproduction

The customer later reported another fault, now associated with a GPU path in OS evidence:

Original English APEI/AERPCIeerror excerpt highlightingdownstreamswitch0000:b0:14.0andvendor/device IDs

Original English topology highlighting theTeslaV100endpoint behind the switch path

Unlike earlier incidents, reboot testing now passed. The source’s GPU filediag diagnostic workload reproduced PCIe errors about five minutes after starting:

Original English diagnostic-load error photo with PCIe switch and endpoint records

After replacing GPUs and cross-testing slots, the source says the remaining failure followed the GPU board. It replaced that board, then reports 2.5 hours of GPU stress and 0.5 hour of CPU/memory stress without recurrence before returning the system for observation. These are bounded case-test results, not a guarantee of permanent repair or a universal stress duration.

Source conclusion

The reboot-time failure was associated with the motherboard in its minimal/cross-replacement tests; the later filediag-triggered failure was associated with the GPU board. Preserve each incident’s evidence and reproduction method rather than assuming every later PCIe event has the first incident’s cause. No supported diagnostic download or flashing/replacement procedure is supplied by this article.