GPU Node Troubleshooting: nvidia-smi Results and Xid Events

The source uses an inspection result named NVIDIASMIStatusCode. The table combines native nvidia-smi return codes with its own -1 timeout marker. Return codes and Xid events are different namespaces.

nvidia-smi status

nvidia-smi Status Codes	Description	Solution
0	Operation succeeded.	No action for a successful query.
3	Requested operation is not available on the target device.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
6	The query did not find the requested object.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
8	External device power cables are not correctly attached.	Check cabling with support using the model service procedure.
9	NVIDIA driver is not loaded.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
10	The NVIDIA kernel driver detected a GPU interrupt issue.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
12	NVML shared library could not be found or loaded.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
13	The local NVML version does not implement the requested function.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
14	infoROM is reported corrupt.	Collect the full report and contact support; confirm the actual fault.
15	The GPU has fallen off the bus or is otherwise inaccessible.	Collect the full report and contact support; confirm the actual fault.
255	Other/internal driver error.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.
-1	Inspection-tool timeout marker in the source; not a documented native nvidia-smi return code.	Inspect the installer log, kernel messages and related Xid events; confirm the installed driver/NVML stack.

Source collection paths/commands:

/var/log/nvidia-installer.log
dmesg | grep -i nv

The installer log may be absent with other package methods. Preserve full kernel/Xid context instead of relying only on a filtered snippet. Code13 is updated to the documented missing-NVML-function meaning, which is more precise than assuming every occurrence is a driver/library version mismatch.

Xid events for initial investigation

Xid messages are driver event reports in OS logs. The source first suggests retrying a failed workload when appropriate, then reviewing application code/logs if the error returns. An empty inspection field means no events were captured by that check, not proof that no GPU issue exists.

XID	Description
13	Graphics engine exception. Investigate the application, CUDA behavior, driver and hardware as appropriate; this code alone does not establish the cause.
31	GPU memory page fault / address-access problem. Investigate the application and platform using the applicable Xid catalog; do not rule out driver or hardware solely from this number.
43	GPU stopped processing. The source directs initial attention to the application; use the event context to confirm the cause.
45	Preemptive channel cleanup after termination or another failure. Often secondary/informational; inspect earlier errors to find the cause.
68	NVDEC0 exception in the source’s catalog; investigate the applicable decoder, driver and hardware context.

Collect evidence for support

The source escalates the following events with the complete GPU-node diagnostic data:

XID	Description
32	Invalid/corrupt push-buffer stream. Investigate communication, PCIe and driver state; the source’s PCIe diagnosis is not conclusive by itself.
38	Driver firmware error in the source’s legacy catalog. Collect the matching driver/firmware context.
48	Uncorrectable double-bit ECC event, also reported to the affected application. Follow the architecture-specific drain, reset/reboot and diagnostic procedure.
61	Internal microcontroller breakpoint/warning; collect evidence and follow the applicable recovery procedure.
62	Internal microcontroller halt; similar investigation to61.
63	Successful recording of ECC page retirement (Volta/older) or row remapping (Ampere). Activation/recovery depends on the architecture and accompanying events.
64	Failure to record page retirement or row remapping. Escalate using the matching memory-error recovery guidance.
74	NVLink error. Inspect the event details and link connections; a code alone is not an automatic GPU-replacement diagnosis.
79	GPU has fallen off the bus. Drain affected work and collect PCIe/power/driver evidence for support; confirm the actual failing component.
92	High single-bit ECC error rate; gather diagnostics and applicable memory-error information.
94	Contained ECC event on supported architectures: the affected application must recover; other applications may remain unaffected when containment succeeds.
95	Uncontained ECC event on supported architectures: the failure is not confined to one application; follow the matching GPU/node recovery procedure.

These lists preserve every source code. Descriptions and recovery differ by GPU architecture and driver release. The source’s absolute hardware-failure claims for Xid74/79 and very-low-probability hardware claims for Xid13/31 have been qualified against NVIDIA’s current documentation. Use the matching Xid catalog and GPU triage guidance for the actual system; do not reset a GPU still running dependent workloads.