GPU ECC Errors: Page Retirement, Row Remapping and Source Thresholds

This is a translation of the stored ECC service note. Its 5 / 10 / 60 and SRAM >4 thresholds are historical source criteria, not a universal or current NVIDIA warranty commitment. Obtain the GPU/system vendor’s applicable diagnostics and RMA decision for the actual architecture and contract.

1. Inspect the reported condition

An application can fail with an ECC error while related GPU/kernel logs record the event. ECC means error-correcting code. The source uses a Linux system with the NVIDIA driver installed.

# Inspect one GPU; substitute its ID
nvidia-smi -i <target gpu> -q -d PAGE_RETIREMENT

Retired pages
    Single Bit ECC             : 2
    Double Bit ECC             : 0
    Pending                    : Yes

# Inspect all GPUs
nvidia-smi -q -d PAGE_RETIREMENT

# Collect the NVIDIA report
nvidia-bug-report.sh

nvidia-smi -q | grep -i 'bit ecc'

Page counts are not raw error-event counts. The PAGE_RETIREMENT fields count retired pages by cause; nvidia-smi -q -d ECC reports a different set of ECC counters. Do not mix them when interpreting a threshold.

In the source’s legacy page-retirement workflow, Pending: Yes indicates retirement awaiting activation through the supported reset/reboot procedure; No means no retirement is pending. It does not prove the GPU has no faults. Retiring a page prevents its future allocation once applied; it does not guarantee that every subsequent workload is unaffected.

The source also queries retirement cause and timestamp to assess a 30-day period. Its broken multi-line placeholder is normalized below; this query does not itself filter the last 30 days:

nvidia-smi -i <GPU_ID> --query-retired-pages=gpu_bus_id,gpu_serial,retired_pages.cause,retired_pages.timestamp --format=csv

2. Thresholds listed in the source

• On the same GPU, Double Bit ECC ≥5 within 30 days.
• Double Bit ECC ≥10 during the warranty period.
• Double Bit ECC + Single Bit ECC ≥60 during the warranty period.
• Field Diagnostics result FAIL.

The source treats any one of these as replacement grounds. Confirm which reported metric and policy applies with support; do not automatically apply the figures to every architecture or raw volatile/aggregate counter.

For cases below its thresholds, the note uses OS reboot or nvidia-smi -r to activate pending retirement. Reset requires the appropriate privileges and supported platform state, with dependent applications stopped; without a selected device the command can affect all GPUs. Recheck the state afterward. This is not a universal “reset resolves all ECC” instruction.

3. Source rationale and recommendation

The original has an empty “NVIDIA RMA reference” heading, so it supplies no linked policy proving the figures. It recommends applying its ECC workflow to reported ECC incidents. The English version keeps that context while directing current diagnosis and eligibility to the vendor.

4. Common fault and architecture-specific table

Common Faults	Diagnosis Criteria	Recommended Replacement Standards
lspci card drop	lspci | grep -i nvidia | grep -iE '3d con|vga com' Number of recognized GPUs does not match the actual number	No separate replacement criterion supplied in the source.
nvidia-smi GPU drop	nvidia-smi Number of recognized GPUs does not match the actual number	No separate replacement criterion supplied in the source.
GPU bandwidth reduction	Compare actual PCIe link speed/width with the model’s expected link using lspci -vvd 10de: | grep -i Lnksta:. The source expects the rated link except for model-specific exceptions; account for idle power/link states before diagnosing a degraded link.	No separate replacement criterion supplied in the source.
ERR! or Unknown Error reporting	ERR! error appears in nvidia-smi output; Unknown Error appears in nvidia-bug-report logs Recommendation: This is often caused by GPU driver bugs; it is recommended to upgrade the GPU driver first for testing.	No separate replacement criterion supplied in the source.
GPU infoROM is corrupted	Error reported in nvidia-smi output or WARNING: infoROM is corrupted at gpu 0000:xx.00.0 in nvidia-bug-report logs Recommendation: Request the vendor to fix or provide a fix method. If the fix fails, replace the card.	No separate replacement criterion supplied in the source.
GPU NVLink link appears as inactive	NVLink-enabled models: nvidia-smi nvlink -s Command shows link as inactive	No separate replacement criterion supplied in the source.
Ampere architecture GPU ECC error reporting	Standard One: Command: nvidia-smi -q -d row_remapper Remapped Rows    Correctable Error              : 0    Uncorrectable Error            : 0    Pending                         : Yes    Remapping Failure Occurred    : Yes Standard Two: fieldiag test failed Standard Three: SRAM Uncorrectable error >4	Historical source guidance (confirm applicability with the GPU/system vendor): Pending status set to Yes indicates that a reboot of the OS is required to mask the error address. Remapping Failure Occurred status set to Yes indicates that masking the error address has failed. If either field remains Yes after rebooting the OS, replace the GPU directly. If both fields are No after rebooting the OS, it is recommended to perform a fieldiag test to confirm whether the GPU needs to be replaced.Note: GPUs have an automatic function to mask ECC error addresses. For ECC errors, it is recommended to reboot the OS first and check if the error meets RMA standards. If the standard is not met, the system can be used for observation.
Other Tesla GPU ECC error reporting	Standard One: Command: nvidia-smi -q -d page_retirement Retired Pages     Single Bit ECC                : 0     Double Bit ECC                : 0     Pending Page Blacklist        : No Standard Two: fieldiag test failed	Historical source guidance (confirm applicability with the GPU/system vendor): The following four conditions, if any one is met, allow direct GPU replacement: Double Bit ECC ≥ 5 on the same GPU within 30 days; Double Bit ECC ≥ 10 during the warranty period; Double Bit ECC + Single Bit ECC ≥ 60 during the warranty period Pending Page Blacklist flag is Yes after rebooting the OS Any one of the above conditions allows for GPU replacement. If ECC count criteria are not met, perform a fieldiag test.Note: GPUs have an automatic function to mask ECC error addresses. For ECC errors, it is recommended to reboot the OS first and check if the error meets RMA standards. If the standard is not met, the system can be used for observation.
Other ECC error counting related items	Command: nvidia-smi -q -d ecc     ECC Errors         Volatile             SRAM Correctable              : 0             SRAM Uncorrectable            : 0             DRAM Correctable              : 0             DRAM Uncorrectable            : 0         Aggregate             SRAM Correctable              : 0             SRAM Uncorrectable            : 0             DRAM Correctable              : 0             DRAM Uncorrectable            : 0	The source does not use these ECC counters alone as its replacement rule. They remain diagnostic evidence. It suggests Field Diagnostics when SRAM uncorrectable errors persist even if its numeric criteria are not met; use the applicable vendor policy.

The table retains the Ampere source fields Pending, Remapping Failure Occurred, SRAM Uncorrectable >4 and its post-reboot follow-up; the Tesla row retains the four listed replacement triggers, including a pending blacklist after reboot. Check actual architecture support and current vendor policy before interpreting these as a replacement decision. Row-remapping and older page-retirement mechanisms are distinct.

Current references: Dynamic Page Retirement FAQ, A100 memory-error management and NVIDIA GPU triage.