nvidia-smi Reports a Lost GPU: Persistence Mode Case

Reported symptom

The source reports a missing GPU or the following nvidia-smi error:

Unable to determine the device handle for GPU 000:05:00.0: GPU is lost .Reboot the system to recover this ....

Source recommendation

On its Linux deployment, the source recommends enabling persistence mode:

nvidia-smi -pm 1

The setting does not survive reboot. Its historical RHEL/CentOS startup example edits /etc/rc.local, adds the command and marks the file executable:

vim /etc/rc.local
# Add this line to the existing startup script:
nvidia-smi -pm 1

chmod +x /etc/rc.local

Use this only on a system where rc.local startup is supported and enabled; executable permissions alone do not establish that a modern distribution will run the file. NVIDIA also documents nvidia-persistenced and distribution-specific service integration.

What persistence changes

Persistence keeps GPU state initialized when no application client holds the device, reducing repeated initialization latency between CUDA jobs. It is not “locking application data into GPU memory.”

The source attributes its incident to repeated initialization, long PCIe configuration accesses and soft-lockup behavior, and recommends persistence for future deployments. That is its case explanation, not proof that every missing GPU has the same cause. GPU state deinitialization must also not be confused with unconditionally unloading the kernel driver after every application. A GPU that has already fallen off the bus may still require a supported reset/reboot and investigation of driver, PCIe, power or hardware faults.