30.2a Xid errors: NVIDIA's GPU error taxonomy — the complete Xid reference¶
When you're working with NVIDIA GPUs in an AI infrastructure, you will eventually encounter Xid errors. These are NVIDIA's standardized error codes that help engineers identify and diagnose hardware or software issues affecting GPU operations. Think of Xid errors as your GPU's way of saying, "Something went wrong — here's a clue about what."
This guide breaks down the Xid error taxonomy into simple, actionable categories so you can quickly understand what each error means and what to do about it.
🧠 What Are Xid Errors?¶
Xid errors are event messages generated by the NVIDIA GPU driver or firmware when an error condition occurs. Each Xid error has a unique numeric identifier (e.g., Xid 13, Xid 48, Xid 79) that maps to a specific type of failure.
- They appear in system logs (e.g., /var/log/messages, dmesg, or nvidia-smi output).
- They can be recoverable (GPU continues running) or fatal (GPU needs reset or replacement).
- They help engineers pinpoint whether the issue is hardware-related (e.g., memory, power) or software-related (e.g., driver, application).
⚙️ Common Xid Error Categories¶
Xid errors fall into several broad categories. Below is a simplified breakdown of the most frequent ones you'll encounter.
🛠️ Hardware Errors (GPU or System Level)¶
These indicate physical problems with the GPU, memory, or system components.
| Xid Code | Meaning | Typical Cause |
|---|---|---|
| 13 | GPU memory page fault | Corrupted memory or driver bug |
| 48 | Double bit ECC error | Uncorrectable memory error — GPU likely needs replacement |
| 62 | GPU has fallen off the bus | Power issue, loose connection, or hardware failure |
| 79 | GPU is stuck (hang) | Thermal throttling, power instability, or firmware bug |
| 94 | Contained ECC error | Correctable memory error — usually not critical, but monitor it |
📊 Software and Driver Errors¶
These point to issues in the driver stack, application, or configuration.
| Xid Code | Meaning | Typical Cause |
|---|---|---|
| 8 | GPU exception | Application crash or illegal instruction |
| 31 | GPU lost due to reset | Driver reset or power management event |
| 43 | GPU stopped processing | Driver timeout or application hang |
| 45 | GPU semaphore timeout | Synchronization issue between GPU and CPU |
| 64 | GPU fell off the bus (software) | Driver bug or PCIe link issue |
🕵️ Thermal and Power Errors¶
These relate to the GPU's environmental conditions.
| Xid Code | Meaning | Typical Cause |
|---|---|---|
| 28 | GPU temperature threshold exceeded | Overheating — check cooling and airflow |
| 32 | GPU power limit exceeded | Overclocking or insufficient power supply |
| 56 | GPU voltage fault | Power delivery problem on the board |
🔍 How to Detect Xid Errors¶
Xid errors appear in several places. Here's where to look:
- System logs: Run dmesg | grep -i xid to see recent Xid errors.
- nvidia-smi output: Use nvidia-smi -q -d ECC to check ECC error counts.
- NVIDIA persistence daemon: Logs are often captured in /var/log/nvidia-persistenced.log.
📤 Output: A typical Xid error line looks like this: NVRM: Xid (PCI:0000:03:00.0): 13, pid=12345, name=python3, GPU has fallen off the bus.
📊 Visual Representation: NVIDIA Kernel Driver Xid Logging¶
This diagram displays how the NVRM kernel driver detects internal hardware errors and logs numerical Xid events to syslog.
🛡️ What to Do When You See an Xid Error¶
Here's a simple triage process for new engineers:
- Identify the Xid code — Look at the number after "Xid" in the log message.
- Check if it's recoverable — If the GPU continues working, it's likely a transient issue. If the GPU is lost (e.g., Xid 62 or 79), you'll need to act.
- Inspect system health — Check temperatures, power, and PCIe link status using nvidia-smi.
- Review recent changes — Did you update drivers, change hardware, or modify power settings?
- Restart the GPU — For recoverable errors, try nvidia-smi -r (GPU reset) or reboot the system.
- Escalate if persistent — If the same Xid error repeats, it's likely a hardware fault. Contact your hardware vendor or NVIDIA support.
📋 Xid Error Reference Table (Complete)¶
Below is a more complete reference of Xid errors you might encounter. This is not exhaustive, but covers the most common ones for AI infrastructure.
| Xid Code | Category | Description | Severity |
|---|---|---|---|
| 1 | Software | GPU exception — illegal instruction | High |
| 2 | Software | GPU exception — invalid address | High |
| 8 | Software | GPU exception — application crash | Medium |
| 13 | Hardware | GPU memory page fault | High |
| 28 | Thermal | GPU temperature threshold exceeded | High |
| 31 | Software | GPU lost due to driver reset | Medium |
| 32 | Power | GPU power limit exceeded | Medium |
| 43 | Software | GPU stopped processing — timeout | Medium |
| 45 | Software | GPU semaphore timeout | Medium |
| 48 | Hardware | Double bit ECC error (uncorrectable) | Critical |
| 56 | Power | GPU voltage fault | High |
| 62 | Hardware | GPU has fallen off the bus | Critical |
| 64 | Software | GPU fell off the bus (driver) | High |
| 79 | Hardware | GPU is stuck (hang) | High |
| 94 | Hardware | Contained ECC error (correctable) | Low |
🧪 Best Practices for New Engineers¶
- Log everything — Always capture dmesg output and nvidia-smi logs before and after an error.
- Monitor ECC counts — Use nvidia-smi -q -d ECC to track correctable and uncorrectable errors over time.
- Don't ignore Xid 94 — While correctable, a rising count can indicate impending failure.
- Document patterns — If you see the same Xid error on multiple GPUs, it might be a system-level issue (power, cooling, PCIe).
- Use NVIDIA's official tools — The nvidia-bug-report.sh script collects all relevant logs for support cases.
✅ Summary¶
Xid errors are your GPU's diagnostic language. By learning the common codes and their meanings, you can quickly move from "something is wrong" to "here's what's wrong and what to do about it." Start by checking dmesg and nvidia-smi whenever you suspect a GPU issue, and use the reference table above to classify the severity.
As you gain experience, you'll develop an intuition for which errors are noise and which demand immediate action. For now, remember: Xid 48 and 62 are red alerts; Xid 94 is a yellow flag.