31.3d Alerting rules: Prometheus alertmanager for GPU fault notifications¶

📘 Context Introduction¶

When you run AI workloads across many GPU servers, you need to know immediately when a GPU starts failing. Waiting for users to report problems is too slow — by then, training jobs may have already crashed or produced corrupted results. This is where Prometheus Alertmanager comes in.

Alertmanager takes the raw metrics collected by Prometheus (like GPU temperature, memory errors, or power issues) and turns them into notifications. When a GPU fault is detected, Alertmanager can send alerts via email, Slack, PagerDuty, or other channels so your team can respond quickly.

This guide will help you understand how to set up alerting rules specifically for GPU fault notifications using Prometheus and Alertmanager.

⚙️ How Prometheus Alertmanager Works with GPU Metrics¶

The flow for GPU fault alerting follows this simple pipeline:

• DCGM (NVIDIA Data Center GPU Manager) collects GPU health metrics (temperature, memory errors, power draw, etc.)
• DCGM Exporter exposes these metrics in a format Prometheus can scrape
• Prometheus stores the metrics and evaluates alerting rules you define
• Alertmanager receives alerts from Prometheus and routes them to your notification channels

Think of it this way: DCGM is the doctor checking GPU vitals, Prometheus is the nurse recording the chart, and Alertmanager is the emergency pager system that calls you when something goes wrong.

🕵️ Common GPU Faults You Can Alert On¶

Here are the most important GPU health signals you should monitor with alerting rules:

• GPU Temperature — If a GPU exceeds safe operating temperature (typically above 85°C), it may throttle performance or shut down
• Memory Errors (ECC) — Correctable and uncorrectable memory errors indicate failing GPU memory
• Power Draw — Sudden drops in power consumption may mean a GPU has crashed or is not being utilized
• GPU Clock Speed — Unexpectedly low clock speeds can signal thermal throttling or hardware issues
• PCIe Link Errors — Connection problems between the GPU and the server motherboard
• XID Errors — NVIDIA's own error reporting system for GPU hardware faults

📊 Sample Alerting Rules Structure¶

Prometheus alerting rules are defined in YAML configuration files. Each rule has three main parts:

📊 Visual Representation: Prometheus Alert Rules and Alertmanager¶

This diagram displays how Prometheus evaluates alert conditions and forwards notifications to Alertmanager.

🛠️ Setting Up Alertmanager for GPU Notifications¶

To receive GPU fault notifications, you need to configure Alertmanager with receivers. A receiver defines where alerts should be sent.

Common Receiver Types for GPU Alerts¶

• Email — Sends alert details to an email address or distribution list
• Slack — Posts alerts to a specific Slack channel
• PagerDuty — Creates incidents for on-call engineers
• Webhook — Sends alerts to a custom HTTP endpoint (useful for integrating with internal tools)

Routing Alerts Based on Severity¶

You can configure Alertmanager to route different GPU faults to different places. For example:

• Critical alerts (like uncorrectable memory errors) go to PagerDuty for immediate action
• Warning alerts (like high but not critical temperature) go to a Slack channel for daytime review
• Info alerts (like GPU utilization drops) go to email for weekly reports

🧠 Best Practices for GPU Fault Alerting¶

Follow these guidelines to avoid alert fatigue while catching real problems:

• Set appropriate thresholds — Don't alert at the first sign of a temperature spike. Use a duration (like 5 minutes) to filter out temporary fluctuations
• Use severity levels — Classify alerts as critical, warning, or info so engineers know how to prioritize
• Include actionable information — Each alert annotation should tell the engineer what to do (e.g., "Check GPU slot 3 in rack A42")
• Test your alerts — Simulate GPU faults in a test environment to verify your rules work correctly
• Avoid duplicate alerts — Use Alertmanager's grouping and inhibition features to prevent notification storms

🔄 Alert Lifecycle: From Detection to Resolution¶

Understanding the alert lifecycle helps you know what to expect:

1. Firing — The GPU condition matches your alert rule (e.g., temperature exceeds 85°C for 5 minutes)
2. Notification sent — Alertmanager delivers the alert to your configured channels
3. Acknowledged — An engineer sees the alert and starts investigating
4. Resolved — The GPU condition returns to normal (temperature drops below threshold)
5. Silenced — If the issue is known (e.g., scheduled maintenance), you can silence the alert temporarily

✅ Key Takeaways for New Engineers¶

• Alertmanager turns GPU metrics into notifications — Without it, you'd have to watch dashboards all day
• Focus on the most impactful GPU faults — Start with temperature, memory errors, and XID errors
• Configure multiple notification channels — Use email for non-urgent alerts and PagerDuty for critical ones
• Test before deploying — Always verify your alert rules in a staging environment
• Keep alerts actionable — Every alert should tell someone what's wrong and what to do about it

By setting up proper alerting rules with Prometheus Alertmanager, you ensure that GPU faults are caught early — before they cause expensive training jobs to fail or corrupt your AI models.