4.5d Parsing GPU logs and DCGM output with awk and grep¶

🧠 Context Introduction¶

When managing AI infrastructure, GPU logs and DCGM (NVIDIA Data Center GPU Manager) output contain critical information about GPU health, performance, and errors. Engineers need to quickly extract meaningful data from these logs — such as temperature spikes, memory errors, or power consumption anomalies — without manually scrolling through hundreds of lines. This is where awk and grep become essential tools. They allow you to filter, search, and format GPU-related log data efficiently, helping you monitor and troubleshoot AI workloads at scale.

⚙️ What Are GPU Logs and DCGM Output?¶

• GPU Logs: System-level logs generated by the NVIDIA driver, typically found in /var/log/syslog or /var/log/messages. They contain events like GPU resets, thermal throttling, or driver errors.
• DCGM Output: Real-time or historical metrics from the NVIDIA Data Center GPU Manager. This includes GPU utilization, memory usage, temperature, power draw, and error counts. DCGM can output data in plain text or JSON format.

Both sources are text-based, making them ideal for parsing with grep (for pattern matching) and awk (for field extraction and formatting).

🕵️ Using grep to Filter GPU Logs¶

grep is your first line of defense for finding specific GPU-related events in large log files.

• Search for GPU errors: Use grep with the keyword "NVRM" (NVIDIA Resource Manager) to find driver-level errors in system logs.
• Filter by severity: Combine grep with patterns like "error", "fail", or "warning" to isolate critical events.
• Case-insensitive search: Use the -i flag to match both uppercase and lowercase variations (e.g., "gpu" vs "GPU").
• Show surrounding context: Use -B (before), -A (after), or -C (context) flags to display lines around a match, helping you understand the event sequence.

Example workflow:
To find all GPU-related errors in the last hour, you might pipe journalctl output into grep with the pattern "NVRM.*error". This gives you a focused list of driver-level issues.

📊 Using awk to Extract Structured Data from DCGM¶

awk excels at processing tabular or delimited output, which is common in DCGM reports.

• DCGM dmon output: When you run dcgmi dmon, it produces columns for GPU index, temperature, power, memory usage, and more. Use awk to extract specific columns.
• Field separation: DCGM output often uses spaces or commas as delimiters. Use awk's -F flag to specify the delimiter (e.g., -F"," for CSV).
• Conditional filtering: Use awk to print only rows where a value exceeds a threshold, such as temperature > 80°C or memory errors > 0.
• Header skipping: Use awk's NR>1 condition to skip the first line (header) when processing data.

Example workflow:
To list all GPUs with memory usage above 90%, you might pipe dcgmi dmon -e 100 output into awk and check if the memory column (field 4) is greater than 90. This gives you a quick view of overutilized GPUs.

🛠️ Combining grep and awk for Advanced Parsing¶

The real power comes from chaining these tools together.

• Step 1 — Filter with grep: First, use grep to isolate lines containing a specific keyword, such as "temperature" or "Xid" (NVIDIA error codes).
• Step 2 — Extract with awk: Then, pipe the filtered output into awk to extract only the fields you need, such as timestamp, GPU index, and temperature value.
• Step 3 — Format output: Use awk's printf function to create clean, readable reports with column headers.

Example workflow:
To get a clean table of GPU temperatures over time, you might: 1. Use grep to find lines with "temperature" from a DCGM log file. 2. Pipe that into awk to print the timestamp (field 1) and temperature value (field 3). 3. Add a header line using awk's BEGIN block.

📋 Comparison Table: grep vs awk for GPU Log Parsing¶

🧪 Practical Parsing Scenarios¶

• Scenario A — Checking for Xid errors:
  Use grep with the pattern "Xid" on your system log to find critical GPU hardware errors. Xid errors indicate issues like memory faults or bus problems.

• Scenario B — Monitoring GPU power consumption:
  Use awk on DCGM output to extract the power column and calculate average power draw across all GPUs. This helps identify power-hungry workloads.

• Scenario C — Identifying thermal throttling:
  Combine grep for "thermal" or "throttle" with awk to print timestamps and GPU IDs where throttling events occurred. This aids in cooling optimization.

• Scenario D — Parsing DCGM JSON output:
  If DCGM outputs JSON (using -o json), you can still use grep to find keys like "temperature" and then use awk to extract values after the colon. For complex JSON, consider jq as an alternative.

📊 Visual Representation: NVIDIA GPU Diagnostic Log Parsing Pipeline¶

This horizontal flowchart shows how raw system logs are processed to extract NVIDIA driver error events (such as GPU Xid codes) and compile them into a readable diagnostic report.

✅ Key Takeaways for New Engineers¶

• grep is your quick-search tool for finding specific GPU events or errors in logs.
• awk is your data-extraction tool for turning raw DCGM output into structured reports.
• Combining both gives you a powerful pipeline for real-time GPU monitoring and troubleshooting.
• Always test your grep patterns and awk field numbers on a small sample before running on large log files.
• Remember to consider log rotation — older logs may be compressed (.gz), so use zcat or zgrep for those files.

By mastering these two tools, you can efficiently parse GPU logs and DCGM output, enabling faster diagnosis of AI infrastructure issues.