3.4d journalctl: filtering by time, unit, priority, and following live logs¶

🌱 Context Introduction¶

When managing AI workloads, you will often need to inspect system logs to troubleshoot issues like service crashes, hardware errors, or application failures. The journalctl tool is your primary interface for reading logs managed by systemd-journald. Unlike traditional log files, journald stores logs in a structured binary format that supports powerful filtering. This section covers the essential filtering techniques every engineer should know: filtering by time, by systemd unit, by priority (log level), and following logs in real time.

⚙️ Filtering by Time¶

Time-based filtering helps you narrow down logs to a specific window of interest — for example, when a GPU driver failed or when a training job started.

• Absolute time ranges: You can specify exact start and end times using the --since and --until flags.
• Relative time keywords: Use human-readable keywords like "yesterday", "today", "now", "1 hour ago", or "3 days ago".
• Date format: Dates follow the format "YYYY-MM-DD HH:MM:SS" (24-hour clock). Timezone is typically local unless specified.

For reference:

journalctl --since "2025-03-01 08:00:00" --until "2025-03-01 12:00:00"

📤 Output: All log entries between 8:00 AM and 12:00 PM on March 1, 2025.

For reference:

journalctl --since "yesterday" --until "now"

📤 Output: All log entries from yesterday up to the current moment.

🛠️ Filtering by Unit¶

In systemd, each service, socket, mount, or timer is called a unit. Filtering by unit lets you see logs only for a specific service — essential when debugging a single AI application or daemon.

• Use the -u or --unit flag followed by the unit name (e.g., nvidia-persistenced.service).
• You can combine multiple -u flags to see logs from several units at once.
• Unit names are case-sensitive and typically end with .service, .socket, .mount, etc.

For reference:

journalctl -u nvidia-persistenced.service

📤 Output: All logs generated by the NVIDIA persistence daemon.

For reference:

journalctl -u sshd.service -u docker.service

📤 Output: Logs from both the SSH daemon and Docker service, merged in chronological order.

🕵️ Filtering by Priority (Log Level)¶

Logs are assigned a priority level from 0 (emergency) to 7 (debug). Filtering by priority helps you cut through noise and focus on critical events.

• Use the -p or --priority flag followed by a level name or number.
• Common levels: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7).
• The filter shows logs at the specified level and above (more severe). For example, -p err shows err, crit, alert, and emerg.

For reference:

journalctl -p err

📤 Output: Only log entries with priority error or higher (critical, alert, emergency).

For reference:

journalctl -u kubelet.service -p warning

📤 Output: Warnings and more severe logs from the kubelet service.

📡 Following Live Logs¶

When you need to watch logs as they happen — for example, during a model training run or a service restart — use the follow mode.

• Use the -f or --follow flag to tail logs in real time.
• Combine -f with other filters (unit, priority, time) for targeted live monitoring.
• Press Ctrl+C to exit follow mode.

For reference:

journalctl -f

📤 Output: New log entries appear in real time as they are written.

For reference:

journalctl -u nvidia-smi.service -f

📤 Output: Live log stream from the NVIDIA SMI service.

📊 Comparison Table: Common journalctl Filtering Options¶

📊 Visual Representation: journald Logging Pipeline & Filters¶

This diagram maps how log data flows from kernel and user services into the systemd-journald binary storage, and how the journalctl command filters this data.

🧠 Practical Tips for Engineers¶

• Combine filters for precision: Use -u with -p and --since to isolate exactly what you need. For example, check errors from a GPU service in the last hour.
• Reverse order: Add -r or --reverse to show newest logs first — helpful for seeing the most recent failure immediately.
• Output control: Use --no-pager to print logs directly to the terminal without a pager (useful in scripts).
• Persistent logs: By default, journald stores logs in memory. To keep logs across reboots, ensure /var/log/journal exists and is writable.
• Disk usage: Check journal size with journalctl --disk-usage and clean old logs with journalctl --vacuum-time=2weeks.

✅ Summary¶

Mastering journalctl filtering is a core skill for operating AI infrastructure. You can now: - Isolate logs by time to focus on specific incidents. - Filter by unit to debug individual services. - Use priority levels to cut through informational noise. - Follow live logs for real-time monitoring.

These techniques will help you quickly diagnose issues in GPU drivers, container runtimes, orchestration services, and AI application stacks. Practice combining filters to become efficient at log analysis.