24.2d NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES environment variables¶

🔍 Context Introduction¶

When you run a container with GPU access, two environment variables control which GPUs are visible and what GPU features are available inside the container. Without setting these correctly, your container may see no GPUs at all, or may crash when trying to use certain NVIDIA driver capabilities (like CUDA, NVML, or video encoding).

The NVIDIA Container Toolkit automatically sets these variables when you use the --gpus flag with Docker, but understanding them helps you troubleshoot issues and fine-tune GPU access for your workloads.

⚙️ What is NVIDIA_VISIBLE_DEVICES?¶

This variable tells the container which physical GPUs to expose.

• Purpose: Controls GPU visibility inside the container.
• Common values:
• all — Makes all GPUs on the host available.
• 0 — Only GPU 0 is visible.
• 0,1 — GPUs 0 and 1 are visible.
• none — No GPUs are visible (useful for testing).
• void — Same as none, but prevents fallback to all.
• Default behavior: If not set, the container sees no GPUs.

💡 Tip: Use all for most training workloads. Use specific GPU IDs for multi-container deployments where each container needs a dedicated GPU.

🛠️ What is NVIDIA_DRIVER_CAPABILITIES?¶

This variable defines which driver features the container can use.

• Purpose: Controls which NVIDIA driver libraries are mounted into the container.
• Common values:
• compute — Allows CUDA and CUDA-related operations.
• utility — Allows nvidia-smi and basic monitoring.
• graphics — Enables OpenGL and Vulkan (for rendering).
• video — Enables video encode/decode (NVENC/NVDEC).
• display — Enables display output (rarely needed in containers).

• all — Enables all capabilities (convenient but heavy).

• Default behavior: If not set, the container defaults to utility only — which means CUDA will not work.

⚠️ Warning: If you run a PyTorch or TensorFlow container without setting compute, your GPU will appear as "not available" even though nvidia-smi works.

📊 Visual Representation: NVIDIA Container Runtime Environment Variables¶

This diagram displays the primary runtime control environment variables used to tune GPU configuration inside containers.

📊 Comparison Table: Common Use Cases¶

🕵️ How They Work Together¶

These two variables are independent but complementary:

• NVIDIA_VISIBLE_DEVICES decides which GPUs are seen.
• NVIDIA_DRIVER_CAPABILITIES decides what those GPUs can do.

Example scenario:
If you set NVIDIA_VISIBLE_DEVICES=0 and NVIDIA_DRIVER_CAPABILITIES=video, then: - Only GPU 0 is visible inside the container. - That GPU can only be used for video encoding/decoding — CUDA operations will fail.

🧪 Practical Tips for New Engineers¶

• Start with all for both when experimenting — it's the most permissive and least likely to cause errors.
• Reduce scope for production — only grant the capabilities your workload actually needs (e.g., don't enable graphics for a training container).
• Check visibility by running nvidia-smi inside the container. If it shows no GPUs, check NVIDIA_VISIBLE_DEVICES.
• Check capabilities by running a small CUDA program. If it fails with "CUDA driver version is insufficient", check NVIDIA_DRIVER_CAPABILITIES.
• Remember the default — if you don't set NVIDIA_DRIVER_CAPABILITIES, you get utility only. Always set it explicitly for compute workloads.

✅ Summary¶

These two environment variables are the gatekeepers of GPU access in containers. Mastering them will save you hours of debugging when your GPU-accelerated application doesn't work as expected.