🗺️ Roadmap: The Nvidia Software Stack — Bridging Silicon and Python¶

Below is the syllabus timeline of lessons in this module. Track your study progress here.

📈 Progress¶

---

📁 22 Base Drivers, CUDA, and Core Libraries¶

• 22.1a Driver branches: Data Center (production), Long-Term Support (LTS), and latest
• 22.1b Installation methods: apt/dnf repository (recommended) vs .run file (manual)
• 22.1c DKMS (Dynamic Kernel Module Support): auto-rebuilding drivers after kernel updates
• 22.1d Driver and CUDA version compatibility matrix
• 22.1e Troubleshooting: nvidia-smi: no devices found and Xid error 79 (GPU fallen off bus)
• 22.2a CUDA programming model: grids, blocks, and threads mapped to GPU hardware
• 22.2b CUDA Toolkit components: nvcc compiler, runtime libraries, and development headers
• 22.2c CUDA Toolkit versioning: major.minor and backward/forward compatibility rules
• 22.2d cuBLAS: GPU-accelerated BLAS (Basic Linear Algebra Subroutines) library
• 22.2e cuFFT: Fast Fourier Transform acceleration for signal processing workloads
• 22.2f cuSPARSE: sparse matrix operations critical for certain Transformer optimizations
• 22.2g CUDA Graphs: pre-compiling GPU execution graphs for reduced launch overhead
• 22.3a cuDNN's role: hardware-optimized implementations of convolutions, pooling, activations
• 22.3b cuDNN heuristics: auto-selecting the best algorithm for given input shapes
• 22.3c cuDNN versioning and compatibility with PyTorch and TensorFlow
• 22.4a NCCL operations: AllReduce, AllGather, Reduce, Broadcast, ReduceScatter
• 22.4b NCCL topology detection: auto-selecting NVLink, PCIe, or InfiniBand for communication
• 22.4c NCCL environment variables for tuning: NCCL_DEBUG, NCCL_IB_HCA, NCCL_NET_GDR_LEVEL

📁 23 Containerization Fundamentals — Docker from First Principles¶

• 23.1a Why Python environments break: conflicting library versions, system package collisions
• 23.1b Virtual machines vs. containers: isolation with and without a hypervisor
• 23.1c The container promise: 'works on my machine' becomes deployable everywhere
• 23.2a PID namespace: isolated process trees — container processes see only themselves
• 23.2b Network namespace: isolated network stacks, interfaces, and routing tables
• 23.2c Mount namespace: isolated filesystem views with bind mounts
• 23.2d Cgroups v1 and v2: resource limits for CPU, memory, and I/O per container
• 23.2e Union filesystems (overlay2): layered image architecture explained
• 23.3a Images vs. containers: the blueprint vs. the running instance
• 23.3b Dockerfile anatomy: FROM, RUN, COPY, ENV, EXPOSE, ENTRYPOINT, CMD
• 23.3c Building images: docker build, layer caching, and .dockerignore optimization
• 23.3d Running containers: docker run flags (--rm, -d, -p, -v, --name, --network)
• 23.3e Volume mounts: -v and --mount for persisting training datasets and checkpoints
• 23.3f Networking: bridge, host, and none modes — choosing for performance
• 23.3g Docker Compose: multi-container service definitions in YAML
• 23.3h Registry operations: docker pull, push, tag, and using private registries
• 23.3i Multi-stage builds: separating build and runtime environments to minimize image size

📁 24 GPU-Accelerated Containers — The NVIDIA Container Toolkit¶

• 24.1a Device files: /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm — what each does
• 24.1b Driver libraries: libcuda.so, libnvidia-ml.so — must be accessible inside containers
• 24.1c The old way: manual --device and -v driver mounts — error-prone and brittle
• 24.2a NVIDIA Container Runtime: a shim that injects GPU devices into container specs
• 24.2b Installation: nvidia-container-toolkit package and docker daemon configuration
• 24.2c Using --gpus all, --gpus 2, --gpus 'device=0,1': selecting GPUs per container
• 24.2d NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES environment variables
• 24.2e Verifying GPU access: nvidia-smi inside a container
• 24.3a CDI specification: declarative JSON device definitions for any hardware
• 24.3b Generating CDI specs: nvidia-cdi-hook generate
• 24.3c CDI with containerd and CRI-O: Kubernetes-compatible GPU exposure
• 24.3d Why CDI replaces the hook-based approach for modern Kubernetes deployments

📁 25 NVIDIA NGC, AI Enterprise, and NIMs¶

• 25.1a What NGC provides: optimized containers, pre-trained models, Helm charts, resources
• 25.1b NGC container registry: docker pull nvcr.io/nvidia/pytorch:24.01-py3
• 25.1c NVIDIA PyTorch container: what is pre-installed and why it differs from pip PyTorch
• 25.1d NGC API key authentication: generating and configuring access
• 25.1e NGC CLI: programmatic access to download models and resources
• 25.2a What NVAIE provides: enterprise support SLAs, security patching, certified stacks
• 25.2b NVAIE licensing: per-GPU subscriptions and vGPU license server configuration
• 25.2c NVAIE certified containers: tested combinations of driver/CUDA/framework versions
• 25.2d CVE patching lifecycle: how NVIDIA responds to security vulnerabilities in its stack
• 25.3a What NIM is: a pre-packaged, optimized inference engine in a container
• 25.3b NIM components: TensorRT-LLM engine, Triton server, OpenAI-compatible API
• 25.3c Deploying a NIM locally: docker run + --gpus + API key
• 25.3d Available NIMs: Llama 3, Mistral, Stable Diffusion, and domain-specific models
• 25.3e NIM caching: model weight caching on host for faster restart times
• 25.4a TensorRT: graph optimization, layer fusion, and precision calibration
• 25.4b TensorRT-LLM: NVIDIA's open-source LLM inference library with kernel optimizations
• 25.4c In-flight batching (continuous batching): GPU utilization for dynamic inference
• 25.4d Paged KV cache: managing memory for thousands of concurrent inference requests
• 25.5a Triton architecture: model repository, backends, and the C++ inference core
• 25.5b Supported backends: TensorRT, ONNX, PyTorch TorchScript, Python, OpenVINO
• 25.5c Dynamic batching and model instances for maximum GPU utilization
• 25.5d gRPC and REST APIs, Prometheus metrics, and model lifecycle management