🗺️ Roadmap: The Cluster Orchestration — Managing the GPU Fleet at Scale¶
Below is the syllabus timeline of lessons in this module. Track your study progress here.
📈 Progress¶
📁 26 Kubernetes Fundamentals — Container Orchestration from Scratch¶
- 26.1a From 'pet' servers to 'cattle' clusters: the philosophy shift
- 26.1b The problem with running containers manually at scale: scheduling, failure, upgrades
- 26.1c What Kubernetes automates: placement, health-checking, scaling, networking, storage
- 26.2a API Server (kube-apiserver): the REST frontend — all control plane communication
- 26.2b etcd: the distributed key-value store — the authoritative cluster state
- 26.2c Scheduler (kube-scheduler): matching Pods to Nodes based on resources and constraints
- 26.2d Controller Manager: Deployment controller, ReplicaSet controller, Node controller
- 26.2e Kubelet: the node agent — ensuring Pods are running as declared
- 26.2f Kube-proxy: network rules for Service routing on each node
- 26.2g Container Runtime Interface (CRI): containerd and CRI-O as Docker replacements
- 26.3a Pods: the smallest deployable unit — one or more containers sharing network/storage
- 26.3b Deployments: declarative Pod management with rolling updates and rollbacks
- 26.3c StatefulSets: ordered Pod creation with stable network identities — for databases
- 26.3d Jobs and CronJobs: batch workloads with completion semantics — AI training runs
- 26.3e Services: stable networking endpoints for Pods (ClusterIP, NodePort, LoadBalancer)
- 26.3f ConfigMaps and Secrets: externalizing configuration and credentials from containers
- 26.3g PersistentVolumes (PV) and PersistentVolumeClaims (PVC): durable storage in K8s
- 26.3h Namespaces: multi-team isolation within one cluster
- 26.3i Resource Requests and Limits: CPU and memory budgets for every Pod
- 26.4a kubectl get, describe, logs, exec, apply, delete — the essential verb set
- 26.4b YAML manifests: writing and applying Deployment and Pod specs
- 26.4c Helm: the package manager for Kubernetes — charts, values, and releases
- 26.4d kustomize: overlay-based configuration management without templating
📁 27 Kubernetes for AI — NVIDIA GPU and Network Operators¶
- 27.1a What the GPU Operator automates: driver DaemonSet, container toolkit, DCGM exporter, MIG config
- 27.1b Operator architecture: the operator pattern and Custom Resource Definitions (CRDs)
- 27.1c Installing the GPU Operator via Helm: values.yaml configuration options
- 27.1d ClusterPolicy CRD: the single object that controls the entire GPU software stack
- 27.1e Using pre-installed drivers: operatortype=preinstalled vs. kernel module management
- 27.1f Verifying GPU Operator: pod status across kube-system and gpu-operator namespaces
- 27.2a What the Network Operator automates: OFED drivers, RDMA device plugin, SR-IOV
- 27.2b NicClusterPolicy CRD: configuring RDMA devices across all nodes
- 27.2c SR-IOV (Single Root I/O Virtualization): virtual functions for containers
- 27.2d Multus CNI: attaching multiple network interfaces to AI training Pods
- 27.2e RDMA device plugin: advertising ib0 and rdma resources to the K8s scheduler
- 27.3a Requesting GPUs in Pod spec: resources.limits.nvidia.com/gpu: 8
- 27.3b Node affinity and taints/tolerations: directing GPU workloads to GPU nodes
- 27.3c Volcano and KUEUE: batch scheduling frameworks for multi-GPU training jobs in K8s
- 27.3d MPI Operator and PyTorch Operator: running distributed training jobs in K8s
- 27.3e Gang scheduling: all-or-nothing Pod placement for multi-node training
📁 28 Advanced GPU Resource Management — MIG, MPS, and Time-Slicing¶
- 28.1a GPU utilization on inference workloads: often 10–30% of peak throughput
- 28.1b The multi-tenancy challenge: sharing expensive GPUs safely between teams
- 28.1c Three technologies compared: Time-Slicing vs MPS vs MIG — the decision framework
- 28.2a How time-slicing works: GPU context switching between processes
- 28.2b Configuring time-slicing in Kubernetes via GPU Operator configmap
- 28.2c Limitations: no memory isolation, no fault isolation, no QoS guarantees
- 28.2d When to use: development environments, low-priority inference, cost optimization
- 28.3a MPS architecture: a daemon that funnels multiple CUDA contexts through one control thread
- 28.3b Benefits: lower context switch overhead vs time-slicing, true concurrent execution
- 28.3c Limitations: single CUDA execution context = single fault domain
- 28.3d Enabling MPS: nvidia-cuda-mps-control daemon and K8s configuration
- 28.4a MIG architecture: physically partitioning the GPC, HBM, L2 cache, and engines
- 28.4b MIG instance profiles on A100: 1g.5gb, 2g.10gb, 3g.20gb, 4g.20gb, 7g.40gb
- 28.4c MIG instance profiles on H100: new profiles with 80 GB HBM3 distribution
- 28.4d Enabling MIG: nvidia-smi -i 0 --mig-mode=1 and rebooting
- 28.4e Creating, listing, and destroying MIG instances and compute instances
- 28.4f MIG in Kubernetes: GPU Operator MIG strategy (single, mixed, all)
- 28.4g Security guarantees: complete memory isolation and fault isolation between MIG slices
- 28.4h Multi-tenant SaaS use case: giving different customers their own H100 slice
📁 29 Traditional HPC Schedulers — Slurm and Apptainer¶
- 29.1a Slurm architecture: slurmctld (controller), slurmd (node daemon), slurmdb (accounting)
- 29.1b Job submission: sbatch scripts, #SBATCH directives, and job arrays
- 29.1c Requesting GPUs in Slurm: --gres=gpu:a100:8 and GRES configuration
- 29.1d Slurm partitions, QoS, and fair-share scheduling for multi-team GPU access
- 29.1e squeue, sinfo, scontrol show job: monitoring and diagnosing job states
- 29.1f Slurm vs. Kubernetes: choosing the right scheduler for your AI workload type
- 29.2a Why Docker is not used in HPC: root requirement and security model
- 29.2b Apptainer's security model: user-namespace execution, no root inside container
- 29.2c SIF (Singularity Image Format): single-file immutable container images
- 29.2d Converting Docker/NGC containers to Apptainer format
- 29.2e Running GPU workloads with Apptainer: --nv flag for NVIDIA support