🗺️ Roadmap: The Virtualization and Cloud — Extending AI Infrastructure Beyond Bare Metal¶
Below is the syllabus timeline of lessons in this module. Track your study progress here.
📈 Progress¶
📁 30 GPU Diagnostics and nvidia-smi Mastery¶
- 30.1a nvidia-smi basic output: reading driver version, CUDA version, GPU list
- 30.1b Power draw, temperature, fan speed, and utilization columns
- 30.1c Memory used vs. total: identifying memory leaks and stuck processes
- 30.1d nvidia-smi -q: exhaustive GPU information dump
- 30.1e nvidia-smi topo -m: printing the GPU interconnect topology matrix
- 30.1f nvidia-smi dmon: continuous device monitoring with configurable metrics
- 30.1g nvidia-smi pmon: per-process monitoring — identifying which PID uses GPU
- 30.1h Controlling GPUs: set-power-limit, set-compute-mode, persistence-mode
- 30.2a Xid errors: NVIDIA's GPU error taxonomy — the complete Xid reference
- 30.2b Critical Xids: 63 (Row Remapping), 74 (NVLink Error), 79 (GPU Fallen Off Bus), 94 (DBE)
- 30.2c ECC error types: Single-bit correctable (SBE) vs. Double-bit uncorrectable (DBE)
- 30.2d nvidia-smi --query-gpu=ecc.errors.uncorrected.volatile.total: querying ECC counters
- 30.2e Row Remapping: VRAM repair mechanism and pending remapping reboots
- 30.2f PCIe AER (Advanced Error Reporting): kernel-level PCIe error decoding
- 30.2g Thermal throttling events: reading p-state and clock throttle reasons
- 30.2h Power capping: when TDP limits cause performance reduction
- 30.2i NVIDIA DCGM health checks: automating hardware validation before training runs
- 30.3a cuda-bandwidthtest: validating memory bandwidth — H2D, D2H, D2D
- 30.3b dcgmDiag: NVIDIA's hardware diagnostic tool for pre-flight cluster checks
- 30.3c gpu-burn: stress testing GPU compute and memory to identify hardware defects
- 30.3d nccl-tests: validating collective communication performance across all GPUs
- 30.3e ibdiagnet and perfquery: InfiniBand fabric diagnostics and performance counters
📁 31 Cluster-Wide Telemetry — DCGM, Prometheus, and Grafana¶
- 31.1a DCGM vs. nvidia-smi: cluster-scale vs single-GPU monitoring
- 31.1b DCGM architecture: nv-hostengine, libdcgm, and the DCGM API
- 31.1c DCGM field IDs: the complete list of GPU metrics available for scraping
- 31.1d dcgmi: the DCGM CLI for interactive diagnostics and field watching
- 31.1e DCGM health watches: proactive monitoring for ECC, PCIe, NVLink, and power
- 31.1f DCGM groups and job stats: tracking GPU usage per Slurm or Kubernetes job
- 31.2a dcgm-exporter architecture: a Kubernetes DaemonSet exposing /metrics endpoint
- 31.2b Customizing collected metrics: the dcgm-exporter configmap
- 31.2c GPU Operator deployment: dcgm-exporter is automatically deployed and configured
- 31.3a Prometheus data model: time-series metrics with labels
- 31.3b Prometheus scrape configuration: targeting dcgm-exporter and node-exporter
- 31.3c PromQL basics: querying GPU temperature, utilization, and memory across nodes
- 31.3d Alerting rules: Prometheus alertmanager for GPU fault notifications
- 31.4a Connecting Grafana to Prometheus as a data source
- 31.4b NVIDIA DCGM Grafana dashboard: importing the official GPU cluster dashboard
- 31.4c Building custom panels: GPU utilization heatmaps, memory trend charts
- 31.4d Alerting in Grafana: PagerDuty, Slack, and email integration for GPU faults
- 31.4e The complete observability stack: node_exporter + dcgm-exporter + Prometheus + Grafana
📁 32 Enterprise Cluster Management — NVIDIA Base Command Manager¶
- 32.1a Manual provisioning pain: installing Ubuntu on 1000 nodes one by one
- 32.1b PXE boot and network installation: the baseline automated approach
- 32.1c Configuration drift: why manually managed clusters diverge from their intended state
- 32.2a BCM overview: the enterprise cluster management platform for DGX SuperPOD
- 32.2b BCM head node architecture: redundant management servers
- 32.2c Software image management: golden images, image profiles, and OS provisioning
- 32.2d Network provisioning: automated switch configuration and fabric setup via BCM
- 32.2e GPU stack deployment: driver, CUDA, container toolkit pushed cluster-wide
- 32.3a BCM user and project management: creating accounts, setting GPU quotas
- 32.3b Integrated Slurm management: job submission, monitoring, and accounting through BCM
- 32.3c Kubernetes integration: BCM as the provisioning layer beneath a K8s cluster
- 32.3d Health dashboards: cluster-wide node status, GPU availability, and job history
- 32.4a Ansible for GPU cluster configuration: playbooks, roles, and inventory management
- 32.4b Terraform for cloud-based AI infrastructure: provider configurations for AWS, Azure, GCP
- 32.4c GitOps: managing cluster configuration as version-controlled YAML in Git repositories
📁 33 GPU Virtualization — vGPU and AI in Virtual Machines¶
- 33.1a Type 1 (bare-metal) hypervisors: VMware vSphere ESXi, Microsoft Hyper-V, KVM
- 33.1b Type 2 hypervisors: VirtualBox, VMware Workstation — for development only
- 33.1c GPU passthrough (PCIe passthrough): dedicating one GPU to one VM
- 33.2a vGPU architecture: host driver intercepts GPU commands from guest VMs
- 33.2b vGPU profiles: Q-series (Quadro/Workstation), C-series (Compute/AI), A-series (apps)
- 33.2c NVIDIA Virtual Compute Server (vCS): vGPU profiles for AI training and inference VMs
- 33.2d vGPU Manager installation on ESXi and KVM hosts
- 33.2e vGPU licensing server: NVIDIA License System (NLS) and license management
- 33.3a MIG-backed vGPU: using MIG instances as the underlying resource for vGPU
- 33.3b Strong isolation: complete hardware partitioning for multi-tenant cloud environments
📁 34 Cloud AI Infrastructure — AWS, Azure, GCP, and Oracle¶
- 34.1a AWS EC2: P4d (A100), P5 (H100), P5e (H200) instance families
- 34.1b Azure: NDv5 (H100), NDmv4 (A100) VM series
- 34.1c Google Cloud: A3 (H100), A2 (A100) machine types
- 34.1d Oracle Cloud Infrastructure (OCI): BM.GPU.H100.8 bare metal instances
- 34.1e On-demand vs. reserved vs. spot/preemptible pricing strategies for AI workloads
- 34.2a Elastic Fabric Adapter (EFA) on AWS: low-latency networking for P5 clusters
- 34.2b Placement groups: packing instances in the same physical rack for minimum latency
- 34.2c VPC design for AI clusters: subnets, security groups, and VPC endpoints
- 34.3a Amazon SageMaker: managed training jobs and endpoint deployment
- 34.3b Azure Machine Learning: workspaces, compute clusters, and MLflow integration
- 34.3c Google Vertex AI: training pipelines and model registry
- 34.3d NVIDIA DGX Cloud: NVIDIA-managed AI supercomputing as a subscription service