🗺️ Roadmap: The Nvidia GPU Architecture — The Silicon at the Heart of AI Infrastructure¶
Below is the syllabus timeline of lessons in this module. Track your study progress here.
📈 Progress¶
📁 10 GPU Microarchitecture — Inside an NVIDIA Data Center GPU¶
- 10.1a GPU Die → GPC → TPC → SM → CUDA Core: the complete hierarchy
- 10.1b Graphics Processing Clusters (GPCs): high-level partitions of the GPU
- 10.1c Texture Processing Clusters (TPCs): grouping SMs for scheduling
- 10.2a SM internal components: CUDA cores, Tensor Cores, schedulers, and register file
- 10.2b Warps: groups of 32 threads executing in lockstep — the SIMT model
- 10.2c Warp schedulers and instruction dispatch units
- 10.2d Occupancy: the ratio of active warps to maximum warps — why it matters
- 10.2e Shared memory and L1 cache within the SM: the scratchpad memory model
- 10.3a What CUDA cores actually are: integer, FP32, and FP64 execution units
- 10.3b CUDA core counts across generations: from thousands to tens of thousands
- 10.3c SIMT (Single Instruction, Multiple Thread) execution model
- 10.4a The GEMM operation in hardware: D = A×B + C executed in a single clock
- 10.4b First-gen Tensor Cores (Volta V100): FP16 accumulation
- 10.4c Second-gen Tensor Cores (Turing/Ampere): INT8, INT4, TF32, BF16
- 10.4d Third-gen Tensor Cores (Hopper H100): FP8, structured sparsity (2:4)
- 10.4e Fourth-gen Tensor Cores (Blackwell B200): FP4 and massive scale
- 10.4f TF32: NVIDIA's training-optimized compromise between FP32 range and FP16 speed
- 10.4g Structured Sparsity (2:4): skipping zeros to double effective Tensor Core throughput
- 10.5a RT Cores: ray tracing hardware — why they exist and when AI uses them (3D data)
- 10.5b Video encode/decode engines (NVENC/NVDEC): relevant for data preprocessing pipelines
- 10.5c Asynchronous Copy Engines: overlapping data transfer and compute
- 10.5d Transformer Engine (Hopper+): dynamic precision selection per layer for LLMs
📁 11 GPU Memory Subsystems — VRAM, Bandwidth, and the Capacity Crisis¶
- 11.1a Latency and bandwidth requirements: why system DDR cannot feed GPU cores
- 11.1b The GPU memory wall: how VRAM capacity determines maximum model size
- 11.1c VRAM allocation breakdown: weights, optimizer states, activations, KV cache
- 11.2a GDDR5, GDDR6, GDDR6X: generational improvements in bandwidth
- 11.2b GDDR physical packaging: mounted chips on PCB, width limited by board lanes
- 11.2c Where GDDR is used today: NVIDIA RTX cards, A2, some inference cards
- 11.3a 3D stacking with silicon interposers: the physical architecture of HBM
- 11.3b HBM2, HBM2e, HBM3, HBM3e: generational bandwidth and capacity milestones
- 11.3c HBM on the H100: 80 GB HBM3 at 3.35 TB/s bandwidth
- 11.3d HBM on the B200: 192 GB HBM3e at 8 TB/s — the new baseline
- 11.3e ECC in HBM: protecting multi-billion dollar training runs from bit errors
- 11.4a Calculating bytes per second: memory clock × bus width × data rate
- 11.4b Memory-bound vs. compute-bound operations in deep learning
- 11.4c KV cache explosion in long-context inference: why 80 GB fills faster than expected
- 11.4d FlashAttention: algorithmic memory optimization for Transformer inference
- 11.5a Register file → Shared Memory → L1 Cache → L2 Cache → HBM
- 11.5b Latency and bandwidth at each level — why kernel optimization targets L1/shared
- 11.5c Managed memory and Unified Memory: bridging CPU and GPU address spaces
📁 12 NVIDIA GPU Generations & Form Factors¶
- 12.1a Volta (V100 — 2017): first Tensor Cores, HBM2, NVLink 2.0 — the AI pivot
- 12.1b Ampere (A100 — 2020): 3rd-gen Tensor Cores, MIG, NVLink 3.0, 80 GB HBM2e
- 12.1c Hopper (H100 — 2022): Transformer Engine, FP8, NVLink 4.0, HBM3, 4th-gen Tensor Cores
- 12.1d Hopper variants: H100 SXM5, H100 PCIe, H800 (export-limited), H200 (HBM3e upgrade)
- 12.1e Blackwell (B100/B200/GB200 — 2024): 5th-gen Tensor Cores, FP4, NVLink 5.0, NVL72 rack
- 12.1f Ada Lovelace (L40S) and its role in inference and rendering workloads
- 12.2a PCIe Edge Cards: standard slot insertion, passive cooling, TDP ≤350W
- 12.2b SXM (Server PCI Express Module): direct baseboard mounting, TDP 700W+
- 12.2c OAM (OCP Accelerator Module): open standard for hyperscaler custom servers
- 12.2d NVL (NVLink) form factor: Blackwell's multi-chip modules (MCM design)
- 12.3a A30, A10, L4: inference-optimized cards for lower TDP deployments
- 12.3b A100, H100, H200: training-class flagship GPUs
- 12.3c B200, GB200: Blackwell generation and the NVL72 superchip concept
- 12.3d NVIDIA Grace CPU: the ARM-based processor paired with Hopper/Blackwell
- 12.3e GH200 Grace Hopper Superchip: CPU+GPU unified memory via NVLink-C2C
- 12.3f GB200 NVL72: 72 Blackwell GPUs + 36 Grace CPUs in a single liquid-cooled rack
📁 13 Multi-GPU Interconnects — Scaling Beyond One GPU¶
- 13.1a Data Parallelism: splitting batches across GPUs and averaging gradients
- 13.1b Model Parallelism: distributing layers across GPUs when one GPU is too small
- 13.1c Tensor Parallelism: splitting individual layers (attention heads) across GPUs
- 13.1d Pipeline Parallelism: assigning sequential model stages to different GPUs
- 13.1e AllReduce: the collective communication operation that synchronizes gradients
- 13.2a NVLink 1.0 through 5.0: bandwidth progression from 160 GB/s to 1.8 TB/s per GPU
- 13.2b Physical implementation: gold contacts on SXM modules, bridges on PCIe cards
- 13.2c NVLink topology in DGX A100: fully connected 8-GPU mesh
- 13.2d Comparing NVLink bandwidth to PCIe Gen 4 x16 (64 GB/s): a 28x improvement
- 13.3a The problem NVSwitch solves: NVLink peer-to-peer vs. switched fabric
- 13.3b NVSwitch generations: 1.0 (Volta), 2.0 (Ampere), 3.0 (Hopper), 4.0 (Blackwell)
- 13.3c NVSwitch chip count per platform and total aggregate bandwidth
- 13.3d SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) in NVSwitch 3.0
- 13.4a HGX A100/H100: the reference baseboard with 8 SXM GPUs and NVSwitch
- 13.4b DGX A100: NVIDIA's complete system — HGX + 2x AMD EPYC + networking
- 13.4c DGX H100: HGX H100 + InfiniBand ConnectX-7 + 400GbE management
- 13.4d DGX B200: Blackwell system specifications and NVL72 rack design
- 13.4e DGX SuperPOD: clustering multiple DGX nodes via InfiniBand for petaflop-scale clusters
- 13.5a NVLink-C2C vs. standard NVLink: die-to-die vs. node-to-node
- 13.5b Grace Hopper Superchip: CPU and GPU sharing a coherent memory space
- 13.5c Implications for memory-intensive inference (no PCIe bottleneck)