11.3a 3D stacking with silicon interposers: the physical architecture of HBM

📦 Nvidia GPU Architecture 📖 GPU Memory Subsystems

🧠 Context Introduction

Modern AI workloads, like training large language models or running real-time inference, demand massive amounts of memory bandwidth. Traditional DRAM (like the RAM in your laptop) is too slow and consumes too much power. This is where High Bandwidth Memory (HBM) comes in. HBM is a specialized memory technology that uses 3D stacking and silicon interposers to pack memory chips vertically and connect them directly to the GPU. Think of it like building a skyscraper of memory layers instead of a single-story warehouse — it saves space, reduces power, and dramatically increases data transfer speed.

For a new engineer, understanding HBM's physical architecture is key to grasping why modern GPUs can process AI models so quickly.

⚙️ What is 3D Stacking?

3D stacking is a manufacturing technique where multiple layers of memory chips (called dies) are placed on top of each other, connected by tiny vertical wires called Through-Silicon Vias (TSVs).

  • Traditional approach: Memory chips are placed side-by-side on a circuit board (2D layout). This takes up a lot of space and forces data to travel longer distances.
  • 3D stacking approach: Memory dies are stacked vertically, like pancakes. This reduces the physical footprint and shortens the distance data must travel.

Key benefits of 3D stacking: - Higher density: More memory in the same physical space. - Lower power consumption: Shorter wires mean less energy lost as heat. - Faster data transfer: TSVs provide a direct, high-speed connection between layers.

🛠️ The Role of the Silicon Interposer

A silicon interposer is a thin slice of silicon that acts as a bridge between the GPU (or CPU) and the stacked HBM memory. It's like a high-speed highway that connects the memory skyscraper to the processor.

  • What it does: The interposer contains thousands of tiny wires (called microbumps and redistribution layers) that route data between the GPU die and the HBM stacks.
  • Why silicon? Silicon is used because it can handle extremely fine wiring and has thermal properties similar to the GPU and memory dies, reducing stress from heat expansion.

Physical layout example (simplified): - A GPU die sits in the center of the interposer. - Multiple HBM stacks (each stack is a 3D tower of memory dies) are placed around the GPU die on the same interposer. - The interposer is then mounted onto a standard package substrate, which connects to the motherboard.

📊 Visual Representation: HBM 2.5D Stacked Architecture

This diagram displays the 2.5D structural stack of HBM: DRAM dies are stacked vertically using TSVs and placed on a silicon interposer next to the GPU.

flowchart LR DRAM_Stack["Vertical DRAM Stack (3D)"] -->|TSVs| BaseDie["Logic Base Die"] BaseDie -->|Microbumps| Interposer["Silicon Interposer (2.5D)"] Interposer --> GPU["GPU compute die"] classDef cpu fill:#eafaf1,stroke:#76b900,stroke-width:2px,rx:6px,ry:6px; classDef memory fill:#f0f7ff,stroke:#3498db,stroke-width:1.5px,rx:4px,ry:4px; classDef system fill:#f1f5f9,stroke:#64748b,stroke-width:1.5px; class GPU cpu; class DRAM_Stack,BaseDie memory; class Interposer system;

📊 HBM Stack Architecture — A Closer Look

Each HBM stack consists of several layers:

Layer Description Purpose
Base die Bottom layer, acts as a controller and buffer Manages data flow between the GPU and the stacked memory dies
Memory dies Multiple layers (typically 4, 8, or 12) stacked above the base die Store the actual data (DRAM cells)
Through-Silicon Vias (TSVs) Vertical wires running through all dies Connect the memory dies to each other and to the base die
Microbumps Tiny solder balls between dies Provide electrical and physical connections between layers

How data flows: 1. The GPU sends a request for data. 2. The request travels through the interposer to the HBM stack's base die. 3. The base die routes the request up through the TSVs to the correct memory die. 4. The data travels back down through the TSVs and across the interposer to the GPU.

This entire process happens in nanoseconds, thanks to the short, direct paths.

🕵️ Why HBM Matters for AI

AI workloads are bandwidth-bound — meaning the GPU can process data faster than the memory can supply it. HBM solves this by providing enormous bandwidth in a compact package.

Comparison: Traditional GDDR vs. HBM

Feature GDDR (e.g., GDDR6) HBM (e.g., HBM2e, HBM3)
Layout 2D chips on a PCB 3D stacked dies on an interposer
Bandwidth per chip ~16-64 GB/s ~200-400 GB/s per stack
Power efficiency Moderate High (lower voltage, shorter wires)
Physical footprint Large (many chips spread out) Small (few stacks close to GPU)
Typical use Consumer GPUs, gaming Data center GPUs (NVIDIA A100, H100, B200)

Real-world example: The NVIDIA H100 GPU uses 6 HBM3 stacks, each providing over 3 TB/s of bandwidth. That's enough to transfer the entire contents of a Blu-ray movie in less than a millisecond.

🔧 Key Engineering Takeaways

  • 3D stacking allows memory to be built vertically, saving space and reducing power.
  • Silicon interposers act as the high-speed bridge between the GPU and HBM stacks.
  • Through-Silicon Vias (TSVs) are the vertical wires that connect the stacked memory dies.
  • HBM is essential for AI because it provides the massive bandwidth needed to feed modern GPUs.
  • When you see a GPU spec like "80 GB of HBM3 memory with 3.35 TB/s bandwidth," you're looking at a system with multiple 3D-stacked memory towers connected via a silicon interposer.

📚 Summary

For a new engineer, think of HBM as a memory skyscraper built on a silicon highway. The skyscraper (3D stacking) packs more memory into a tiny area, while the highway (silicon interposer) ensures data moves at lightning speed between the memory and the GPU. This physical architecture is why NVIDIA's data center GPUs can handle the most demanding AI models — and it's a foundational concept for anyone working in AI infrastructure and operations.