11.3c HBM on the H100: 80 GB HBM3 at 3.35 TB/s Bandwidth¶

📘 Context Introduction¶

The NVIDIA H100 Tensor Core GPU is a cornerstone of modern AI infrastructure. One of its most critical components is the High Bandwidth Memory (HBM) subsystem. For new engineers, understanding HBM is essential because it directly impacts how fast AI models can be trained and how large those models can be. The H100 features 80 GB of HBM3 memory with a staggering 3.35 TB/s bandwidth — a massive leap over previous generations. This section breaks down what that means in simple terms.

⚙️ What Is HBM and Why Does It Matter?¶

HBM (High Bandwidth Memory) is a specialized type of memory designed for high-performance computing and AI workloads. Unlike traditional DDR RAM, HBM is stacked vertically in layers (like a skyscraper) and placed very close to the GPU chip. This design allows for:

• Extremely high bandwidth — data can move between the GPU and memory much faster.
• Lower power consumption — less energy is wasted moving data over long distances.
• Compact footprint — saves physical space on the GPU card.

For AI workloads, memory bandwidth is often the bottleneck. A model like GPT-3 (175 billion parameters) requires moving huge amounts of data between memory and compute cores. Higher bandwidth means faster training and inference.

📊 H100 HBM3 Specifications at a Glance¶

🕵️ Breaking Down the Numbers¶

80 GB Capacity — Why 80 GB?¶

• 80 GB is the total memory available on the H100 GPU.
• This capacity is split across 5 HBM3 stacks, each providing 16 GB.
• For context, the previous generation (A100) had 40 GB or 80 GB of HBM2e memory. The H100 standardizes on 80 GB.
• This capacity allows engineers to load larger AI models entirely into GPU memory, avoiding slower transfers to CPU RAM or disk.

3.35 TB/s Bandwidth — What Does That Mean?¶

• 3.35 TB/s means the GPU can read or write 3.35 terabytes of data every second.
• To put this in perspective:
• A typical SSD reads at about 0.5 GB/s (6,700x slower).
• A fast DDR5 RAM stick reads at about 50 GB/s (67x slower).
• This bandwidth is achieved through:
• Wide memory bus — 5,120 individual data lines (compared to 256 or 512 for DDR).
• High clock speeds — HBM3 operates at higher frequencies than previous HBM versions.
• 3D stacking — shorter physical distances reduce signal travel time.

📊 Visual Representation: H100 SXM5 Memory Architecture¶

This diagram displays the memory layout of the H100 SXM5, showing how its central compute die is connected to 5 active stacks of HBM3.

🛠️ How Engineers Use This Information¶

When designing AI infrastructure, engineers consider these memory specs for:

• Model sizing — A model with 70 billion parameters in FP16 (2 bytes per parameter) needs roughly 140 GB of memory. With 80 GB on a single H100, you may need to split the model across multiple GPUs (model parallelism).
• Batch size optimization — Higher bandwidth allows larger batch sizes without slowing down training.
• Data pipeline design — Knowing the bandwidth helps engineers decide how fast data must be fed to the GPU to avoid starvation (the GPU waiting for data).
• Workload selection — Some workloads (like large language models) are bandwidth-bound, while others (like small CNNs) are compute-bound.

🧠 Key Takeaways for New Engineers¶

• HBM3 is not just "more memory" — it is a fundamentally different architecture optimized for speed and efficiency.
• 80 GB capacity is the practical limit for single-GPU model sizes (without model parallelism).
• 3.35 TB/s bandwidth is the speed limit for data movement — exceeding this means the GPU will stall.
• Memory bandwidth often matters more than memory capacity for training large models.
• The H100's HBM3 subsystem is a major reason why it outperforms previous GPUs for AI workloads.

🔍 Real-World Analogy¶

Think of the GPU as a chef in a kitchen: - GPU compute cores = the chef's hands (cooking speed). - HBM memory = the refrigerator (ingredient storage). - Bandwidth = the size of the door and how fast the chef can grab ingredients.

With 80 GB capacity, the refrigerator is large enough to hold ingredients for a huge meal. With 3.35 TB/s bandwidth, the chef can grab ingredients almost instantly — no waiting. This combination means the chef never slows down, even when cooking the most complex recipes (largest AI models).

📚 Further Context¶

This topic is part of the broader GPU memory subsystem discussion. For new engineers, understanding HBM is the foundation for learning about: - Memory bandwidth utilization — how to measure and optimize it. - Model parallelism — splitting models across multiple GPUs. - Data loading pipelines — ensuring data arrives fast enough to keep the GPU busy. - GPU memory management — techniques like gradient checkpointing and mixed precision training.