11.5a Register file → Shared Memory → L1 Cache → L2 Cache → HBM¶
📘 Context Introduction¶
When you run an AI workload on an NVIDIA GPU, data doesn't just sit in one place. It flows through a carefully designed memory hierarchy — from the fastest, smallest storage (registers) to the largest, slowest storage (HBM). Understanding this flow is critical for engineers who want to optimize performance, reduce latency, and avoid memory bottlenecks.
Think of it like a factory assembly line: the fastest tools are kept right at the worker's fingertips (registers), while bulk materials are stored in a warehouse far away (HBM). The goal is to keep the "workers" (GPU cores) busy by moving data efficiently through each level.
⚙️ The Memory Hierarchy — A Quick Overview¶
Each level in the hierarchy trades speed for capacity:
| Memory Level | Location | Speed (Relative) | Capacity | Managed By |
|---|---|---|---|---|
| Register File | Inside each Streaming Multiprocessor (SM) | Fastest (~1 cycle) | Very small (256 KB per SM) | Compiler / Hardware |
| Shared Memory | On-chip, per SM | Very fast (~5-10 cycles) | Small (up to 164 KB per SM) | Programmer (explicit) |
| L1 Cache | On-chip, per SM | Fast (~10-20 cycles) | Small (configurable, up to 128 KB) | Hardware (automatic) |
| L2 Cache | On-chip, shared across all SMs | Moderate (~100-200 cycles) | Larger (up to 40 MB on modern GPUs) | Hardware (automatic) |
| HBM (High Bandwidth Memory) | Off-chip, on the GPU package | Slowest (~300-800 cycles) | Largest (up to 80 GB on enterprise GPUs) | Programmer / System |
🕵️ Detailed Breakdown of Each Level¶
📍 Register File¶
- What it is: The smallest and fastest memory, located directly inside each GPU core (CUDA core).
- How it works: Each thread running on the GPU gets its own private set of registers. No other thread can access them.
- Why it matters: If a thread runs out of registers, data "spills" into slower memory (local memory in L1/L2), hurting performance.
- Key fact: The register file is split among all active threads on an SM. More threads per SM means fewer registers per thread.
📍 Shared Memory¶
- What it is: A user-managed cache on each SM. All threads in the same thread block can access it.
- How it works: Programmers explicitly load data into shared memory using CUDA code. This is ideal for data that multiple threads need to reuse.
- Why it matters: Shared memory is about 10x faster than global memory (HBM). Using it well can dramatically speed up operations like matrix multiplications.
- Key fact: Shared memory and L1 cache share the same physical on-chip memory. You can configure the split (e.g., 48 KB shared + 16 KB L1, or vice versa).
📍 L1 Cache¶
- What it is: An automatic, hardware-managed cache on each SM. It stores recently accessed data from global memory.
- How it works: When a thread reads data from HBM, a copy is cached in L1. If another thread requests the same data, it's served from L1 instead of going back to HBM.
- Why it matters: L1 reduces latency for repeated memory accesses. It's transparent to the programmer — the hardware handles it.
- Key fact: L1 is typically 32 KB to 128 KB per SM, depending on the GPU architecture (e.g., Ampere, Hopper).
📍 L2 Cache¶
- What it is: A larger, shared cache that sits between all SMs and HBM.
- How it works: If data is not found in L1, the request goes to L2. L2 caches data that is frequently accessed by multiple SMs.
- Why it matters: L2 reduces the number of expensive HBM accesses. It's especially important for workloads where multiple SMs share data (e.g., convolution layers in neural networks).
- Key fact: L2 cache size varies by GPU model. For example, the NVIDIA A100 has 40 MB of L2 cache.
📍 HBM (High Bandwidth Memory)¶
- What it is: The main global memory of the GPU. It's physically separate from the GPU chip but connected via a high-speed interface.
- How it works: All data (model weights, input tensors, activations) ultimately resides in HBM. The GPU cores read from and write to HBM through the cache hierarchy.
- Why it matters: HBM provides massive bandwidth (up to 2 TB/s on modern GPUs), but latency is high. Efficient AI workloads minimize HBM accesses by maximizing cache reuse.
- Key fact: HBM is stacked vertically (3D) to save space and reduce power. It's the reason modern GPUs can handle large AI models.
📊 Visual Representation: GPU Memory Hierarchy and Latency Scales¶
This diagram displays the GPU memory hierarchy, ordering registers, caches, shared memory, and HBM by speed and storage capacity.
🛠️ Practical Implications for Engineers¶
- Optimize for data locality: Keep frequently accessed data in shared memory or registers. Avoid random access patterns that cause cache misses.
- Monitor memory usage: Use NVIDIA profiling tools (like ncu or Nsight Systems) to see how much time threads spend waiting for memory. High "memory stall" cycles indicate poor data locality.
- Tune the L1/Shared Memory split: If your kernel uses a lot of shared memory, increase its allocation. If it benefits from automatic caching, give more space to L1.
- Understand occupancy: More threads per SM can hide memory latency, but fewer registers per thread may cause spills. Balance is key.
📊 Simple Analogy — The Office Desk¶
| Memory Level | Office Analogy |
|---|---|
| Registers | Pens and sticky notes in your hand |
| Shared Memory | A shared whiteboard your team uses |
| L1 Cache | Your personal desk drawer |
| L2 Cache | The filing cabinet in the corner of the room |
| HBM | The warehouse across the street |
You want to keep the most frequently used items in your hand (registers) or on the whiteboard (shared memory). Only go to the warehouse (HBM) when absolutely necessary.
✅ Key Takeaway¶
The memory hierarchy is a trade-off between speed and capacity. As an engineer, your job is to understand where your data lives at each stage of execution and design your kernels to maximize reuse at the fastest levels. This is the foundation of GPU performance optimization for AI workloads.