13.5b Grace Hopper Superchip: CPU and GPU sharing a coherent memory space

📦 Nvidia GPU Architecture 📖 Multi-GPU Interconnects

🧠 Context Introduction

The Grace Hopper Superchip represents a fundamental shift in how CPUs and GPUs work together. Traditionally, a CPU and GPU have separate memory pools — the CPU uses system RAM (often DDR), and the GPU uses its own dedicated video memory (VRAM). Data must be copied back and forth between these two memory spaces, which creates bottlenecks and wastes energy.

The Grace Hopper Superchip solves this by physically connecting an NVIDIA Grace CPU and an NVIDIA Hopper GPU using NVLink-Chip-to-Chip (NVLink-C2C). This connection allows both processors to share a single, unified memory space. Engineers can now treat the CPU and GPU as one cohesive unit, dramatically simplifying programming and accelerating data-intensive workloads.

⚙️ How It Works — Coherent Memory in Practice

  • Unified Memory Pool: The CPU and GPU both access the same physical memory. There is no need to manually copy data between system RAM and GPU VRAM.
  • Hardware Cache Coherency: The NVLink-C2C interconnect ensures that when the CPU writes data, the GPU sees the updated value immediately — and vice versa. This is managed at the hardware level, not by software.
  • Single Address Space: Both processors use the same virtual memory addresses. A pointer created on the CPU can be directly dereferenced on the GPU without any translation or copying.
  • High Bandwidth: The NVLink-C2C link provides up to 900 GB/s of bidirectional bandwidth, which is 7x faster than PCIe Gen5.

📊 Comparison: Traditional CPU+GPU vs. Grace Hopper Superchip

Feature Traditional CPU+GPU Setup Grace Hopper Superchip
Memory Architecture Separate CPU RAM and GPU VRAM Single coherent memory space
Data Transfer Explicit copy via PCIe (cudaMemcpy) Automatic via hardware coherency
Programming Model Requires manual memory management Simplified — use pointers freely
Bandwidth ~64 GB/s (PCIe Gen5) Up to 900 GB/s (NVLink-C2C)
Latency High (copy overhead) Very low (direct access)
Power Efficiency Lower (extra copies waste energy) Higher (no redundant data movement)

📊 Visual Representation: Grace Hopper Unified Coherent Memory Space

This diagram displays GH200 unified memory coherency, allowing the Grace CPU and Hopper GPU to access each other's memory spaces directly.

flowchart LR GraceCPU["Grace CPU (512GB LPDDR5X)"] -->|NVLink-C2C Coherent Page Migration| HopperGPU["Hopper GPU (96GB HBM3)"] HopperGPU -->|Direct Hardware Page Tables| GraceCPU classDef cpu fill:#eafaf1,stroke:#76b900,stroke-width:2px,rx:6px,ry:6px; classDef memory fill:#f0f7ff,stroke:#3498db,stroke-width:1.5px,rx:4px,ry:4px; classDef system fill:#f1f5f9,stroke:#64748b,stroke-width:1.5px; class HopperGPU cpu; class GraceCPU memory;

🛠️ Why This Matters for Engineers

  • Simpler Code: You no longer need to write complex data migration logic. Just allocate memory once and let both processors use it.
  • Faster Iteration: Prototyping and debugging become quicker because you avoid the "copy, run, copy back" cycle.
  • Larger Models: AI models that previously required splitting across multiple GPUs due to VRAM limits can now run on a single Grace Hopper system, using the combined memory capacity.
  • Real-Time Applications: Applications like autonomous driving or robotics benefit from low-latency, coherent access between sensor processing (CPU) and neural network inference (GPU).

🕵️ Key Technical Details

  • NVLink-C2C Physical Layer: Uses a custom, high-speed SerDes (Serializer/Deserializer) technology to achieve the 900 GB/s bandwidth.
  • Memory Pool Size: The Grace CPU supports up to 480 GB of LPDDR5X memory, which is directly accessible by the Hopper GPU.
  • Cache Hierarchy: Each processor maintains its own caches (L1, L2), but the NVLink-C2C interconnect keeps them coherent. If the GPU modifies a cache line, the CPU's cache is automatically invalidated or updated.
  • Atomic Operations: Both processors can perform atomic read-modify-write operations on shared memory locations, enabling efficient synchronization.

💡 Practical Example for a New Engineer

Imagine you are training a large language model (LLM) that requires 200 GB of memory. On a traditional system, you would need to: 1. Load the model into CPU RAM (say 256 GB available). 2. Copy the model to GPU VRAM (if VRAM is only 80 GB, you must split the model across multiple GPUs). 3. Manage data movement for each training step.

With the Grace Hopper Superchip: 1. Allocate 200 GB of memory once. 2. The CPU loads the training data into this shared memory. 3. The GPU reads the data directly from the same memory — no copies needed. 4. Both processors can update model parameters without any explicit synchronization code.

✅ Summary

  • The Grace Hopper Superchip unifies CPU and GPU memory into a single, coherent space using NVLink-C2C.
  • Engineers benefit from simpler programming, higher performance, and lower power consumption.
  • This architecture is ideal for large-scale AI workloads, real-time inference, and memory-intensive applications.
  • For new engineers, the key takeaway is: think of the CPU and GPU as one processor with one memory pool — not two separate devices that need to communicate.