13.2a NVLink 1.0 through 5.0: bandwidth progression from 160 GB/s to 1.8 TB/s per GPU¶

🧠 Context Introduction¶

When training large AI models, a single GPU is rarely enough. Engineers must connect multiple GPUs together so they can share data quickly. NVLink is NVIDIA's proprietary high-speed interconnect that allows GPUs to talk directly to each other — bypassing the CPU and system memory entirely. Over five generations, NVLink has evolved from a modest 160 GB/s per GPU to a staggering 1.8 TB/s per GPU, enabling the massive parallelism required for modern AI workloads like GPT-4 and beyond.

This guide breaks down each generation of NVLink (1.0 through 5.0) in simple terms, focusing on the bandwidth progression and what it means for engineers building AI infrastructure.

⚙️ What Is NVLink and Why Does Bandwidth Matter?¶

• NVLink is a direct GPU-to-GPU connection that creates a high-speed "bridge" between GPUs.
• Bandwidth (measured in GB/s or TB/s) determines how fast data can move between GPUs. Higher bandwidth means:
• Faster training of large models (less waiting for data to arrive).
• Ability to scale to more GPUs without bottlenecking.
• Better utilization of GPU compute power.

📊 NVLink Generations at a Glance¶

Note: Bandwidth values are bidirectional — meaning data can flow in both directions simultaneously. Each generation roughly doubles or triples the previous bandwidth.

🕵️ Generation-by-Generation Breakdown¶

🔹 NVLink 1.0 (160 GB/s) — The Foundation¶

• Introduced with: NVIDIA Pascal P100 GPUs.
• How it works: 4 NVLink links per GPU, each link providing 40 GB/s bidirectional bandwidth.
• What it enabled: First step toward GPU-to-GPU direct communication. Used in early DGX-1 systems.
• Engineer takeaway: This was revolutionary at the time — it allowed 4 GPUs to work together without PCIe bottlenecks.

🔹 NVLink 2.0 (300 GB/s) — Doubling Down¶

• Introduced with: NVIDIA Volta V100 GPUs.
• How it works: 6 NVLink links per GPU, each link providing 50 GB/s bidirectional bandwidth.
• What it enabled: Better scaling for deep learning. DGX-2 systems could connect 16 V100 GPUs.
• Engineer takeaway: The extra links reduced communication overhead, making multi-GPU training more efficient.

🔹 NVLink 3.0 (600 GB/s) — The Ampere Leap¶

• Introduced with: NVIDIA Ampere A100 GPUs.
• How it works: 12 NVLink links per GPU, each link providing 50 GB/s bidirectional bandwidth.
• What it enabled: Full connectivity between 8 GPUs in a DGX A100 system (each GPU connected to every other GPU).
• Engineer takeaway: This was a major milestone — it allowed "all-to-all" communication patterns critical for large model parallelism.

🔹 NVLink 4.0 (900 GB/s) — Hopper's Refinement¶

• Introduced with: NVIDIA Hopper H100 GPUs.
• How it works: 18 NVLink links per GPU, each link providing 50 GB/s bidirectional bandwidth.
• What it enabled: Support for larger GPU clusters (up to 256 GPUs via NVLink Switch systems).
• Engineer takeaway: The increased link count reduced latency and improved bandwidth for distributed training across many GPUs.

🔹 NVLink 5.0 (1.8 TB/s) — The Blackwell Revolution¶

• Introduced with: NVIDIA Blackwell B200 GPUs.
• How it works: 18 NVLink links per GPU, each link providing 100 GB/s bidirectional bandwidth (doubled from 50 GB/s).
• What it enabled: Unprecedented bandwidth for trillion-parameter models. Each GPU can now transfer 1.8 TB of data per second.
• Engineer takeaway: This is a game-changer for AI infrastructure — it allows engineers to build clusters that train models with over 10 trillion parameters efficiently.

📊 Visual Representation: NVLink Bandwidth Generational Progression¶

This flowchart tracks the bidirectional bandwidth scaling per GPU across generations (NVLink 1 to NVLink 5).

🛠️ Practical Implications for Engineers¶

When Designing AI Infrastructure:¶

• NVLink 1.0–2.0: Suitable for smaller models (under 1 billion parameters). Limited scaling.
• NVLink 3.0–4.0: Good for models up to 100 billion parameters. Common in enterprise AI clusters.
• NVLink 5.0: Required for frontier models (1 trillion+ parameters). Used in NVIDIA's DGX B200 and SuperPOD systems.

Key Considerations:¶

• NVLink is not PCIe: NVLink is a dedicated GPU-to-GPU interconnect, while PCIe connects GPUs to the CPU. NVLink is much faster.
• NVLink Switch: For NVLink 4.0 and 5.0, NVIDIA introduced NVLink Switch — a dedicated switch chip that connects many GPUs in a fabric topology.
• Bandwidth vs. Latency: Higher bandwidth doesn't always mean lower latency. NVLink 5.0 also improves latency by 2x compared to NVLink 4.0.

🧪 Simple Analogy¶

Think of NVLink bandwidth like highway lanes:

• NVLink 1.0 = A 4-lane highway (160 cars per second).
• NVLink 3.0 = A 12-lane highway (600 cars per second).
• NVLink 5.0 = An 18-lane superhighway where each lane is twice as wide (1,800 cars per second).

The more lanes (links) and the wider each lane (per-link bandwidth), the faster data flows between GPUs.

✅ Summary¶

• NVLink has evolved from 160 GB/s (NVLink 1.0) to 1.8 TB/s (NVLink 5.0) per GPU — a 11.25x increase in bandwidth.
• Each generation adds more physical links and/or increases per-link speed.
• For engineers, understanding NVLink generations helps in selecting the right GPU hardware for AI workloads.
• NVLink 5.0 is currently the gold standard for large-scale AI training, enabling models with trillions of parameters.

📚 Further Reading (for context)¶

• NVIDIA NVLink official documentation
• DGX platform architecture guides
• GPU performance benchmarks for multi-GPU training

This guide is part of the NVIDIA-Certified Associate: AI Infrastructure and Operations curriculum — designed for engineers new to the field.