13.2c NVLink topology in DGX A100: fully connected 8-GPU mesh¶

🌐 Context Introduction¶

In the world of AI and high-performance computing, a single GPU is often not enough. Training large models like GPT or BERT requires massive parallel processing across multiple GPUs. The NVIDIA DGX A100 is a powerful system that packs 8 A100 GPUs into a single node. But how do these GPUs talk to each other efficiently? That's where NVLink and its unique fully connected mesh topology come into play. This section explains what a fully connected 8-GPU mesh means and why it matters for AI workloads.

⚙️ What is a Fully Connected 8-GPU Mesh?¶

A fully connected mesh means that every GPU is directly connected to every other GPU via dedicated NVLink bridges. In the DGX A100, this creates a complete graph where each GPU has 6 NVLink connections (each with 12 links per GPU), resulting in a total of 12 NVLinks per GPU (6 connections × 2 links each).

• Total GPUs: 8
• Total NVLink connections: 28 (each GPU connects to 7 others, but each connection is counted once)
• Bandwidth per GPU: 600 GB/s (bidirectional)
• Aggregate bandwidth: 4.8 TB/s across all GPUs

This topology eliminates the need for data to hop through intermediate GPUs, reducing latency and maximizing throughput.

📊 Why This Topology Matters for AI Workloads¶

🛠️ How NVLink Connections Work in the DGX A100¶

Each A100 GPU has 12 NVLink lanes (or links). In the DGX A100, these are arranged as follows:

• 6 NVSwitch chips inside the DGX A100 chassis manage the connections
• Each GPU connects to 6 of the 8 GPUs directly via NVLink
• The remaining 2 GPUs are connected through the NVSwitch fabric, but still appear as a fully connected mesh from the software perspective

Key point for engineers: When you run a multi-GPU training script (e.g., using PyTorch Distributed Data Parallel), the framework automatically detects this topology and optimizes communication patterns.

📊 Visual Representation: DGX A100 GPU NVLink Mesh topology¶

This diagram shows the full mesh topology of a 8-GPU DGX A100 system, connected symmetrically via 6 NVSwitch boards.

🕵️ Visualizing the Topology¶

Imagine 8 dots (GPUs) arranged in a circle. In a fully connected mesh, every dot has a direct line to every other dot. This is different from a ring or tree topology where data must pass through intermediate nodes.

• Ring topology: GPU 0 → GPU 1 → GPU 2 → ... → GPU 7 → GPU 0 (data hops through neighbors)
• Fully connected mesh: GPU 0 ↔ GPU 1, GPU 0 ↔ GPU 2, ... GPU 0 ↔ GPU 7 (all direct)

For AI training, the mesh topology is superior because: - All-reduce operations (common in gradient averaging) complete faster - No single point of failure in the communication path - Better utilization of available bandwidth

📈 Practical Implications for AI Training¶

When you launch a training job on a DGX A100, the NVLink mesh enables:

1. Faster model parallelism: Splitting a large model across GPUs (e.g., tensor parallelism) benefits from direct GPU-to-GPU transfers
2. Efficient data parallelism: Gradient synchronization across 8 GPUs happens in near-constant time
3. Mixed-precision training: FP16 and BF16 gradients are transferred at full NVLink speed

Example scenario: Training a 175B parameter model using 8 A100 GPUs. Without NVLink, communication overhead could consume 30-50% of training time. With the fully connected mesh, this overhead drops to under 10%.

🧠 Key Takeaways for New Engineers¶

• NVLink topology in DGX A100 is a fully connected 8-GPU mesh — every GPU talks directly to every other GPU
• Bandwidth is massive: 600 GB/s per GPU, 4.8 TB/s aggregate
• No intermediate hops means lower latency and faster training
• Software handles the complexity — frameworks like PyTorch and TensorFlow automatically use NVLink for communication
• This topology is unique to DGX systems — standard servers with PCIe GPUs use slower, shared bus architectures

🔍 Further Exploration¶

To verify the NVLink topology on a DGX A100 system, you can use the nvidia-smi tool. It will show the NVLink link status and bandwidth for each GPU pair. The output will display 12 active links per GPU, confirming the fully connected mesh.

For reference:

nvidia-smi nvlink --status

📤 Output: Shows each GPU's NVLink connections, link status (Active/Inactive), and bandwidth counters.

This concludes the overview of NVLink topology in DGX A100. Understanding this mesh architecture is foundational for optimizing AI workloads on NVIDIA's flagship AI infrastructure.