15.1b East-West traffic: GPU-to-GPU gradient synchronization — bandwidth-sensitive¶
🌐 Context Introduction¶
When training large AI models across multiple GPUs, the GPUs must constantly share information to stay synchronized. This communication happens between GPUs (not between a GPU and a storage server), which is called East-West traffic. The most critical type of East-West traffic is gradient synchronization — where GPUs exchange small but frequent updates during training. This traffic is extremely bandwidth-sensitive, meaning the speed of your network directly impacts how fast your model trains.
⚙️ What Is Gradient Synchronization?¶
- During distributed training, each GPU computes its own set of gradients (mathematical adjustments to the model).
- These gradients must be averaged across all GPUs before the model can be updated.
- This averaging process is called all-reduce or gradient synchronization.
- The traffic pattern is many-to-many: every GPU talks to every other GPU simultaneously.
📊 Why Is It Bandwidth-Sensitive?¶
| Factor | Explanation |
|---|---|
| Frequency | Gradients are exchanged after every training step (often hundreds of times per second) |
| Size | Each gradient exchange can be hundreds of megabytes to gigabytes |
| Burst nature | All GPUs send data at the exact same moment, creating traffic spikes |
| Synchronization | Training stalls until all GPUs finish exchanging — slow network means idle GPUs |
🕵️ The East-West Traffic Challenge¶
- North-South traffic (GPU to storage) is less time-critical — data can be buffered.
- East-West traffic (GPU to GPU) must complete before the next training step can begin.
- If the network is too slow, GPUs sit idle waiting for gradients — this is called network tail latency.
- A single slow link can delay all GPUs in the cluster.
📊 Visual Representation: East-West Intra-Cluster Node-to-Node Traffic¶
This diagram displays East-West data paths connecting multiple compute nodes in a parallel processing environment.
🛠️ How Engineers Address Bandwidth Sensitivity¶
- Use high-bandwidth interconnects: NVIDIA NVLink, InfiniBand (200 Gbps or 400 Gbps), or high-speed Ethernet (RoCEv2).
- Minimize hops: Place GPUs that communicate frequently on the same switch or within the same rack.
- Optimize topology: Use a fat-tree or Dragonfly topology to avoid oversubscription.
- Enable RDMA (Remote Direct Memory Access): Allows GPUs to read/write each other's memory directly without involving the CPU — reduces latency.
- Tune collective algorithms: Use ring all-reduce or tree all-reduce to balance traffic across links.
🔍 Real-World Example¶
Imagine training a large language model on 64 GPUs across 8 servers (8 GPUs per server).
- After each training step, all 64 GPUs must exchange gradients.
- Each GPU sends ~500 MB of gradient data to every other GPU.
- Total traffic per step: 64 × 500 MB = 32 GB moving across the network.
- With a 200 Gbps InfiniBand link, this takes ~1.3 seconds.
- With a 25 Gbps Ethernet link, this takes ~10 seconds — GPUs are idle 8.7 seconds longer per step.
✅ Key Takeaways for New Engineers¶
- East-West gradient sync is the bottleneck in most AI training clusters.
- Bandwidth matters more than latency for this traffic pattern — you need fat pipes.
- Network oversubscription (sharing bandwidth among many GPUs) kills training performance.
- Always check your network bandwidth when troubleshooting slow training — it's often the hidden culprit.
- Use tools like
ib_write_bwornccl-teststo measure actual GPU-to-GPU bandwidth in your cluster.
📚 Summary¶
Gradient synchronization is the heartbeat of distributed AI training. Every training step depends on all GPUs quickly exchanging their gradients. If the East-West network is too slow, your expensive GPUs spend more time waiting than computing. As an engineer, your job is to ensure the network is fast, non-blocking, and dedicated to this traffic — because in AI infrastructure, bandwidth is time, and time is money.