🗺️ Roadmap: The AI Data Center Networking — Moving Petabytes Without Latency¶

Below is the syllabus timeline of lessons in this module. Track your study progress here.

📈 Progress¶

---

📁 14 Enterprise Networking Baseline¶

• 14.1a Layer 1 (Physical): cables, signals, optical fiber, and transceivers
• 14.1b Layer 2 (Data Link): Ethernet frames, MAC addresses, and switches
• 14.1c Layer 3 (Network): IP addresses, routing, and routers
• 14.1d Layer 4 (Transport): TCP (reliable) vs. UDP (fast) — and why AI uses RDMA instead
• 14.1e Layers 5–7 (Session/Presentation/Application): HTTP, TLS, application protocols
• 14.2a How a network switch works: MAC address tables and frame forwarding
• 14.2b VLANs (Virtual LANs): segmenting one physical network into isolated logical networks
• 14.2c IP addressing: Class A/B/C, CIDR notation, and subnetting for AI clusters
• 14.2d ARP (Address Resolution Protocol): resolving IP addresses to MAC addresses
• 14.2e Routing protocols overview: static routes, BGP for large AI cloud fabrics

📁 15 The AI Traffic Problem — Why Standard Networks Fail AI Clusters¶

• 15.1a North-South traffic: client requests to inference servers — latency-sensitive
• 15.1b East-West traffic: GPU-to-GPU gradient synchronization — bandwidth-sensitive
• 15.1c All-to-All communication: the AllReduce collective and its network demands
• 15.1d Incast and fan-in congestion: when 1000 GPUs send to one switch simultaneously
• 15.2a Three-tier (Access-Aggregation-Core): the traditional enterprise design and why it fails AI
• 15.2b Spine-Leaf (Clos) topology: equal-cost paths and predictable low latency
• 15.2c Fat-Tree topology: non-blocking bandwidth in high-performance clusters
• 15.2d Rail topology: dedicated switch rails per GPU across all nodes
• 15.2e Oversubscription ratios: 1:1 (non-blocking) vs 3:1 and when each is acceptable

📁 16 High-Speed Ethernet for AI — Spectrum-X, RoCEv2, and Lossless Fabrics¶

• 16.1a 10/25/40/100/200/400/800 GbE: generational progression and use cases per tier
• 16.1b Optical transceivers: SFP28, QSFP28, QSFP-DD, OSFP — form factors and speeds
• 16.1c Direct Attach Copper (DAC) vs. Active Optical Cable (AOC) vs. pluggable optics
• 16.2a What RDMA is: zero-copy, kernel-bypass data transfer directly between memory regions
• 16.2b RoCEv1 vs. RoCEv2: routable UDP vs. Layer 2 only
• 16.2c RDMA Verbs: the programming interface for Send, Receive, Write, and Read operations
• 16.2d Queue Pairs (QPs): the communication endpoints in RDMA
• 16.3a Why Ethernet drops packets and why RDMA cannot tolerate packet loss
• 16.3b Priority Flow Control (PFC): per-priority pause frames to stop congestion
• 16.3c Explicit Congestion Notification (ECN): end-to-end congestion signaling
• 16.3d DCQCN (Data Center Quantized Congestion Notification): the RDMA congestion algorithm
• 16.3e PFC deadlock risks and mitigation strategies
• 16.4a Spectrum-X architecture: Spectrum-4 switch ASIC + BlueField-3 DPU
• 16.4b Adaptive Routing: dynamically selecting the least-congested path per packet
• 16.4c RoCE Acceleration in BlueField: hardware offload of transport processing
• 16.4d End-to-End congestion control (E2E CC) for AI workloads

📁 17 InfiniBand Architecture — The Gold Standard for AI Cluster Networking¶

• 17.1a InfiniBand vs Ethernet: channel-based vs packet-switched, lossless by design
• 17.1b InfiniBand speeds: SDR, DDR, QDR, FDR, EDR, HDR (200 Gb/s), NDR (400 Gb/s), XDR preview
• 17.1c InfiniBand packet structure: Local Routing Header (LRH) and Global Routing Header (GRH)
• 17.1d Virtual Lanes (VLs): providing QoS and deadlock avoidance in InfiniBand
• 17.2a HCAs (Host Channel Adapters): InfiniBand NICs — ConnectX-6, ConnectX-7 series
• 17.2b InfiniBand switches: Quantum (HDR) and Quantum-2 (NDR) switch chips
• 17.2c Multi-tier IB fabrics: edge switches, leaf switches, spine switches
• 17.2d Optical cabling and MTP/MPO connectors for 400 Gb/s IB links
• 17.3a Subnet Manager (SM): topology discovery, LID assignment, and routing table computation
• 17.3b OpenSM: the open-source subnet manager for InfiniBand
• 17.3c SM high availability: master and standby redundancy
• 17.3d ibstat, ibstatus, iblinkinfo: verifying IB link state and fabric topology
• 17.4a IB RDMA vs. RoCEv2: native vs. encapsulated RDMA
• 17.4b NCCL (NVIDIA Collective Communications Library): the RDMA library for GPU-to-GPU
• 17.4c MPI (Message Passing Interface) over InfiniBand: HPC heritage in AI clusters
• 17.4d GPUDirect RDMA: transferring data directly between GPU VRAM and network HCA

📁 18 NVIDIA Data Processing Units (DPUs) — Offloading Infrastructure¶

• 18.1a How virtualization, security scanning, and storage protocols consume CPU cycles
• 18.1b Quantifying lost compute: 30%+ of host CPU consumed by infrastructure overhead
• 18.1c Disaggregation: separating infrastructure functions from the application CPU
• 18.2a BlueField-2: ConnectX-6 NIC + ARM cores + on-chip memory
• 18.2b BlueField-3: ConnectX-7 NIC + 16 Arm A78 cores + 32 GB LPDDR5 — a full server on a card
• 18.2c P-mode vs DPU-mode: the DPU as a PCIe endpoint vs. autonomous server
• 18.2d OVS (Open vSwitch) hardware offload: accelerating VM networking
• 18.3a DOCA SDK: applications, libraries, and services for DPU programming
• 18.3b DOCA Firewall, DOCA DMA, DOCA RDMA: built-in application frameworks
• 18.3c Deploying storage initiators (NVMe-oF) on the DPU: freeing the host CPU entirely
• 18.3d Zero-Trust security enforcement at the DPU: traffic isolation between tenants

📁 19 Enterprise Storage Architectures for AI¶

• 19.1a Sequential throughput (MB/s): training dataset streaming — dominated by large reads
• 19.1b Random IOPS (I/O per second): checkpoint and metadata access patterns
• 19.1c Latency (microseconds to milliseconds): read latency for inference serving
• 19.1d Queue depth: how many concurrent I/O requests NVMe can process (64K QD)
• 19.2a DAS (Direct Attached Storage): NVMe U.2/E3 drives in the server — maximum speed
• 19.2b NAS (Network Attached Storage): NFS shares for datasets — convenient but limited
• 19.2c SAN (Storage Area Network): block-level access over Fibre Channel or iSCSI
• 19.2d All-Flash Arrays (AFA): enterprise flash storage platforms and their AI use cases
• 19.3a NVMe-oF concepts: extending the NVMe protocol over a network fabric
• 19.3b NVMe/RDMA: NVMe over InfiniBand or RoCEv2 — sub-10μs latency
• 19.3c NVMe/TCP: NVMe over standard TCP/IP — simpler deployment, higher latency
• 19.3d NVMe/FC: NVMe over Fibre Channel for traditional SAN environments
• 19.3e Namespaces, controllers, and queue pairs in NVMe-oF architecture

📁 20 Parallel File Systems — Shared Storage for AI Clusters at Scale¶

• 20.1a NFS limitations: single server metadata bottleneck, no parallel I/O
• 20.1b The thundering herd problem: 1000 GPUs opening the same dataset simultaneously
• 20.1c NFS tuning workarounds and why they are insufficient beyond 10 Gb/s
• 20.2a Lustre: the dominant open-source parallel file system for HPC and AI
• 20.2b Lustre components: MDS (Metadata Servers), MDTs, OSS (Object Storage Servers), OSTs
• 20.2c Lustre striping: distributing a single file across multiple OSTs for parallel reads
• 20.2d IBM Spectrum Scale (GPFS): enterprise parallel filesystem with global namespace
• 20.2e WEKA: all-flash parallel file system designed natively for AI workloads
• 20.2f VAST Data: disaggregated NFS+S3 architecture for massive unstructured AI datasets
• 20.3a S3 API: the universal object storage protocol and its AI tooling ecosystem
• 20.3b MinIO: S3-compatible object storage deployable on-premises
• 20.3c Tiered storage policies: moving cold datasets from NVMe to object storage automatically

📁 21 Accelerating the Storage Pipeline — GPUDirect Storage¶

• 21.1a Standard path: NVMe → PCIe → System RAM (CPU bounce buffer) → PCIe → GPU VRAM
• 21.1b The CPU copy bottleneck: memory bandwidth consumed by unnecessary staging
• 21.1c How this limits GPU utilization during data-intensive training
• 21.2a GDS architecture: DMA engine reads from NVMe directly into GPU BAR memory
• 21.2b NVIDIA Magnum IO SDK: the suite of IO acceleration libraries
• 21.2c cuFile API: the GDS programming interface for AI frameworks
• 21.2d Performance improvements: up to 2x storage throughput with GDS enabled
• 21.2e Requirements: compatible NVMe drivers, supported Linux kernels, GDS package
• 21.3a Combining GDS and GPUDirect RDMA for network storage servers
• 21.3b Storage fabric scenarios: Lustre over InfiniBand with GDS enabled
• 21.3c Dataset pre-caching strategies and prefetch pipelines