17.2a HCAs (Host Channel Adapters): InfiniBand NICs — ConnectX-6, ConnectX-7 series¶

🌐 Context Introduction¶

In an AI data center, the network is the nervous system connecting thousands of GPUs. Host Channel Adapters (HCAs) are the specialized network interface cards (NICs) that plug into servers and provide the physical connection to the InfiniBand fabric. Think of an HCA as a supercharged Ethernet NIC — but designed specifically for the ultra-low latency and high bandwidth demands of AI training and HPC workloads.

NVIDIA's ConnectX-6 and ConnectX-7 are the current-generation HCAs powering modern AI clusters. Understanding their capabilities helps you choose the right hardware for your cluster and troubleshoot performance issues.

⚙️ What is an HCA (Host Channel Adapter)?¶

An HCA is a PCIe card installed in a server that:

• Connects the server's CPU/GPU memory to the InfiniBand network
• Handles data transfer with minimal CPU involvement (RDMA — Remote Direct Memory Access)
• Provides the physical port(s) for InfiniBand cables (QSFP or OSFP connectors)
• Supports advanced features like GPUDirect RDMA (GPU-to-GPU communication without touching CPU memory)

Key difference from Ethernet NICs: HCAs offload almost all networking processing to hardware, achieving microsecond-level latency versus millisecond-level latency on traditional TCP/IP.

📊 ConnectX-6 vs ConnectX-7 — Comparison Table¶

🛠️ ConnectX-6 — The Workhorse of Current AI Clusters¶

The ConnectX-6 HCA is the backbone of many production AI clusters today. It supports HDR InfiniBand (200 Gbps per port).

Key capabilities for engineers:

• Dual-port option: Two 200 Gbps ports per card, giving 400 Gbps total bandwidth per HCA
• PCIe 4.0 x16: Provides ~32 GB/s bandwidth to the host — matches the 200 Gbps port speed
• GPUDirect RDMA: Enables direct data transfer between GPU memory and the InfiniBand network, bypassing CPU and system memory
• Adaptive Routing: Automatically balances traffic across multiple paths in the fabric to avoid congestion

Typical deployment: Used in NVIDIA DGX A100 systems and many third-party AI servers.

📊 Visual Representation: Host Channel Adapter (HCA) Card Interface¶

This diagram displays the connection layout of a Host Channel Adapter (HCA) card: bridging the system PCIe bus with InfiniBand network ports.

🚀 ConnectX-7 — The Next-Generation Powerhouse¶

The ConnectX-7 is designed for the latest AI systems like DGX H100 and H200. It supports NDR InfiniBand (400 Gbps per port).

Key improvements over ConnectX-6:

• Double the bandwidth: 400 Gbps per port vs 200 Gbps
• PCIe 5.0 x16: Provides ~64 GB/s to the host — matches the 400 Gbps port speed
• NDR200 mode: Can operate at 200 Gbps per port for backward compatibility with HDR switches
• Enhanced in-network computing: Supports NVIDIA SHARP v3 for collective operations (all-reduce, all-gather) performed inside the network switches
• Better power efficiency: Despite higher performance, power per Gbps is lower

Typical deployment: Used in NVIDIA DGX H100/H200 systems and next-generation AI servers.

🕵️ How to Identify Which HCA is Installed¶

When you log into a server, you can check the HCA model using standard Linux tools.

For reference:

ibstat

📤 Output: Shows InfiniBand device information including: - CA type: ConnectX-6 or ConnectX-7 - Firmware version: e.g., 20.35.1000 - Port state: Active or Down - Link layer: InfiniBand - Rate: 200 Gbps (HDR) or 400 Gbps (NDR)

For reference:

lspci | grep Mellanox

📤 Output: Lists all Mellanox (NVIDIA) devices on the PCIe bus, including HCA model numbers.

🔌 Physical Connections and Cabling¶

Understanding the physical interface helps with cabling and troubleshooting:

• ConnectX-6: Uses QSFP56 connectors (same form factor as QSFP but supports 200G)
• ConnectX-7: Uses OSFP connectors (slightly larger than QSFP) for 400G NDR, or QSFP112 for 200G NDR200 mode

Cabling types: - Passive copper cables: Short distances (up to 3 meters), lower cost - Active optical cables (AOC): Longer distances (up to 100 meters), higher cost - Transceivers + fiber: For distances beyond 100 meters (used in large clusters)

🧠 Practical Considerations for New Engineers¶

When to choose ConnectX-6 vs ConnectX-7¶

• Use ConnectX-6 if your cluster uses DGX A100 or older servers, and your switches are HDR (200G)
• Use ConnectX-7 if you are building a new cluster with DGX H100/H200 or newer, and your switches are NDR (400G)
• Mixing is possible: ConnectX-7 can run in NDR200 mode (200G) to connect to HDR switches, but you lose half the bandwidth

Common troubleshooting tips¶

• Link not coming up: Check cable type (HDR vs NDR), check port speed configuration, verify switch port is configured for the same speed
• Performance lower than expected: Verify PCIe link width and speed (use lspci -vvv and look for "LnkSta" line), check for thermal throttling
• GPU Direct not working: Ensure NVIDIA drivers and CUDA are installed, verify GPU and HCA are on the same PCIe root complex

📚 Summary¶

As an engineer starting in AI infrastructure, remember that the HCA is the gateway between your server and the cluster network. Choosing the right one and understanding its capabilities directly impacts training job performance and cluster utilization.