17.3c SM high availability: master and standby redundancy¶

📘 Context Introduction¶

In an InfiniBand fabric, the Subnet Manager (SM) is the central brain that discovers, configures, and maintains the network topology. If the SM goes down, the entire fabric can lose connectivity — which is catastrophic for AI training workloads that rely on low-latency, high-bandwidth communication.

To prevent this, InfiniBand supports SM high availability using a master and standby redundancy model. This ensures that if the primary SM fails, a backup SM takes over seamlessly, keeping the fabric operational. For new engineers, think of this like having a co-pilot in an airplane: if the main pilot is unavailable, the co-pilot immediately takes control.

⚙️ How Master and Standby Redundancy Works¶

• The fabric has one active master SM at any time.
• One or more standby SMs are configured to monitor the master.
• Standby SMs do not actively manage the fabric — they only observe.
• If the master SM fails or becomes unreachable, a standby election occurs.
• The standby with the highest priority (configured by the engineer) becomes the new master.
• The new master then recomputes the routing tables and reconfigures the fabric.

🛠️ Key Concepts for New Engineers¶

🕵️ Why This Matters for AI Infrastructure¶

• AI training jobs can run for days or weeks — any network disruption wastes expensive GPU compute time.
• SM high availability ensures zero single point of failure in the fabric.
• Without redundancy, a single SM crash could cause all nodes to lose connectivity, forcing a full job restart.
• Master and standby redundancy is built into InfiniBand — no external clustering software is required.

📊 Visual Representation: Subnet Manager Active-Standby Failover¶

This diagram displays High Availability failover: Standby SMs monitor the Active SM and assume control if failures occur.

📊 Comparison: Master SM vs. Standby SM¶

🔄 Failover Process Step-by-Step¶

1. Master SM sends heartbeats to all standby SMs at regular intervals.
2. Standby SMs detect missing heartbeats — if no heartbeat is received for a timeout period, the master is assumed dead.
3. Standby election begins — each standby checks its priority and compares with others.
4. Highest priority standby declares itself the new master.
5. New master performs a full sweep of the fabric to discover all switches, routers, and endpoints.
6. New master recomputes routing tables and pushes updated forwarding rules to all switches.
7. Fabric resumes normal operation — AI training jobs can continue with minimal disruption.

✅ Best Practices for Engineers¶

• Always configure at least two SMs in your fabric — one master and one standby.
• Assign distinct priorities to avoid tie-breaking issues during elections.
• Run the master SM on a dedicated, reliable server (not on a compute node that may be rebooted).
• Place the standby SM on a different physical server or rack to protect against hardware failures.
• Test failover scenarios before deploying production AI workloads.
• Monitor SM health using fabric management tools (e.g., ibdiagnet, ibstatus).

🧠 Simple Analogy¶

Imagine a traffic control tower at an airport: - The master SM is the primary air traffic controller managing all flights. - The standby SM is a backup controller watching the radar but not giving commands. - If the primary controller gets sick, the backup immediately takes over — flights continue without crashing.

In your InfiniBand fabric, the "flights" are data packets from GPUs, and the "runways" are switch ports. SM high availability keeps those packets moving safely.

📌 Summary¶

• Master and standby redundancy is the standard method for SM high availability in InfiniBand.
• The master SM actively manages the fabric; standby SMs wait to take over.
• Failover is automatic and triggered by missing heartbeats.
• Proper priority configuration and physical separation of SMs are critical for reliability.
• For AI infrastructure, this redundancy prevents costly job interruptions due to network management failures.