32.2b BCM head node architecture: redundant management servers¶
In any AI cluster, the head node is the brain of the operation. It handles job scheduling, user access, monitoring, and configuration management. If that single node fails, the entire cluster can become unresponsive. To prevent this, NVIDIA Base Command Manager (BCM) supports a redundant management server architecture. This design ensures that if the primary head node goes down, a secondary node takes over automatically, keeping your AI workloads running without interruption.
🧠 Context: Why Redundancy Matters¶
In a production AI environment, downtime is costly. Training jobs can run for days or weeks, and losing the management plane means losing visibility and control. Redundant management servers solve this by:
- Eliminating a single point of failure for cluster management.
- Providing automatic failover so engineers don’t need to manually intervene.
- Maintaining job queues and logs even during a head node outage.
⚙️ How Redundant Management Servers Work¶
BCM implements redundancy using an active-passive model. Here’s the breakdown:
- Active (Primary) Head Node: Handles all cluster management tasks — scheduling jobs, monitoring nodes, managing licenses, and serving the web UI.
- Passive (Standby) Head Node: Runs in the background, constantly syncing state and configuration from the active node. It does not serve requests unless the active node fails.
- Shared Storage: Both nodes access the same shared filesystem (e.g., NFS, GPFS, or Lustre) for critical data like job logs, user home directories, and cluster configuration files.
- Virtual IP (VIP): A floating IP address is assigned to the active node. When failover occurs, the VIP moves to the standby node, so users and compute nodes always connect to the same address.
🛠️ Key Components of the Redundant Architecture¶
| Component | Role in Redundancy |
|---|---|
| Primary Head Node | Actively manages the cluster; runs BCM services (e.g., bcmsvc, bcmd, bcmweb). |
| Standby Head Node | Mirrors the primary’s state; takes over if primary fails. |
| Shared Storage | Stores cluster database, job spool files, and configuration. Both nodes mount this storage. |
| Virtual IP (VIP) | Provides a single, consistent endpoint for all cluster communication. |
| Heartbeat Mechanism | A periodic health check between the two nodes. If the standby stops receiving heartbeats, it initiates failover. |
| Failover Scripts | Automate the process of stopping services on the failed node and starting them on the standby. |
🕵️ Failover Process (Step-by-Step)¶
When the primary head node becomes unavailable (due to hardware failure, network loss, or software crash), the following happens:
- Heartbeat loss detected — The standby node stops receiving the periodic "I'm alive" signal from the primary.
- Failover triggered — The standby node runs a series of pre-configured scripts.
- VIP reassigned — The floating IP address moves from the primary to the standby network interface.
- Services started — BCM services (scheduler, database, web interface) are started on the standby node.
- Compute nodes reconnect — All worker nodes automatically re-establish communication with the new active head node using the VIP.
- Jobs resume — Running jobs continue (if they were unaffected), and queued jobs remain in the queue.
Note: The entire failover typically completes within 30–60 seconds, depending on the size of the cluster and the complexity of the services.
✅ Best Practices for Engineers¶
When setting up redundant management servers in BCM, keep these points in mind:
- Use dedicated network interfaces for the heartbeat link to avoid false failovers caused by network congestion.
- Ensure shared storage is highly available — if the storage goes down, both head nodes become useless. Consider using a RAID-protected NFS server or a clustered filesystem.
- Test failover regularly — simulate a primary node failure during a maintenance window to verify that the standby takes over correctly.
- Monitor both nodes — even the standby node needs health checks. A silent failure on the standby means you lose redundancy.
- Keep configuration in sync — use BCM’s built-in configuration management tools (like
bcm config sync) to ensure both nodes have identical settings.
📊 Comparison: Single vs. Redundant Head Node¶
| Feature | Single Head Node | Redundant Head Nodes |
|---|---|---|
| Fault tolerance | None — single point of failure | High — automatic failover |
| Maintenance window | Requires cluster downtime | Can failover to standby for maintenance |
| Complexity | Low | Moderate (requires shared storage, VIP, heartbeat) |
| Cost | Lower (one server) | Higher (two servers + shared storage) |
| Job continuity | Lost if head node fails | Preserved (jobs continue running) |
🔁 Summary¶
Redundant management servers in BCM provide a robust safety net for your AI cluster. By pairing an active head node with a passive standby, using a virtual IP, and relying on shared storage, you ensure that a single hardware or software failure does not bring your entire infrastructure to a halt. For any production AI environment, this architecture is not optional — it is essential.
Key takeaway: Redundancy is about resilience. It allows engineers to sleep better knowing that their long-running AI training jobs are protected against head node failures.
📊 Visual Representation: BCM Head Node server components¶
This diagram displays the BCM Head Node running cluster configuration daemons, DHCP/TFTP, and CMD service modules.