28.3a MPS architecture: a daemon that funnels multiple CUDA contexts through one control thread¶

🌱 Context Introduction¶

When multiple applications or processes need to share a single GPU, they normally each create their own separate CUDA context. This can lead to inefficiencies because the GPU must switch between these contexts, wasting memory and compute cycles. NVIDIA's Multi-Process Service (MPS) solves this by introducing a daemon that acts as a traffic controller — it funnels all those separate CUDA contexts through a single control thread, making the GPU work more efficiently.

Think of it like a single checkout lane at a store: instead of having multiple cashiers (each with their own register) handling one customer at a time, MPS is like having one very fast cashier who can process multiple customers' items simultaneously without switching registers.

⚙️ How MPS Architecture Works¶

The MPS architecture consists of three key components working together:

• MPS Daemon (nvidia-cuda-mps-control): This is the central process that runs on the GPU node. It creates a single CUDA context that all client processes share.

• MPS Client: Each application or process that wants to use the GPU becomes an MPS client. Instead of creating its own CUDA context, it connects to the MPS daemon.

• MPS Control Thread: This is the single thread inside the daemon that manages all GPU work submissions from every connected client. It serializes and submits work to the GPU in an efficient, batched manner.

The flow looks like this:

1. Each client application sends its GPU work requests (kernels, memory operations) to the MPS daemon.
2. The daemon's control thread collects these requests from all clients.
3. The control thread submits the work to the GPU as a single, unified stream.
4. The GPU processes the work without needing to switch between different contexts.

🛠️ Key Benefits of the MPS Architecture¶

• Reduced GPU Context Switching: Since all clients share one context through the daemon, the GPU doesn't waste time switching between different memory spaces.

• Better GPU Utilization: The single control thread can batch small work from multiple clients together, keeping the GPU busy with meaningful work instead of idle time.

• Lower Memory Overhead: Each CUDA context consumes GPU memory. With MPS, you have one context instead of many, freeing up memory for actual computation.

• Improved Latency for Small Kernels: Small GPU operations from different clients can be submitted more quickly because they don't each need to set up their own context.

📊 MPS vs. Normal Multi-Process GPU Access¶

📊 Visual Representation: Multi-Process Service (MPS) Control Server¶

This diagram displays MPS: combining multiple application CUDA contexts into a single unified driver context to bypass overhead.

🕵️ When to Use MPS vs. Other Options¶

• Use MPS when: You have many small or medium-sized GPU workloads running simultaneously (e.g., multiple inference servers, batch processing jobs). The single control thread reduces overhead significantly.

• Avoid MPS when: You need strict isolation between processes (MPS shares memory space) or when running very large, single-process workloads that already saturate the GPU.

• MPS vs. MIG: MIG (Multi-Instance GPU) provides hardware-level isolation with separate memory and compute units. MPS provides software-level sharing through the daemon. MIG is better for security-sensitive workloads; MPS is better for maximizing utilization of a single GPU.

🚀 Starting and Managing the MPS Daemon¶

To use the MPS architecture, you must start the daemon before launching your applications:

1. Start the MPS daemon by running the control command as root or with appropriate permissions.
2. Set the visible devices for MPS to manage (e.g., a specific GPU or set of GPUs).
3. Launch your applications normally — they automatically connect to the running daemon.
4. Monitor the daemon using the control command to check active clients and GPU utilization.
5. Stop the daemon when finished to free resources.

The daemon runs in the background and automatically manages all connected client processes until you explicitly stop it.

🔍 Important Considerations for New Engineers¶

• The MPS daemon runs as a single process. If it crashes, all connected clients lose their GPU connection.

• MPS works best on GPUs that support concurrent kernel execution (most modern NVIDIA GPUs).

• Not all CUDA features are supported under MPS (e.g., certain debugging tools may not work).

• The daemon consumes some CPU and memory resources itself, so it's not free — but the savings from reduced context switching usually outweigh this cost.

• MPS is transparent to your application code — you don't need to modify your CUDA programs to use it.

✅ Summary¶

The MPS architecture uses a daemon with a single control thread to funnel multiple CUDA contexts from different processes into one shared context. This reduces GPU context switching overhead, lowers memory usage, and improves overall GPU utilization — especially for workloads with many small, concurrent operations. For new engineers, think of MPS as a smart traffic cop that keeps GPU traffic flowing smoothly without the stop-and-go of multiple separate lanes.