28.2a How time-slicing works: GPU context switching between processes¶
🌐 Context Introduction¶
In many AI workloads, multiple processes or containers may need to share a single GPU. While technologies like Multi-Instance GPU (MIG) provide hardware-level partitioning, time-slicing offers a simpler, software-based approach. Time-slicing allows the GPU to rapidly switch between different processes, giving each one a turn to execute. This is similar to how a CPU handles multiple applications at once — by dividing time into small slices and context-switching between tasks. For new engineers, understanding this mechanism is key to managing GPU utilization without requiring physical partitioning.
⚙️ What is GPU Time-Slicing?¶
- Definition: Time-slicing is a temporal sharing technique where the GPU allocates fixed or variable time intervals (slices) to different processes.
- How it works: The GPU scheduler rapidly switches between processes, giving each one a short burst of execution time.
- No hardware isolation: Unlike MIG, time-slicing does not create separate physical GPU partitions — all processes share the same GPU memory and compute units.
- Transparent to applications: The processes are unaware of the switching; they simply see the GPU as available for their work.
🕵️ How GPU Context Switching Works¶
The core mechanism behind time-slicing is context switching. Here's a simplified breakdown:
- Process Queue: The GPU driver maintains a queue of processes waiting to use the GPU.
- Time Quantum: Each process is assigned a time quantum (e.g., 10 milliseconds) to execute.
- Execution: Process A runs on the GPU for its time slice.
- Preemption: When the time slice expires, the GPU pauses Process A.
- Save State: The GPU saves the current execution state (registers, memory pointers, etc.) of Process A.
- Load State: The GPU loads the saved state of Process B.
- Resume: Process B begins execution from where it left off.
- Repeat: This cycle continues, giving each process fair access to the GPU.
📊 Key Characteristics of Time-Slicing¶
| Feature | Description |
|---|---|
| Granularity | Time slices are typically in the millisecond range |
| Overhead | Context switching adds small latency (microseconds) |
| Memory Sharing | All processes share the same GPU memory (VRAM) |
| Isolation | No memory or fault isolation between processes |
| Fairness | All processes get equal or weighted time slices |
| Performance | Ideal for bursty or low-utilization workloads |
📊 Visual Representation: GPU Time-Slicing Temporal Scheduler¶
This flowchart demonstrates how Timeslicing schedules threads sequentially on GPU hardware over fast time-quantum intervals.
🛠️ When to Use Time-Slicing¶
Time-slicing is best suited for:
- Interactive workloads: Jupyter notebooks, model debugging, or small batch jobs.
- Low-utilization scenarios: When a single process does not fully saturate the GPU.
- Cost-sensitive environments: When you want to maximize GPU usage without buying more hardware.
- Quick prototyping: Engineers testing multiple models on a single GPU.
Avoid time-slicing for: - High-throughput training: Large models that need full GPU memory and compute. - Latency-sensitive inference: Real-time applications where context-switching delays are unacceptable. - Memory-intensive workloads: Processes that require nearly all GPU VRAM.
🔄 Comparison: Time-Slicing vs. MIG vs. MPS¶
| Aspect | Time-Slicing | MIG (Multi-Instance GPU) | MPS (Multi-Process Service) |
|---|---|---|---|
| Isolation | Software-level | Hardware-level | Software-level |
| Memory | Shared | Partitioned | Shared |
| Overhead | Low | None | Low |
| Max partitions | Unlimited (limited by memory) | Up to 7 per GPU | Limited by memory |
| Best for | Bursty workloads | Dedicated, isolated jobs | Concurrent small processes |
🧠 Practical Implications for Engineers¶
- Monitoring: Use tools like nvidia-smi to observe GPU utilization. With time-slicing, you may see high utilization even if individual processes are idle.
- Memory limits: Since memory is shared, one process can consume all VRAM and starve others. Set memory limits using container runtimes (e.g., Docker's --memory flag).
- Performance tuning: Adjust the time quantum via GPU driver settings (e.g., NV_GPU_DEFAULT_TIMESLICE_MS) to balance fairness and throughput.
- Debugging: If a process hangs, it can block the entire GPU. Use nvidia-smi --query to inspect process states.
✅ Summary¶
- Time-slicing enables multiple processes to share a single GPU by rapidly switching between them.
- Context switching involves saving and loading process states, adding minimal overhead.
- It is ideal for low-utilization, bursty, or interactive workloads.
- Unlike MIG, it offers no hardware isolation — all processes share memory and compute.
- Engineers should monitor memory usage and set limits to prevent resource starvation.
This guide provides a foundational understanding of GPU time-slicing for engineers new to AI infrastructure. For deeper dives, explore NVIDIA's official documentation on GPU scheduling and resource management.