25.4c In-flight batching (continuous batching): GPU utilization for dynamic inference¶

🧭 Context Introduction¶

When deploying large language models (LLMs) for real-world applications, you will quickly notice that not all requests arrive at the same time. Some users ask short questions, others send long paragraphs. Some requests come in a burst, while others trickle in slowly. Traditional batching — waiting to collect a full batch before processing — wastes valuable GPU compute cycles because the GPU sits idle while waiting for more requests.

In-flight batching (also called continuous batching) solves this problem by allowing the GPU to process different requests at different stages simultaneously. Instead of waiting for an entire batch to finish, the system dynamically adds new requests to the GPU's processing pipeline as soon as a slot opens up. This keeps the GPU constantly busy, dramatically improving utilization and throughput for dynamic, real-world inference workloads.

⚙️ The Problem: Static Batching Wastes GPU Resources¶

Traditional inference engines use static batching, which works like a bus that only departs when every seat is filled. Here's what happens:

• Requests arrive at different times — some early, some late.
• The system waits until a fixed batch size (e.g., 32 requests) is collected.
• All requests in the batch must finish processing before the next batch can start.
• Short requests are held hostage by longer ones in the same batch.

This leads to low GPU utilization because the GPU spends significant time waiting rather than computing.

🛠️ How In-Flight Batching Works¶

In-flight batching treats the GPU like a dynamic assembly line rather than a fixed bus schedule. Here's the core idea:

• Each request is broken into smaller units (typically individual tokens or small groups of tokens).
• The GPU processes tokens from multiple requests simultaneously, interleaving them in a fine-grained manner.
• When one request finishes (e.g., a short question gets its answer), its GPU memory slot is immediately freed.
• A new waiting request is instantly loaded into that freed slot and begins processing.

This means at any given moment, the GPU is working on tokens from many different requests at different stages of completion.

🧩 Key Components¶

• KV Cache Management: Each request maintains its own key-value (KV) cache. In-flight batching dynamically allocates and deallocates KV cache slots as requests enter and leave.
• Scheduling Queue: A scheduler decides which waiting request gets the next available slot, often prioritizing fairness or latency requirements.
• Token-Level Scheduling: The system schedules work at the token level, not the request level, enabling true interleaving.

📊 Visual Representation: Traditional Batching vs. Continuous Batching¶

This diagram contrasts static batching (blocking execution until all sequences exit) with continuous batching (running new requests immediately).

📊 Comparison: Static vs. In-Flight Batching¶

🕵️ Why This Matters for Engineers¶

As an engineer working with AI infrastructure, understanding in-flight batching helps you:

• Maximize GPU ROI: You can serve more users with fewer GPUs because utilization stays high.
• Handle real-world traffic patterns: User requests are never perfectly uniform — in-flight batching adapts naturally.
• Reduce latency spikes: No more "long request blocking short request" problems.
• Scale efficiently: When you add more users, the system gracefully absorbs the load rather than collapsing under queue pressure.

🎯 Practical Impact¶

• A single GPU using static batching might serve 10 requests per second with high latency variance.
• The same GPU using in-flight batching can serve 30–50 requests per second with lower and more consistent latency.
• This means 3–5x throughput improvement without buying additional hardware.

🧰 Implementation in NVIDIA Ecosystem¶

NVIDIA's TensorRT-LLM includes native support for in-flight batching. When you configure a TensorRT-LLM inference server:

• The engine automatically manages KV cache slots across active requests.
• The scheduler handles dynamic addition and removal of requests.
• You can tune parameters like max batch size and max sequence length to balance throughput and memory usage.

For production deployments, NVIDIA NIM (NVIDIA Inference Microservices) leverages TensorRT-LLM with in-flight batching enabled by default, providing an optimized inference endpoint without requiring manual configuration.

✅ Key Takeaways¶

• In-flight batching keeps the GPU constantly busy by interleaving tokens from multiple requests.
• It eliminates GPU idle time caused by waiting for batches to fill or finish.
• It handles variable-length requests gracefully — short requests finish quickly without being blocked.
• Memory is used efficiently — slots are freed the moment a request completes.
• TensorRT-LLM and NVIDIA NIM implement this natively, making it easy to deploy in production.
• For engineers, this means higher throughput, lower latency, and better hardware utilization — all without changing your application code.

💡 Remember: In-flight batching is the reason modern LLM inference servers can handle thousands of concurrent users on a single GPU. It's one of the most impactful optimizations you can leverage in AI infrastructure.