25.4d Paged KV cache: managing memory for thousands of concurrent inference requests¶

🧠 Context Introduction¶

When deploying large language models (LLMs) for real-world applications, engineers often face a critical challenge: how to serve thousands of users simultaneously without running out of GPU memory. Each inference request generates a Key-Value (KV) cache — a temporary storage of attention computations that grows with every new token generated. Without careful management, this cache quickly consumes all available memory, causing requests to fail or slow down dramatically.

The Paged KV cache is a memory management technique inspired by operating system virtual memory paging. It breaks the KV cache into fixed-size blocks (pages) that can be allocated, freed, and shared across multiple requests. This dramatically improves memory utilization and enables serving many more concurrent users than traditional contiguous memory approaches.

⚙️ The Problem: KV Cache Memory Explosion¶

• Each inference request maintains a KV cache that grows linearly with the number of generated tokens.
• For a 7B parameter model with a 4096-token context, a single request can consume ~1-2 GB of GPU memory just for the KV cache.
• With thousands of concurrent requests, memory demands quickly exceed GPU capacity (typically 80 GB on an A100 or H100).
• Traditional contiguous memory allocation leads to fragmentation — small gaps between allocated caches that cannot be reused efficiently.

📊 How Paged KV Cache Works¶

The Paged KV cache divides the KV cache into fixed-size blocks (pages) , typically 16 or 32 tokens per page. This approach offers three key benefits:

🧩 Block-Level Allocation¶

• Instead of reserving a large contiguous block for each request, memory is allocated page-by-page as needed.
• Pages are stored in a block table that maps logical token positions to physical memory addresses.
• This eliminates the need to pre-allocate for the maximum possible sequence length.

🔄 Dynamic Growth and Shrinkage¶

• As a request generates more tokens, new pages are allocated on demand.
• When a request completes, its pages are immediately freed and returned to a global pool.
• Memory pressure is reduced because pages are only held for active, generating requests.

🤝 Page Sharing Across Requests¶

• Multiple requests that share the same prompt prefix (e.g., system instructions) can share the same KV cache pages.
• This is especially powerful in chatbot applications where many users start with identical context.
• Shared pages are reference-counted and only freed when all sharing requests complete.

🛠️ Comparison: Traditional vs. Paged KV Cache¶

📊 Visual Representation: PagedAttention KV Cache Block Mapping¶

This diagram displays PagedAttention, mapping logical key-value (KV) sequence tokens to non-contiguous physical DRAM memory pages.

🕵️ Real-World Impact: Serving Thousands of Requests¶

When using a Paged KV cache with TensorRT-LLM, engineers observe:

• 3-10x improvement in concurrent request capacity on the same GPU hardware.
• Reduced latency variance — memory pressure is more predictable because pages are managed uniformly.
• Graceful degradation under load — instead of crashing, the system can queue requests until pages become available.
• Better batch utilization — the inference engine can pack more requests into each batch because memory is no longer the primary bottleneck.

🔧 Practical Considerations for Engineers¶

🧠 Memory Budget Planning¶

• Calculate the page size based on model dimensions (hidden size, number of layers, number of attention heads).
• A typical page holds 16 or 32 tokens worth of KV data. For a 7B model, each page consumes approximately 2-4 MB.
• The total number of available pages = GPU memory / page size. Reserve ~20% of memory for model weights and activations.

📈 Monitoring and Tuning¶

• Monitor page utilization — the ratio of allocated pages to total available pages. High utilization (>90%) indicates memory pressure.
• Track page allocation rate — how quickly pages are being consumed per second. This correlates with request throughput.
• Adjust max sequence length and batch size to match your workload's typical token generation patterns.

🔄 Integration with TensorRT-LLM¶

• The Paged KV cache is built into TensorRT-LLM's inference engine and is enabled by default.
• Engineers configure it through the build configuration when creating the TensorRT engine.
• Key parameters include:
• max_num_tokens — the total number of tokens across all requests in a batch
• max_batch_size — the maximum number of concurrent requests
• page_size — the number of tokens per page (typically 16 or 32)

📝 Example Configuration (for reference)¶

max_batch_size = 256
max_num_tokens = 8192
page_size = 32

📤 Output: This configuration allows up to 256 concurrent requests, with a total of 8192 tokens distributed across all requests, using 32-token pages.

🎯 Key Takeaways for New Engineers¶

• Paged KV cache is essential for scaling LLM inference to thousands of concurrent users.
• Think of it like virtual memory — pages are allocated on demand, shared when possible, and freed immediately after use.
• Memory efficiency directly translates to higher throughput and lower cost per request.
• TensorRT-LLM handles the complexity — engineers only need to configure page size and capacity limits.
• Monitor page utilization to identify memory bottlenecks before they affect user experience.

📚 Further Learning Path¶

• Study the vLLM project (the original research that introduced PagedAttention and Paged KV cache).
• Experiment with TensorRT-LLM's example configurations for different model sizes (7B, 13B, 70B).
• Practice monitoring GPU memory usage with nvidia-smi and TensorRT-LLM's built-in profiling tools.
• Understand how prefix caching works — it's a complementary technique that caches shared prompt prefixes across requests.

The Paged KV cache is one of the most impactful innovations in LLM inference optimization. Mastering it will give you a significant advantage in building scalable, cost-effective AI infrastructure.