11.4c KV cache explosion in long-context inference: why 80 GB fills faster than expected

📦 Nvidia GPU Architecture 📖 GPU Memory Subsystems

🧠 Context Introduction

When deploying large language models (LLMs) for long-context inference — such as processing entire documents, codebases, or multi-turn conversations — engineers often find that GPU memory (VRAM) fills up much faster than expected. A single NVIDIA A100 or H100 with 80 GB of memory can become exhausted in seconds, even with a modest batch size. This phenomenon is known as KV cache explosion.

Understanding why this happens is critical for engineers working with AI infrastructure, as it directly impacts model serving, throughput, and cost.

⚙️ What is the KV Cache?

The KV cache (Key-Value cache) is a memory structure used during autoregressive text generation. When an LLM generates tokens one at a time, it stores the intermediate attention keys and values from previous tokens to avoid recomputing them.

  • Without KV cache: The model recomputes attention for all tokens at every step — extremely slow.
  • With KV cache: The model stores keys and values for each token, then appends new ones as generation proceeds.

The KV cache grows linearly with sequence length and quadratically with batch size and model size.

📊 Why 80 GB Fills Faster Than Expected

Let's break down the math using a common model: LLaMA-2 70B (70 billion parameters).

🧮 The Formula

For each token, the KV cache size is:

KV cache per token = 2 × (number of layers) × (hidden dimension) × (precision bytes) × (batch size)

  • 2 accounts for both keys and values.
  • Number of layers: 80 for LLaMA-2 70B.
  • Hidden dimension: 8192.
  • Precision bytes: 2 bytes for FP16 or BF16.
  • Batch size: number of sequences processed in parallel.

🔢 Example Calculation

For a single sequence (batch size = 1) with 128,000 tokens (a long document):

  • KV cache per token = 2 × 80 × 8192 × 2 × 1 = 2,621,440 bytes ≈ 2.6 MB per token
  • For 128,000 tokens: 2.6 MB × 128,000 = 332,800 MB ≈ 325 GB

✅ 325 GB — far exceeding 80 GB VRAM.

Even with a shorter context of 32,000 tokens: - 2.6 MB × 32,000 = 83.2 GB — still over 80 GB.

🕵️ The Explosion Factors

Factor Impact Why It Matters
🧩 Model size (layers × hidden dim) Linear increase Larger models have more layers and wider hidden dimensions
📏 Sequence length Linear increase Longer prompts and generations consume more memory per token
🔄 Batch size Linear increase Each additional sequence multiplies the cache
🎯 Precision (FP16 vs FP32) 2× or 4× increase Lower precision helps, but FP16 is already standard
🧠 Number of attention heads Indirect impact More heads increase hidden dimension proportionally

📊 Visual Representation: KV Cache Growth during Inference

This diagram shows how the Key-Value (KV) Cache grows linearly with context length and batch size, consuming substantial VRAM.

flowchart LR Token["New Generated Token"] -->|"Store Key & Value"| KVCache["KV Cache Storage in VRAM"] KVCache -->|Linear Growth per Step| VRAM_Cap["VRAM Capacity Limit"] VRAM_Cap --> OOM["Out of Memory / Eviction"] classDef cpu fill:#eafaf1,stroke:#76b900,stroke-width:2px,rx:6px,ry:6px; classDef memory fill:#f0f7ff,stroke:#3498db,stroke-width:1.5px,rx:4px,ry:4px; classDef system fill:#f1f5f9,stroke:#64748b,stroke-width:1.5px; class Token cpu; class KVCache,VRAM_Cap memory; class OOM system;

🛠️ Why This is a Bottleneck for Engineers

  • Out-of-memory (OOM) errors occur even with moderate batch sizes.
  • Throughput drops because you must reduce batch size or sequence length.
  • Cost increases — you may need multiple GPUs or expensive memory optimization techniques.
  • Latency spikes — swapping KV cache to CPU memory is extremely slow.

📈 Visualizing the Growth

For a 70B parameter model with FP16 precision:

  • 1,000 tokens: ~2.6 GB
  • 8,000 tokens: ~20.8 GB
  • 32,000 tokens: ~83.2 GB (exceeds 80 GB)
  • 128,000 tokens: ~325 GB (requires 4+ GPUs)

🧰 Mitigation Strategies

Engineers can use several techniques to manage KV cache explosion:

  • KV cache quantization — Store keys/values in 4-bit or 8-bit format instead of FP16.
  • PagedAttention (vLLM) — Manage KV cache in non-contiguous blocks to reduce fragmentation.
  • Sliding window attention — Only keep recent tokens in cache.
  • Multi-query attention (MQA) — Share keys and values across heads.
  • FlashAttention — Reduce memory reads/writes during attention computation.
  • Context compression — Summarize or prune earlier tokens.
  • Offloading — Move KV cache to CPU memory when not actively used.

✅ Key Takeaways for New Engineers

  • KV cache is the dominant memory consumer in long-context inference, not model weights.
  • 80 GB fills fast because even moderate sequence lengths (32K tokens) can exceed capacity.
  • Always calculate KV cache size before deploying a model for long-context tasks.
  • Use memory-efficient inference engines like vLLM, TensorRT-LLM, or FlashAttention.
  • Monitor VRAM usage with tools like nvidia-smi or NVIDIA DCGM to catch OOM early.

📚 Further Reading

  • NVIDIA TensorRT-LLM documentation on KV cache optimization
  • vLLM: PagedAttention paper (Kwon et al., 2023)
  • FlashAttention: Fast and Memory-Efficient Exact Attention (Dao et al., 2022)
  • Hugging Face Transformers: Long context inference guides

Understanding KV cache explosion is essential for engineers building scalable AI infrastructure. By planning for memory growth and using optimization techniques, you can serve long-context models efficiently on NVIDIA GPUs.