The mistake

When people estimate required VRAM for running an LLM, they usually start with model size. That is reasonable, but incomplete.

For a model like Llama 3.1 8B, FP16 weights take roughly 16 GB just to load the model.

The part people often miss is context length.

Model weights are only the starting point

Model weights are mostly fixed for a chosen precision. If you run an 8B parameter model at FP16, the quick estimate is:

8B parameters x 2 bytes ~= 16 GB

That number tells you what it takes to hold the model. It does not tell you what it takes to serve long prompts or long conversations.

KV cache grows with context

During inference, the model stores key/value tensors for previous tokens so it does not recompute attention from scratch every time it generates the next token.

That memory is the KV cache. It grows with:

  • number of layers
  • number of KV heads
  • head dimension
  • precision
  • context length
  • batch size and concurrent requests

A quick Llama 3.1 8B estimate

For Llama 3.1 8B, a rough FP16 KV cache estimate is:

layers x 2 x kv_heads x head_dim x context_tokens x bytes

32 x 2 x 8 x 128 x 256,000 x 2 ~= 32 GiB

So the model weights may be around 16 GB, but a 256k context can need roughly 32 GB just for KV cache.

That means the context window can cost more VRAM than the model itself.

Why this changes serving design

The painful part is that KV cache is not just a one-time cost. If you serve multiple users at the same time, each active sequence needs its own cache.

That is why long-context inference quickly becomes a capacity planning problem:

  • one request at 256k context is expensive
  • multiple long-context requests multiply the pressure
  • batching improves throughput but also increases active cache memory
  • quantizing weights does not automatically solve KV cache size

Practical checklist

When sizing GPUs for an LLM service, ask these questions:

  • What precision are the model weights loaded in?
  • What maximum context length do we actually expose?
  • How many concurrent long-context requests do we support?
  • Is KV cache stored at FP16, BF16, FP8, or another format?
  • Do we need the maximum context for every request, or only a small tier of workloads?

The short version: do not size the GPU from parameter count alone. Context length is GPU memory too.