The Memory Trick Behind Fast LLM Serving
Bigger GPUs help, but fast LLM serving is mostly a scheduling and KV cache problem. PagedAttention, block allocation, batching, and prefix caching decide how much useful work fits in memory.
The wrong mental model
It is tempting to explain LLM serving as a straight line:
request -> model -> response
That is not how a high-throughput server actually behaves.
A serving engine is running a scheduling loop. At every step it asks which requests can move forward, how many tokens can fit in the next batch, and whether there is enough KV cache memory to keep those requests alive.
The secret behind fast LLM serving is not just a bigger GPU. It is smarter memory management.
What the server is really doing
An autoregressive model generates one step at a time. The server is constantly balancing active requests, available memory, and the next token budget.
which requests can run? how many tokens fit? enough KV blocks? run next decode step
For each scheduling step, it has to decide:
- which waiting requests can start
- which running requests can decode another token
- how many prompt or decode tokens fit in the batch
- whether new KV cache space can be allocated
- whether cached prefix blocks can be reused
This is why serving performance often changes dramatically even when the model and GPU stay the same.
Why KV cache allocation is hard
During inference, every active sequence stores key/value tensors for attention. That memory grows as the prompt and generated output grow.
The naive approach is to give each request one large contiguous KV cache region. That sounds simple, but it wastes memory when requests have different lengths or finish at different times.
Imagine two requests:
Request A -> 30 tokens
Request B -> 10 tokens
If the server reserves too much space up front, memory sits unused. If it reserves too little, the request needs to move or fail. If many requests start and finish at different times, the allocator can end up with fragmented space that is technically free but awkward to use.
KV cache memory is dynamic, request-shaped, and constantly changing.
PagedAttention changes the shape of the problem
PagedAttention borrows an operating-system idea: do not require one contiguous allocation per request.
Instead, split KV cache into fixed-size blocks. A request owns a logical sequence of blocks, and those logical blocks can point to physical blocks that are not adjacent in GPU memory.
In vLLM, the block size is configurable through --block-size; common CUDA choices include 16 tokens. The important idea is the abstraction: requests get pages of KV cache instead of one giant slab.
Using the same example:
Request A -> 30 tokens -> 2 blocks -> block0, block1
Request B -> 10 tokens -> 1 block -> block3
The server does not need Request A and Request B to live next to each other in memory. It only needs a block table that says where each request's KV pages are.
The block pool
vLLM's KV cache manager starts with a pool of cache blocks. Free blocks can be tracked in a queue, and blocks can be popped when a request needs more space.
That gives the scheduler a simple question to answer:
new required blocks <= currently free blocks?
If yes, the request can continue. If not, the scheduler has to wait, evict reusable cached blocks, or reject work depending on policy.
Batching does not merge the conversations
During the forward pass, tokens from different requests can be flattened into one batch:
[A1, A2, A3, ..., B1, B2, B3]
That does not mean Request A can attend to Request B.
The attention metadata keeps sequences logically separate. The GPU gets a batch that is efficient to run, while the attention layer still knows which KV blocks belong to each request.
The batch is shared compute. The context is still private to each request.
Prefix caching is the second win
Many production requests share the same beginning:
- system prompt
- tool definitions
- policy text
- few-shot examples
- long retrieved context
Without prefix caching, the server recomputes KV cache for the same prefix again and again.
With prefix caching, already-computed KV blocks can be reused when a new request starts with the same prefix. vLLM does this with a hash-based design: a full block can be identified from its parent block hash, the block's tokens, and extra values such as LoRA IDs, multimodal hashes, or cache salts.
The cache hit is not at the whole-prompt level. It is at the KV block level.
That is why block boundaries matter. If only full blocks are cached, then a request may share ten tokens with an existing request but only hit the blocks that are completely identical.
What this means in practice
If you are building or operating an LLM service, GPU size is only one part of the capacity story.
You also need to understand:
- how many active requests are in the scheduler
- how much KV cache each request needs
- whether your workload has shared prefixes
- whether prompt length or decode length is the bottleneck
- how block size affects waste and reuse
- how eviction behaves when memory pressure rises
The practical takeaway: fast serving is a memory-management problem before it is a GPU-shopping problem.
The best serving systems are not just loading the model onto a large accelerator. They are continuously packing, scheduling, reusing, and freeing KV cache so more useful tokens fit into the same memory budget.
Further reading
Field notes
Sign in to leave a comment.
Sign in