LLM Inference Systems

Before transformers, KV cache, and GPU serving, language models had a smaller job: look at context and predict what text should come next.

Transformers were not the first attempt to model language. They arrived after rules, n-grams, recurrent networks, LSTMs, seq2seq models, and attention exposed the same bottleneck: sequence processing was too slow and too forgetful.

N-gram models were useful because they turned language into counts. They failed because a fixed local window cannot remember distant context, understand meaning, or generalize well to unseen phrases.

RNNs improved on local n-gram counting by carrying a hidden state through the text. The tradeoff was dependency: token 10 cannot be processed until token 9 has updated the state.

Plain RNNs rewrote one hidden state at every step. LSTMs added gates and a cell state, giving the model a cleaner path for preserving useful information over longer spans.

RNNs and LSTMs gave language models memory, but that memory still had to pass through one step after another. Long context made useful clues fade, compete, and arrive too slowly.

Attention changed the sequence-model bottleneck. Instead of forcing every old clue through one running state, it let the model look back and score which earlier tokens mattered now.

Self-attention is attention applied inside one sequence. Each token builds a better representation by comparing itself with the other tokens in the same input.

Self-attention changed the shape of sequence processing. Instead of moving one state through the text step by step, it builds token-to-token scores for the whole sequence with matrix operations.

A transformer is a stack of repeated blocks that use self-attention, small neural networks, residual paths, and normalization to turn token vectors into context-aware predictions.

Transformers replaced RNNs because they gave tokens shorter paths to context and gave training hardware a more parallel shape. The tradeoff was higher attention cost.

Embeddings turn tokens into learned vectors. The vector is not a dictionary definition; it is a set of numbers the model can use as the starting point for computation.

Self-attention can compare tokens, but attention alone does not know their order. Transformers need positional information so the same tokens in different positions can mean different things.

Query, key, and value vectors are three learned views of the same token representation. Attention uses queries and keys to decide what matches, then mixes values to carry information forward.

Attention separates matching from information flow. Queries and keys decide which tokens are relevant, while values carry the information that gets mixed into the updated token representation.

Multi-head attention runs several learned attention views in parallel. Each head uses its own projections, then the model combines the results into one updated token representation.

Multiple attention heads help because one token may need several kinds of context at the same time. Heads give the layer several learned ways to score and mix tokens before the results are combined.

After attention lets tokens exchange context, the feed-forward network works on each token separately. It is the per-token thinking step inside a transformer block.

Residual connections let a transformer layer add a transformation without forcing the original representation through a narrow gate. They preserve a route around attention and feed-forward steps.

VRAM planning for LLMs is not just about model weights. Long context windows can make the KV cache as expensive as the model itself.

Bigger GPUs help, but fast LLM serving is mostly a scheduling and KV cache problem. PagedAttention, block allocation, batching, and prefix caching decide how much useful work fits in memory.

Speculative decoding speeds up LLM inference by letting a cheap draft model guess several tokens, then asking the expensive target model to verify those guesses in one parallel pass.

If language models can be compiled directly into silicon, inference stops looking like software running on a GPU and starts looking like the computer itself.