One token can need more than one clue

Multi-head attention gives a transformer several learned attention views in the same layer.

The natural question is:

why not just use one bigger attention head?

The short answer is that a token may need to use several kinds of context at once.

One attention pattern may be too narrow.

Multiple heads help because different relationships can matter for the same token at the same time.

The important word is "learned."

The model is not born with a head for grammar, a head for dates, and a head for code. Training adjusts the projections so different heads can become useful in different ways.

A tiny example

Read this short line:

the cache key changed after deploy

At changed, the model may need several clues:

what changed?        -> key
what kind of key?    -> cache
when or why?         -> after deploy
what is the action?  -> changed itself

One attention head might put more weight on:

cache key

Another might put more weight on:

deploy

Another might keep the local verb information strong.

Together, those views can produce a richer updated representation:

changed' = action + object + surrounding cause

That is the kind of situation where multiple heads help.

The heads do not vote on words

It is tempting to picture each head as a little expert that votes for the next token.

That is not quite right.

Heads do not directly say:

next token should be "failed"

Inside a transformer block, heads update token representations.

Those updated representations still pass through other parts of the model:

attention
feed-forward network
residual paths
normalization
more transformer blocks
final logits

So a head is not a final answer machine.

It is one way of moving contextual information between tokens.

Why one big head is different

Imagine using one large attention head.

For a token, that head computes one attention distribution over the allowed tokens.

In rough form:

changed -> cache: 0.35, key: 0.40, deploy: 0.15, changed: 0.10

That single distribution has to blend the context into one weighted mix.

With several heads, the layer can keep several mixes separate before the output projection combines them.

head 1 -> cache/key mix
head 2 -> deploy mix
head 3 -> self/local mix

This gives the layer more flexibility.

The output projection can then decide how to combine those separate pieces.

The value is not just more parameters. It is several separate attention distributions before the layer blends the results.

A code-like example

Consider:

if retry_count > limit: raise TimeoutError

At raise, useful context may include:

condition: retry_count > limit
action: raise
exception: TimeoutError
syntax: if ... :

Different heads can attend through different learned views of the same tokens.

One view might connect raise with TimeoutError.

Another might connect it with the condition.

Another might preserve local syntax.

Again, these are not assigned jobs. They are possible behaviors that can emerge because each head has its own projections.

The failure mode it avoids

The failure mode is a crowded single view.

If one attention distribution has to capture every useful relationship, it may blur them together.

For the cache sentence, one blended view might know that cache, key, and deploy all matter, but not preserve the different reasons they matter.

That can make the updated representation less clean.

Multiple heads reduce that pressure by letting the layer form several smaller context summaries.

one crowded summary
vs
several learned summaries, then a learned blend

That does not mean the model always gets the right answer. It means the architecture gives the layer more room to represent different relationships.

The tradeoff

Multiple heads are not free.

Each head has its own Q, K, and V projections and its own attention scores.

During inference, decoder models also keep key and value vectors in the KV cache for each layer and head.

So heads affect:

compute
memory bandwidth
KV cache layout
implementation complexity

With the same model dimension, more heads usually means smaller heads. That means head count is not a simple "more heads equals proportionally more KV cache bytes" rule. But it still changes the per-head layout and the attention kernels used during serving.

There is also a quality tradeoff.

With a fixed model dimension, more heads usually means each head is narrower.

Too few heads can limit the number of separate attention views.

Too many heads can make each view too small or redundant.

Model builders choose the number of heads as part of the architecture, and serving systems have to support that choice efficiently.

Why this matters for inference systems

At serving time, the model is not retraining the heads.

The projections are fixed weights.

For every prompt and every generated token, the server has to run the attention math that those weights define.

In prefill, many token positions can be processed together.

In decode, the model generates one new token at a time, but that token still attends over the cached keys and values from earlier tokens.

The number of heads shapes that work.

heads -> per-head Q/K/V structure to compute and cache
longer context -> more cached tokens for each head to attend over

This is why attention architecture eventually becomes a systems topic.

The design that helps the model represent language also creates concrete compute and memory patterns for inference servers.

The mental model

Think of a token as trying to update its note about the sentence.

One head gives it one highlighter color.

Multiple heads give it several highlighter colors.

One color might mark the object.

Another might mark timing.

Another might mark local structure.

Then the layer rewrites the note into one clean vector.

That is the mental model:

several learned highlights
one combined representation

Quick check

Why do multiple attention heads help?

A. They let one layer compute several learned attention views before combining them
B. They hard-code one grammar rule per head
C. They let the model generate all future tokens at once
D. They remove the need for positional information

The best answer is A.

Multiple heads give the layer several learned ways to gather context.

Next step

Multiple heads explain how attention can use several token relationships in one layer.

Next: What Is a Feed-Forward Network in a Transformer?

That article looks at the other major sublayer inside a transformer block: the per-token network that transforms each token after attention has mixed in context.