One attention view is useful

The previous articles built up self-attention from a simple idea:

each token asks which other tokens matter

For one attention operation, the model builds queries, keys, and values.

Roughly:

query -> what am I looking for?
key   -> what do I contain?
value -> what information do I pass along?

Then each token scores the other tokens and mixes useful value vectors into its own representation.

That is already powerful.

But one attention view has a limitation.

It has to squeeze every useful relationship into one scoring pattern at that layer.

Multi-head attention gives the model several learned attention views at the same time.

The basic shape

Instead of making one set of queries, keys, and values, multi-head attention makes several sets.

For example, a small transformer layer might use four heads:

token vectors
  -> head 1: Q1, K1, V1
  -> head 2: Q2, K2, V2
  -> head 3: Q3, K3, V3
  -> head 4: Q4, K4, V4

Each head runs attention with its own learned projections.

Then the head outputs are joined and projected back into the model's normal vector size.

head outputs -> concatenate -> output projection -> updated token vectors

That is multi-head attention.

It is not a separate model for each head. It is one layer with several parallel attention calculations inside it.

A tiny example

Take this sentence:

the cache key changed after deploy

At the token changed, useful context might include:

cache
key
deploy

One head might give high attention to cache and key.

Another head might give more attention to deploy.

Another might mostly keep the current token's own information.

In rough sketch form:

head 1 for changed -> cache, key
head 2 for changed -> deploy
head 3 for changed -> changed

Do not read this as a hard-coded rule.

The model was not programmed with:

head 1 = nouns
head 2 = causes
head 3 = current word

The heads learn projections from training data. Sometimes a head becomes easy to describe after the fact. Often it does not.

Heads are learned subspaces

A careful way to say it is:

each head sees the token vectors through its own learned projection

That projection changes the space where attention scores are computed.

So the same token can look different to different heads.

For key, one head may represent it strongly as part of:

cache key

Another head may represent it more as a token that can be:

changed
rotated
revoked

These are not separate dictionary entries. They are different learned views of the vector.

Heads are not hard-coded grammar specialists. They are learned projections that let attention score tokens in different subspaces.

That distinction matters because it prevents a common misunderstanding.

You might see diagrams that label heads as "syntax head" or "position head."

Those labels can be useful when analyzing a trained model, but they are descriptions of behavior, not hand-written assignments.

Why the model combines the heads

Each head produces an output vector for every token.

For the token changed, the outputs might carry different kinds of context:

head 1 output -> cache-key context
head 2 output -> deploy context
head 3 output -> current-token context

The transformer does not keep these as separate final answers.

It concatenates them:

[head 1 output | head 2 output | head 3 output | ...]

Then it applies another learned projection.

That output projection lets the layer blend the heads into one vector per token.

changed' = mixed result from all heads

The apostrophe means "updated version."

After multi-head attention, changed can carry information from several attention views at once.

Why heads usually get smaller

There is a practical detail.

If the model dimension is 768 and there are 12 heads, each head often works with a smaller dimension, such as 64.

768 total dimension / 12 heads = 64 dimensions per head

That keeps the total size manageable.

The layer is not usually making 12 full-size attention operations and then multiplying the whole model size by 12.

It splits the attention work across heads, then combines it back.

This is one reason multi-head attention can add multiple views without making the layer explode in size.

The failure mode

Multi-head attention does not guarantee perfect understanding.

Heads can be redundant.

Several heads may learn similar patterns.

Some heads may be useful only in certain layers or contexts.

And more heads are not always better. If the model size is fixed, adding heads can make each head smaller. That can reduce what each head can represent.

There is also the serving cost.

Attention still has to compute token-pair scores. More heads mean more per-head score tables, even if each head is narrower.

long context + attention heads -> structured per-head Q/K/V work

That matters later during inference.

With a fixed model dimension, increasing the number of heads does not simply multiply KV cache bytes, because each head usually gets smaller. But the head count still shapes the memory layout and the kernels the serving system has to run.

The mental model

Think of one attention head as one camera angle on the same text.

It is not a different sentence.

It is the same token sequence viewed through a different learned lens.

Multi-head attention gives the layer several camera angles, then edits those views back into one representation for each token.

That is the mental model:

same tokens
several learned views
one combined update

Quick check

What is a good description of an attention head?

A. A learned Q/K/V projection path that computes one attention view
B. A hard-coded grammar rule inside the transformer
C. A separate language model with its own vocabulary
D. A table that stores exact n-gram counts

The best answer is A.

A head is one learned attention view inside the larger attention layer.

Next step

Multi-head attention gives a transformer several learned views at once.

Next: Why Multiple Attention Heads Help.

That article looks at why those multiple views are useful, and why the benefit is more subtle than assigning one neat job to each head.