The question after self-attention

The earlier self-attention articles described the shape:

tokens compare with tokens
use the useful ones
build better token representations

That leaves one practical question.

When a token compares with another token, what is being compared?

The answer is query, key, and value vectors.

Query, key, and value vectors are learned projections of a token vector, not text labels stored beside the token.

Start with one token vector

Suppose the model has this token:

key

At a layer inside the transformer, key is not just a word. It is a vector.

key -> [numbers, numbers, numbers, ...]

Self-attention makes three different learned views of that vector:

query vector
key vector
value vector

They come from learned weight matrices.

token vector -> Wq -> query
token vector -> Wk -> key
token vector -> Wv -> value

The names are easy to over-read. They do not mean the model writes down the English word "query" or "value."

They are just vectors with different jobs.

Query asks

The query vector is used by the current position to ask what kind of context it needs.

For a tiny phrase:

the cache key changed

when the model updates changed, its query might behave like:

I need context about what changed

That is not a sentence inside the model. It is a direction in vector space that gets compared with other vectors.

So a query is the asking side of attention.

row token -> query

Key is matched against

Every token also has a key vector.

Keys are what the query gets compared against.

For changed, the query is compared with keys from the allowed tokens:

query(changed) vs key(the)
query(changed) vs key(cache)
query(changed) vs key(key)
query(changed) vs key(changed)

The comparison produces attention scores.

the     -> low
cache   -> high
key     -> high
changed -> self

Value is what gets mixed

Once the model has scores, it uses them to mix value vectors.

That is the third role.

scores from query-key matches
        +
value vectors from tokens
        =
new context vector

If changed pays high attention to cache and key, then the value vectors for cache and key contribute more to the updated representation of changed.

changed' = mix(value(cache), value(key), value(changed), ...)

The scores say how much to use.

The values provide what gets used.

A tiny numeric sketch

Imagine only three allowed tokens:

cache
key
changed

The query for changed is compared with their keys:

key(cache)   -> score 3.0
key(key)     -> score 2.5
key(changed) -> score 1.0

After softmax, those become weights like:

cache   -> 0.57
key     -> 0.35
changed -> 0.08

Then the model mixes values:

0.57 * value(cache)
+ 0.35 * value(key)
+ 0.08 * value(changed)

That mixed vector is context for the changed position.

Why not use one vector for everything?

The model could try to compare token vectors directly and also mix those same vectors.

But attention gives the model separate learned views.

One view can become useful for matching:

does this token answer the current need?

Another view can become useful for content:

what information should this token contribute if selected?

That separation is the point.

It gives the model more flexibility than one all-purpose representation.

The failure mode

The common mistake is to think Q, K, and V are like database columns:

query = the search text
key   = the lookup label
value = the stored record

That metaphor can help for one minute, but it breaks quickly.

In a transformer, all three are dense learned vectors made from token representations.

The key is not a human-readable tag.

The value is not the original word copied into a slot.

The query is not a natural-language question.

Q, K, and V are learned roles in the attention math, not manually assigned meanings.

The mental model

Think of each token as making three cards for itself.

One card says:

what I am looking for

One card says:

what I can match

One card says:

what I can contribute

The model compares the first card from one token with the second card from other tokens.

Then it mixes the third cards according to those scores.

That is query, key, and value attention.

Quick check

In attention, which vectors decide the weights?

A. Queries and keys
B. Values only
C. Token strings directly
D. The next token label

The best answer is A.

Queries are compared with keys to produce attention weights. Those weights are then used to mix values.

Next step

Query, key, and value vectors give attention three learned roles.

Next: Why Attention Uses Q, K, and V.

That article looks at why splitting matching from content makes attention more flexible than a single-vector lookup.