Attention, but inside the same text

The previous article described attention as a lookup.

At the current position, the model asks:

Which earlier tokens matter right now?

Self-attention uses the same basic idea, but the tokens all come from the same sequence.

input tokens -> compare tokens with each other -> update token representations

That is why it is called self-attention.

Self-attention means tokens in one sequence attend to other tokens in that same sequence.

A tiny example

Take this short sentence:

the cache key changed

The token changed should know something about:

cache
key

The token key should know something about:

cache

The token the may matter less.

Self-attention gives every token a way to ask:

Which other tokens help me understand myself?

self-attentiontokens score other tokens from the same input

It does not just copy words

Self-attention is not a string search.

It does not say:

if token == "cache", always look at "key"

Instead, the model learns soft scores from data.

For one position, the scores might look like:

cache   -> high
key     -> high
changed -> self
the     -> low

The model then mixes information from those tokens into a new representation for the current token.

Every token gets updated

The important part is that this happens for every token.

For:

the cache key changed

the model builds updated representations:

the'     = the     + context from other tokens
cache'   = cache   + context from other tokens
key'     = key     + context from other tokens
changed' = changed + context from other tokens

The apostrophe here just means "updated version."

After self-attention, key is no longer just the word key. It can carry context that says:

this is a cache key

That is the useful part.

Why this helped compared to RNNs

An RNN carries one state through the sequence.

token 1 -> state -> token 2 -> state -> token 3

Self-attention lets tokens compare more directly.

token 1 <-> token 2
token 1 <-> token 3
token 2 <-> token 3

That changes the shape of the problem.

Instead of hoping one running state preserved the right clue, each token can learn which other tokens to use.

Self-attention gives tokens direct routes to each other.

The attention matrix

A simple way to picture self-attention is a table.

Rows are the tokens asking questions.

Columns are the tokens being looked at.

          the  cache  key  changed
the        .     .     .      .
cache      .     .     .      .
key        .     *     .      .
changed    .     *     *      .

The * cells mean higher attention.

In real models, the values are numbers, not stars. But the table shape is the point.

Each token can score the other tokens.

Self-attention can be bidirectional or causal

There is one detail to keep separate.

Self-attention means tokens attend within the same sequence.

It does not automatically say which directions are allowed.

Some models allow a token to look both left and right:

left tokens <- current token -> right tokens

Decoder language models, like GPT-style models, use causal masking during generation. A token can look only at previous tokens, not future tokens.

past tokens -> current token
future tokens are hidden

Both are self-attention. The difference is the mask.

What self-attention did not solve by itself

Self-attention solves the token-to-token lookup problem.

It does not automatically solve everything.

The model still needs:

positional information
feed-forward layers
residual connections
normalization
many stacked blocks

Those pieces become the transformer.

Self-attention is the core interaction pattern, but the full transformer is more than one attention step.

The mental model

Think of a sentence as a small meeting.

In an RNN, one person walks through the room carrying a summary note.

In self-attention, every word gets to ask the room:

Who here helps explain me?

Then each word updates itself using the answers.

That is self-attention: tokens in the same sequence using each other as context.

Quick check

What does self-attention mean?

A. A token attends to tokens from the same input sequence
B. A model ignores all previous tokens
C. A model stores only exact n-gram counts
D. A token can only use one running hidden state

The best answer is A.

Self-attention is attention inside one sequence.

Next step

Self-attention gives tokens a way to compare with each other.

Next: Why Self-Attention Can Look at Every Token.

That article looks at why this comparison can be computed across many token pairs at once, which is one reason transformers scaled better than recurrent models.