Self-attention became the center

The previous article showed why self-attention can compare many token pairs at once.

That was the important move.

But self-attention by itself is not the whole model.

A transformer wraps self-attention in a repeated structure:

token vectors
  -> self-attention
  -> feed-forward network
  -> repeat
  -> prediction

A transformer is a stack of blocks that repeatedly lets tokens exchange information, refine their representations, and pass the result upward.

The word "transformer" can sound bigger than it is. At the core, it is a disciplined way to keep improving token vectors.

A tiny example

Start with a short input:

the cache key changed

At the beginning, each token has a vector.

the     -> vector
cache   -> vector
key     -> vector
changed -> vector

After one self-attention step, changed can mix in clues from cache and key.

changed' = changed + context from cache/key

After another block, the model can refine that representation again.

changed'' = better context-aware version

The transformer does not read this sentence by passing one hidden state from left to right. It keeps a vector for each token and updates those vectors through stacked blocks.

self-attentiontokens score other tokens from the same input

What is inside one block

A simplified transformer block has a few repeated parts:

self-attention
feed-forward network
residual connections
normalization

Self-attention lets tokens use other tokens.

The feed-forward network transforms each token's representation on its own.

Residual connections carry the old representation around the new work.

Normalization keeps values in a range that is easier to train.

Do not worry if some of those parts are still fuzzy. The path will unpack them one by one.

For now, the useful picture is:

mix across tokens -> refine each token -> repeat

Why stack many blocks

One attention step can connect tokens.

Many blocks let the model build richer features.

In an early block, the model might learn local relationships:

cache + key

In a later block, it might combine that with a broader event:

cache key changed during deploy

In a still later block, it might use that context to make the next-token distribution sharper:

hit rate started to -> drop

This is not a hand-written pipeline. The model learns the internal features during training.

The stack just gives it room to build them gradually.

The transformer changed the shape of training

RNNs and LSTMs processed sequences as chains.

token 1 -> state -> token 2 -> state -> token 3

Transformers use self-attention over token vectors.

tokens -> token-pair scores -> updated token vectors

That shape works well with large matrix operations.

During training, the model can process many positions from the same sequence together. It can compute many attention scores together. It can update many token representations together.

That does not mean every part of every language-model workload is parallel.

Decoder language models still generate new tokens one at a time, because token 101 depends on the token chosen at position 100.

But the transformer made training and prompt processing far more parallel than a recurrent chain.

The tradeoff

The transformer bought parallelism and direct token-to-token routes.

It paid with attention cost.

For a full attention layer:

100 tokens  -> 10,000 token pairs
1,000 tokens -> 1,000,000 token pairs

That is the same tradeoff from the self-attention article.

The model gets flexible context, but it has to score many possible relationships.

Transformers made long-range lookup easier, but they did not make long context free.

That cost will matter later when the path reaches KV cache, context length, and serving memory.

What a transformer is not

A transformer is not a database of facts.

It is not a grammar engine.

It is not a magic box that reads the whole future.

In a GPT-style decoder model, each position is masked so it can only use earlier positions and itself while training the next-token task.

visible: past and current tokens
hidden: future tokens

The architecture gives the model a powerful way to compute with context. The mask decides which context is allowed.

The mental model

Think of the input as a team of tokens.

In an RNN, one clipboard moves down the line.

In a transformer, every token keeps its own notebook.

Each block does two things:

ask nearby and distant tokens for useful clues
rewrite the notebook with a better representation

Stack enough blocks, and the notebooks become context-rich.

That is the transformer: repeated self-attention and refinement over token vectors.

Quick check

What is the best short description of a transformer?

A. A stack of blocks that use self-attention and feed-forward layers to update token representations
B. A model that stores only exact word counts
C. A model that processes text only through one recurrent hidden state
D. A hand-written set of grammar rules

The best answer is A.

Self-attention is the key interaction pattern, but the transformer is the full repeated architecture around it.

Next step

Now that the block shape is clear, the next question is historical and practical:

Why did this architecture replace RNNs for language modeling?

That is the next article: Why Transformers Replaced RNNs for Language Modeling.