The replacement was not just fashion

RNNs and LSTMs were serious models.

They handled sequences better than n-grams. They made memory learnable. They powered useful systems for translation, speech, tagging, and early neural language modeling.

So why did transformers take over?

Because transformers attacked two bottlenecks at the same time:

context bottleneck
hardware bottleneck

Transformers gave tokens shorter paths to useful context and gave training a shape that fit GPUs much better.

That combination mattered more as datasets, models, and sequence lengths grew.

The RNN bottleneck

An RNN moves one state through the text.

token 1 -> state 1
token 2 -> state 2
token 3 -> state 3
token 4 -> state 4

That state is useful. It carries memory.

But it is also narrow.

If token 100 needs a clue from token 7, the clue has to survive through many updates.

token 7 clue -> state 8 -> state 9 -> ... -> state 100

LSTMs made that memory more deliberate, but they did not remove the chain.

The transformer path

A transformer keeps token representations available across the sequence.

For:

the cache key changed during deploy

the model can build attention scores between allowed token positions.

changed -> cache
changed -> key
deploy  -> changed

That gives old clues shorter routes to later positions.

The model is still learning which routes matter. It can still attend to the wrong thing. But the architecture no longer forces every clue through one running state.

A tiny example

Imagine a prompt:

The cache key changed during the deploy, so the hit rate started to

The next token might be:

drop

An RNN must carry cache key changed forward in its hidden state until the end.

A transformer can let the final position attend back to those words.

started to -> cache
started to -> key
started to -> changed

That does not guarantee the right answer. The model still needs training, enough capacity, and the right data.

But the path for the information is cleaner.

self-attentiontokens score other tokens from the same input

The hardware reason

The second reason was speed at scale.

RNN sequence processing has a dependency along time:

state 4 waits for state 3
state 3 waits for state 2
state 2 waits for state 1

You can batch many sequences, but each sequence still has a step-by-step dependency.

Transformers turn much of training into matrix work over token vectors and token-pair scores.

all token vectors -> attention scores -> updated token vectors

GPUs are very good at that kind of large, regular math.

This made it easier to train bigger models on bigger datasets.

Do not overstate the parallelism

There is an important caveat.

Transformers did not make language generation fully parallel.

For a decoder language model, generation still goes one new token at a time:

prompt -> choose token 1
prompt + token 1 -> choose token 2
prompt + token 1 + token 2 -> choose token 3

The next token depends on the previous sampled token.

So the transformer advantage is not:

all output tokens are generated at once

The better statement is:

training and prompt processing can use much more parallel matrix work than an RNN chain

Transformers improved the shape of the computation, but autoregressive decoding still has a sequential loop.

That distinction becomes important later when we talk about prefill, decode, batching, and KV cache.

The tradeoff

Transformers did not win for free.

Full self-attention has a token-pair cost.

sequence length n -> roughly n x n attention scores

That is manageable for shorter contexts and powerful hardware.

It becomes painful as context grows.

RNNs have their own costs, but they do not build the same full pair table at each layer.

So the tradeoff is real:

RNN: cheaper step shape, weaker parallelism, narrow state path
Transformer: stronger parallel training shape, direct context routes, expensive attention

Modern serving systems spend a lot of engineering effort managing the transformer side of that tradeoff.

Why language modeling favored transformers

Language modeling rewards context.

The model is constantly asking:

given the previous tokens, what comes next?

Sometimes the answer depends on the last word.

timed -> out

Sometimes it depends on a clue much earlier.

opened the bracket ... expected a close

Sometimes it depends on the interaction between many clues.

user role + endpoint + error code + deployment time

Transformers gave the model a better way to combine those clues across positions.

They also scaled well enough that adding data, parameters, and compute kept improving results for a long time.

That scaling behavior helped make transformers the default architecture for modern large language models.

The failure mode they reduced

The failure mode was not:

RNNs cannot read sequences

They can.

The failure mode was:

the relevant clue is far away
the running state loses or blurs it
the model predicts mostly from recent context

Transformers reduced that failure by giving later positions a learned lookup over earlier positions.

They still fail. Attention can be diffuse. Long prompts can be expensive. Models can use the wrong clue or ignore the right one.

But the architecture made the common long-context path less fragile.

The mental model

Think of an RNN as a single courier carrying a summary through a long hallway.

The courier can remember useful things, but every new door may change the note.

Think of a transformer as a room with a large table.

Each token places its current note on the table.

Each position can look across the allowed notes, choose useful ones, and rewrite its own note.

The table is more expensive, but it gives better routes for information and better work for GPUs.

That is why transformers replaced RNNs for language modeling.

Quick check

Which statement is most accurate?

A. Transformers replaced RNNs because they improved context routes and training parallelism, while accepting higher attention cost
B. Transformers generate every future token at the same time
C. RNNs were only hand-written grammar systems
D. Transformers removed the need for token representations

The best answer is A.

The replacement was about architecture shape, scaling behavior, and hardware fit.

Next step

So far we have talked about tokens as if they already have vectors.

But how does a model turn a token into a vector in the first place?

That is the next article: What Are Embeddings?.