The move after n-grams

N-gram models had a simple weakness: they only remembered a fixed local window.

If the important clue was outside that window, the model could not use it.

Recurrent neural networks, or RNNs, tried a different shape:

read token -> update memory -> read next token -> update memory

Instead of counting only the last few words, an RNN carries a hidden state through the sequence.

The hidden state is the RNN's running memory.

That was the core improvement. The model could, in principle, let earlier information influence later predictions.

It also created the core bottleneck.

The state update loop

An RNN processes text one token at a time.

For a sentence like:

the cache key changed

the computation looks like:

token: the      -> state 1
token: cache    -> state 2
token: key      -> state 3
token: changed  -> state 4

Each new state depends on two things:

previous state + current token -> next state

That dependency is the whole point. The model can use what it has already read while processing the current token.

But it means state 4 cannot exist until state 3 exists. State 3 cannot exist until state 2 exists. State 2 cannot exist until state 1 exists.

Why this helped compared to n-grams

An n-gram model has a hard window.

For a trigram, the model sees:

previous two words -> next word

An RNN does not have that exact cutoff. It keeps passing a state forward.

If the text says:

The deployment changed the cache key, and after several minutes the hit rate started to

an RNN has at least a path for cache key to influence the later prediction:

drop

That is better than a fixed trigram window. The model is no longer limited to exact local counts.

It can also learn softer representations. request, call, and operation can affect the hidden state in related ways instead of living only as separate count keys.

The hidden state is a bottleneck

The problem is that the hidden state has to carry everything useful.

Imagine reading this:

The user who opened the incident before the deploy and noticed the cache miss pattern later confirmed that the

By the time the model reaches the final the, useful clues may be far behind:

  • user
  • incident
  • deploy
  • cache miss pattern
  • confirmed

The RNN has to compress those clues into one evolving state. Some details survive. Some fade. Some get overwritten.

RNN memory is not a database. It is a lossy running summary.

This is why plain RNNs struggled with long-range dependencies. The signal from early tokens had to pass through many update steps before it reached later tokens.

Training made the problem worse

The sequential chain also affects training.

To learn from a mistake at the end of a long sequence, the training signal has to travel backward through many time steps.

loss at token 100
  -> update state 99
  -> update state 98
  -> update state 97
  -> ...

This can become unstable. Gradients can shrink too much or grow too much as they move through the chain.

That is the classic vanishing and exploding gradient problem.

In practical terms: the model may fail to learn that something early in the text matters much later.

The hardware problem

Modern hardware likes parallel work.

GPUs are happiest when they can run many similar operations at the same time.

RNNs fight that shape along the sequence dimension:

state 1 must finish before state 2
state 2 must finish before state 3
state 3 must finish before state 4

You can parallelize across examples in a batch, but within one sequence the dependency chain remains.

That matters when sequences get long and datasets get huge. The model spends a lot of time stepping through tokens in order.

Why LSTMs came next

Plain RNNs were useful, but their memory was fragile.

LSTMs changed the state update by adding gates:

forget gate -> what should be dropped?
input gate  -> what should be added?
output gate -> what should be exposed?

The goal was to make memory more deliberate. Instead of blindly rewriting the hidden state at every step, the model could learn what to preserve.

That helped with longer dependencies.

But it did not remove the sequential dependency:

token 10 still waits for token 9

LSTMs improved memory. They did not make sequence processing fully parallel.

The wall

RNNs were a real step forward from n-grams.

They replaced local counting with learned state. They could represent softer patterns. They gave earlier tokens a path to influence later tokens.

But the same design had three costs:

  • the hidden state compressed too much information
  • long-range signals were hard to preserve and train
  • sequence steps had to happen in order

That last point is the one transformers attacked most aggressively.

Instead of carrying one state through the text, transformers let tokens compare themselves with other tokens through attention.

The mental model

Think of an RNN as a reader with a notepad that only has one line.

After every token, it rewrites the line:

old note + new token -> new note

That note can carry context forward, which is better than an n-gram window. But the note is limited, lossy, and must be written in order.

The model gets memory, but the price is sequential processing.

Quick check

Why can an RNN not process all tokens in a sentence at the same time?

A. It needs the previous hidden state before computing the next hidden state
B. It cannot represent words as vectors
C. It only works with trigrams
D. It requires a database index for every token

The best answer is A.

Each state depends on the state before it, so the sequence forms a chain.

Next step

RNNs made memory learnable, but the memory was still fragile.

Next: Why LSTMs Were Better Than Plain RNNs.

That article looks at gates, cell state, and why LSTMs helped preserve information without solving the full sequence bottleneck.