The problem with plain recurrent memory

A plain RNN carries one hidden state through the text.

That state is useful. It lets earlier tokens influence later tokens.

But the update is blunt:

old hidden state + current token -> new hidden state

Every step rewrites the same running summary.

If the model reads:

The database replica that lagged during the deploy later caused the alert to

the useful clue might be far behind:

replica lagged

A plain RNN has to keep that clue alive while many other words pass through the same state.

The hidden state is memory, but it is memory that gets rewritten at every token.

That is why plain RNNs often forgot long-range information.

LSTMs added a slower memory lane

LSTM stands for long short-term memory.

The important idea is not the name. The important idea is the extra memory path.

A plain RNN mostly has one evolving state:

hidden state -> hidden state -> hidden state

An LSTM separates the job into two pieces:

cell state   -> longer-lived memory
hidden state -> exposed working output

The cell state gives useful information a cleaner path across time steps.

Gates decide what changes

The LSTM still reads one token at a time, but it does not blindly rewrite everything.

It uses gates.

Think of a gate as a learned valve. For each piece of information, the model can choose how much to let through.

The three common gates are:

forget gate -> what should be dropped from memory?
input gate  -> what should be added to memory?
output gate -> what should be exposed now?

This is the main improvement over a plain RNN.

Instead of one simple overwrite:

old memory + token -> new memory

the model gets a controlled update:

keep useful parts
drop stale parts
write new parts
expose what matters for this step

LSTMs made memory an explicit decision, not just a side effect of the state update.

A tiny example

Suppose the model reads:

The cache key changed during the deploy, so the hit rate eventually

The phrase cache key changed should still matter when the model reaches eventually.

A plain RNN has to preserve that clue inside the hidden state while every later token modifies the same state.

An LSTM has a better mechanism:

forget gate: keep the cache-change clue
input gate: add deploy timing
output gate: expose enough context to predict drop

The gates are not hand-written rules. They are learned from data.

But the shape matters. The model has a path for saying:

this old information is still useful, do not erase it yet

That is why LSTMs helped with longer dependencies.

Why the cell state helped gradients

The same problem appears during training.

If the model makes a mistake late in a sequence, the learning signal has to travel backward through many time steps.

In a plain RNN, that signal repeatedly passes through the same kind of nonlinear state update. It can shrink until it becomes useless, or grow until training becomes unstable.

That is the vanishing or exploding gradient problem.

LSTMs helped because the cell state can carry information through a more stable path. The gates still learn what to keep and change, but the architecture gives gradients a better route across many steps.

What LSTMs did not fix

LSTMs were better than plain RNNs, but they kept the recurrent shape.

Token 10 still depends on token 9.

token 1 -> token 2 -> token 3 -> ... -> token 10

That means the sequence still has to be processed in order.

GPUs can batch many sequences together, but one sequence still has a dependency chain along time.

So LSTMs improved memory, but they did not solve the scaling problem that later made transformers attractive.

The failure mode

The failure mode changed, but it did not vanish.

Plain RNN failure:

important old clue gets overwritten too easily

LSTM improvement:

gates can preserve useful old clues longer

Remaining LSTM cost:

every token still waits for the previous recurrent step

This is the tradeoff to remember.

LSTMs made recurrent memory less fragile. They did not make sequence processing parallel.

The mental model

Think of a plain RNN as a person rewriting one sticky note after every word.

The note can carry context forward, but every rewrite risks losing something.

Think of an LSTM as the same person with a clipboard that has locks:

forget gate -> erase only some lines
input gate  -> write only useful new lines
output gate -> read out only what is needed now

The clipboard is still updated in order. It is just harder to accidentally wipe out the important line.

That is why LSTMs were better than plain RNNs.

Quick check

What was the main improvement LSTMs made over plain RNNs?

A. They removed the need to process tokens sequentially
B. They added gates and a cell state to control memory updates
C. They replaced words with exact n-gram counts
D. They made every token attend to every other token directly

The best answer is B.

LSTMs improved how recurrent memory is preserved and updated. They did not remove the step-by-step dependency.

Next step

LSTMs made memory more deliberate, but long sequences were still hard.

Next: Why Sequence Models Struggled With Long Context.

That article looks at why recurrent models still had trouble as context grew, even after LSTMs made memory less fragile.