Layers should improve a representation

A transformer block does several things to a token representation.

It runs attention.

It runs a feed-forward network.

It runs normalization around those steps.

If every transformation completely replaced the old vector, deep stacks would be fragile.

One bad update could wipe out useful information.

Residual connections make the update less brittle.

Instead of saying:

new vector = transformation(old vector)

the block can say:

new vector = old vector + transformation(old vector)

A residual connection keeps the original representation on a direct route around the transformation.

A tiny example

Suppose a token vector already carries a useful clue:

token: key
context: this is a cache key

The attention step wants to add one more clue:

changed during deploy

Without a residual path, the attention output has to produce everything the next step needs.

attention output must carry:
- cache-key meaning
- deploy-change clue
- anything else still needed

With a residual path, the block can preserve the old vector and add the attention update.

old token vector + attention update -> next token vector

That is the core trick.

The transformation does not have to be perfect. It can act more like an edit.

Residual means add the input back

The word "residual" can sound more complicated than it is.

In this setting, it means the layer keeps a shortcut around a sublayer.

For attention:

x -> self-attention -> attention_output
x + attention_output -> next representation

For the feed-forward network:

y -> FFN -> ffn_output
y + ffn_output -> next representation

The exact ordering with layer normalization can vary across transformer designs, but the residual idea is the same.

The input gets a route around the transformation.

Why that helps information flow

Transformers are deep.

A token representation may pass through many blocks:

block 1 -> block 2 -> block 3 -> ... -> block N

If each block fully replaces the representation, useful details can be distorted again and again.

Residual connections give information a simpler path through the stack.

keep x
add useful update
keep result
add another useful update

That does not mean the model ignores the transformations.

It means the model can choose to make small or large changes while preserving a stable route for what already matters.

Why that helps gradients

Training also needs gradients to move backward through the network.

The model learns by asking:

How should earlier weights change to reduce the error?

In a very deep network, that signal can become weak or unstable as it passes through many transformations.

Residual connections provide a more direct backward route.

later loss
  -> block N
  -> residual path
  -> block N-1
  -> residual path
  -> earlier blocks

This is one reason residual connections made it easier to train very deep neural networks, including transformers.

Residual connections help both forward information and backward gradients move through many layers.

What residual connections do not do

Residual connections are not memory by themselves.

They do not decide which tokens attend to which other tokens.

They do not replace the feed-forward network.

They do not make a model understand text on their own.

They are a wiring pattern.

input -> transformation
input -> shortcut
sum both paths

That wiring pattern makes the transformations easier to stack.

Failure mode and tradeoff

Without residual connections, a deep transformer would be harder to optimize.

The model would have to preserve every useful feature through every attention and FFN output.

That creates a failure mode:

early layer builds useful feature
later layer overwrites it
even later layer cannot recover it

Residual connections reduce that risk by carrying the old representation forward.

There is still a tradeoff.

If updates become too large, adding them back can make activations unstable. That is one reason residual connections usually appear together with layer normalization and carefully chosen initialization.

The shortcut helps, but it is not the only stabilizing piece.

Mental model

Think of each transformer block as editing a document.

Without a residual connection, every editor rewrites the whole document from memory.

With a residual connection, each editor starts from the current document and adds a tracked change.

The original content remains available unless the edit truly changes it.

That is the feel of a residual path:

preserve what exists
add a learned update
pass the result onward

Quick check

Why do transformers use residual connections?

A. To preserve a route for information and gradients around transformations
B. To let a token see future tokens during decoding
C. To store exact strings from the prompt
D. To remove the need for attention

The best answer is A.

A residual connection keeps a shortcut path while attention and FFN layers add learned updates.

Next step

Residual connections make deep stacks easier to train and easier to pass information through.

But adding updates over and over can still make the scale of vectors drift.

Next: Why Transformers Use Layer Normalization.