Attention is not the whole block

The previous articles focused on attention.

Attention lets tokens look at other tokens.

For a short input:

the cache key changed

attention lets changed use clues from:

cache
key

That is the mixing step.

But a transformer block usually does more than mix tokens. After attention, it runs a small neural network on each token position.

That network is the feed-forward network, often shortened to FFN or MLP.

The feed-forward network updates each token separately after attention has mixed context across tokens.

A tiny picture

Think about one transformer block like this:

token vectors
  -> self-attention mixes information between tokens
  -> feed-forward network updates each token by itself

For four tokens, attention can move information sideways:

the     <-> cache <-> key <-> changed

The FFN does not move information sideways.

It applies the same kind of transformation at every position:

the'     -> FFN -> the''
cache'   -> FFN -> cache''
key'     -> FFN -> key''
changed' -> FFN -> changed''

The apostrophes just mean "updated version."

Attention says:

Which other tokens should I use?

The FFN says:

Now that I have context, how should I reshape this token representation?

What the FFN is made of

At a high level, the FFN is usually two learned linear transformations with a nonlinearity between them.

Roughly:

token vector
  -> expand to a wider hidden vector
  -> apply nonlinearity
  -> project back to the model width

If the model width is small in a toy example:

model width: 4 numbers
ffn hidden:  16 numbers

then the FFN might do:

4 numbers -> 16 numbers -> 4 numbers

Real models use much larger widths, but the shape is the same.

Why expand and shrink?

The expansion gives the model more room to compute features.

Imagine a token representation after attention carries several hints:

this token is "key"
nearby token says "cache"
later token says "changed"

The FFN can turn those hints into more useful internal features:

cache-key concept
configuration-change concept
possible miss-rate effect

Then it compresses the result back into the normal model width so the next layer can keep going.

The FFN is not storing English labels like those. The labels are just a human sketch. In the model, these are learned numeric directions.

The same FFN is reused at every token

One easy mistake is to imagine a separate FFN for each word.

That is not the usual transformer pattern.

The same learned FFN is applied to every token position.

position 1 -> same FFN weights
position 2 -> same FFN weights
position 3 -> same FFN weights
position 4 -> same FFN weights

The inputs differ because attention gave each token different context.

So the output can still differ a lot.

For example:

key after seeing "cache"   -> cache-key meaning
key after seeing "keyboard" -> input-device meaning

The FFN sees a context-aware vector, not just the raw word.

The important boundary

The FFN operates per token.

That means token key does not directly read token cache inside the FFN step.

The reading already happened in attention.

self-attention: token-to-token communication
feed-forward:  per-token transformation

This boundary is useful.

Attention handles routing information between positions.

The FFN handles computation at each position after that routing.

If attention is the communication step, the FFN is the per-token processing step.

Failure mode and tradeoff

There is a tradeoff.

FFNs can be expensive because they often expand the vector to a much larger hidden size.

For each token, the model has to multiply through those learned matrices.

With more tokens:

more active tokens -> more FFN work

With a wider FFN:

larger hidden width -> more parameters and compute

A small FFN may not have enough capacity to reshape useful features.

A very large FFN can make the model stronger, but it costs more memory bandwidth, more compute, and more serving time.

In inference systems, that cost matters because every generated token passes through the FFN in every layer.

Mental model

Think of attention as a team meeting.

Every token gathers notes from the other relevant tokens.

Then the FFN is each token going back to its desk and working on its own notes.

No more cross-token conversation happens during that step.

The token just transforms what it already has.

That is why the transformer block needs both pieces:

attention -> gather context
FFN       -> compute on that context

Quick check

What does the FFN do inside a transformer block?

A. It updates each token position independently after attention
B. It chooses which future tokens the model is allowed to see
C. It stores the KV cache for serving
D. It replaces attention with a recurrent state

The best answer is A.

Attention mixes information between tokens. The FFN transforms each token's representation after that mix.

Next step

Now the block has two strong transformations:

self-attention
feed-forward network

But stacking many transformations creates a new problem. Information and gradients need stable routes through the block.

Next: Why Transformers Use Residual Connections.