Attention needs order help

The previous article introduced embeddings.

An embedding tells the transformer which token it is starting with:

cache -> vector
key   -> vector

But there is a missing piece.

The model also needs to know where each token appeared.

position 1: cache
position 2: key

Without that, attention can compare tokens, but it does not naturally know their order.

Attention alone knows token relationships, not token positions.

A tiny example

Compare these two sentences:

dog bites man
man bites dog

They contain the same tokens:

dog
bites
man

But they do not mean the same thing.

Order changes the meaning.

The same problem appears in engineering text:

cache invalidates token
token invalidates cache

Those sentences use the same words, but the direction of the action changes.

If the model only knows which token vectors are present, it is missing a crucial signal.

Why self-attention does not solve this by itself

Self-attention builds scores between token vectors.

Roughly:

each token asks: which other tokens matter to me?

That is powerful.

But if the input vectors do not include position, the attention operation treats the sequence more like a set of items than an ordered line.

For:

cache key expired

the model sees token vectors for:

cache
key
expired

If we swap the first two tokens:

key cache expired

the same token vectors are still present.

Attention can still compare cache with key, but it needs another signal to know which one came first.

Adding position to the input

A transformer gives each token representation two kinds of information:

what token is this?
where is it?

A simple picture is:

token representation = token embedding + position information

For a three-token input:

cache key expired

the model can build:

cache   + position 1
key     + position 2
expired + position 3

Now cache at position 1 is not represented exactly the same way as cache at position 2.

That lets later layers learn order-sensitive patterns.

Learned or fixed positions

There are different ways to provide positional information.

Some models learn position vectors during training:

position 1 -> learned vector
position 2 -> learned vector
position 3 -> learned vector

Other designs use fixed mathematical patterns.

Some modern systems use position handling inside attention itself, such as rotary position embeddings.

The implementation can vary.

The reason is the same:

tokens need order information

The transformer needs a way to distinguish:

token A before token B

from:

token B before token A

A failure mode

Without positional information, the model can blur together inputs that should be different.

Imagine a tiny command-like sentence:

delete old backup

Now swap two tokens:

backup old delete

The second line is awkward, but it has the same token set.

A real model sees more than three words, and training data is messier than this example. Still, the failure mode is the same.

If order is not represented, the model has a weak basis for learning syntax, direction, and sequence structure.

Language is not just a bag of tokens. It is tokens in order.

A tradeoff

Position handling also creates design choices.

One issue is length.

If a model learns position vectors only up to a certain context length, extending beyond that length can be awkward.

For example:

trained positions: 1 through 4,096
requested context: 8,192 tokens

The model now needs some way to handle positions it did not originally learn in the same way.

Different positional methods behave differently when context length grows.

That matters for inference systems because long context is already expensive. Position handling is one of the details that affects how well a model behaves as prompts get longer.

Why this belongs before Q, K, and V

The next attention articles will talk about query, key, and value vectors.

Those vectors are computed from token representations.

So the starting representation matters.

If the representation includes only:

the token

then attention has no direct order signal.

If the representation includes:

the token + its position

then query and key computations can learn order-sensitive behavior.

For example, the model can learn that a nearby previous token is often more useful than a far future token, or that a word immediately after not may need special handling.

Position information gives attention the raw material for those patterns.

The mental model

Think of token embeddings as name tags.

They say:

I am cache
I am key
I am expired

Position information adds seat numbers.

cache   sits in seat 1
key     sits in seat 2
expired sits in seat 3

Self-attention can then ask not only:

which token matters?

but also:

where is that token relative to me?

That is the missing order signal.

Quick check

Why do transformers need positional information?

A. Because attention alone does not know token order
B. Because embeddings are dictionary definitions
C. Because the model cannot compare token vectors
D. Because every sentence has the same length

The best answer is A.

Self-attention can compare tokens, but it needs positional information to reason about sequence order.

Next step

Now we have token vectors and position signals.

Next: What Are Query, Key, and Value Vectors?

That article explains the three learned views a transformer uses to decide what each token should look for, what each token offers, and what information should be mixed in.