Tokens need numbers

The earlier articles described how transformers compare tokens with self-attention.

But a transformer cannot compare raw text directly.

It does not begin with:

"cache" as a string
"key" as a string

It begins with numbers.

Before attention can run, each token has to become a vector:

cache -> [0.12, -0.44, 0.91, ...]
key   -> [0.18, -0.39, 0.77, ...]

That vector is called an embedding.

An embedding is a learned vector representation for a token.

Not a dictionary definition

An embedding is easy to misunderstand.

It is not a small paragraph that explains the word.

It is not:

cache = "a place to store data temporarily"

It is a list of learned numbers:

cache = [0.12, -0.44, 0.91, 0.03, ...]

Those numbers do not have simple labels like:

dimension 1 = storage
dimension 2 = speed
dimension 3 = computers

Some directions in the vector space may become meaningful, but the model is not hand-writing definitions into the vector.

The values are adjusted during training because they help the model make better predictions.

A tiny lookup table

At the start, you can picture embeddings as a lookup table.

Suppose the tokenizer has these token ids:

17 -> cache
42 -> key
93 -> expired

The embedding table stores one vector per token id:

17 -> [0.12, -0.44, 0.91, ...]
42 -> [0.18, -0.39, 0.77, ...]
93 -> [-0.31, 0.08, 0.55, ...]

If the input is:

cache key expired

the tokenizer first gives token ids:

17 42 93

Then the embedding table gives vectors:

E[17] E[42] E[93]

Now the transformer has numeric inputs it can process.

Similar use can lead to nearby vectors

During training, tokens that behave similarly often end up with related vectors.

For example:

database
cache
queue
broker

These are different tokens. But in engineering text, they may appear in related contexts:

database connection
cache connection
queue connection
broker connection

The model can learn that these tokens participate in similar patterns.

That does not mean they become identical.

It means the vector space gives the model a softer starting point than exact string matching.

token id alone: 42 is just 42
embedding: 42 has a learned numeric shape

Why this helped language models

N-gram models mostly counted exact local patterns.

If they saw:

cache miss

many times, they could count it.

But they did not naturally know that:

cache miss
database miss
lookup miss

might share some structure.

Embeddings gave neural models a way to learn softer relationships.

They let the model start from:

tokens as points in a learned space

instead of:

tokens as unrelated labels

That is one reason neural language models were able to generalize beyond exact phrases.

The vector is only the starting point

There is another important detail.

The embedding for a token is not the final meaning of that token in a sentence.

Take:

the cache key changed

The token key starts with one embedding.

But after self-attention, that representation can absorb context from cache and changed.

So the path looks like this:

token id -> embedding -> context-aware representation

The original embedding is context-free.

The later representation is context-aware.

That distinction matters.

key in "cache key"
key in "keyboard key"
key in "key insight"

The token may start from the same learned vector, but the transformer layers can update it differently depending on the surrounding tokens.

A failure mode

Embeddings are powerful, but they are not magic.

A token can be ambiguous.

port

It could mean:

network port
shipping port
left side of a ship

The embedding table gives port a starting vector, but that vector alone cannot know which meaning is intended.

The model needs context.

If the surrounding text is:

the service listens on port 443

self-attention can help push the representation toward the networking sense.

If the surrounding text is:

the cargo arrived at the port

the context should push it somewhere else.

Embeddings start the computation; context layers refine it.

A tradeoff

Embedding tables can be large.

If a model has:

50,000 tokens
4,096 numbers per embedding

then the embedding table contains:

50,000 x 4,096 numbers

That is a lot of learned parameters.

Larger vectors can give the model more room to represent useful patterns, but they cost memory and compute.

Smaller vectors are cheaper, but they may squeeze too much information into too little space.

This is the same kind of engineering tradeoff that appears throughout inference systems:

more capacity -> more cost
less capacity -> more pressure on representation

The mental model

Think of a token id as a row number.

The embedding table is a shelf of learned vectors.

When the model sees a token id, it pulls the vector from that row.

token id -> row lookup -> vector

That vector is not the whole meaning of the word.

It is the model's learned starting handle for that token.

Then attention, feed-forward layers, residual connections, and normalization keep transforming it.

Quick check

What is an embedding?

A. A learned vector used to represent a token numerically
B. A hand-written dictionary definition
C. A rule that tells the model which grammar pattern to use
D. A count of how often two words appeared together

The best answer is A.

Embeddings turn token ids into learned numeric vectors so the transformer has something to compute with.

Next step

Embeddings give each token a vector.

But the transformer also needs to know where each token appeared.

Next: Why Transformers Need Positional Information.

That article explains why attention alone sees token relationships but does not automatically know token order.