The counting trick

An n-gram model is one of the simplest ways to predict text.

It looks at the last n - 1 words and asks:

What usually comes next after this exact pattern?

A bigram model uses one previous word:

P(next | previous)

A trigram model uses two previous words:

P(next | previous two words)

If the training data contains this phrase many times:

connection timed out

then after:

connection timed

the trigram model can assign high probability to:

out

The model is not understanding networking. It is counting local patterns.

That was powerful enough to be useful. It was also exactly why n-gram models hit a wall.

The model only sees a small window

The first limit is memory.

A trigram model sees only the previous two words. A 5-gram model sees only the previous four. Anything outside the window is invisible.

That fails quickly:

The cache key that changed during the deploy caused the hit rate to

The word you want may be:

drop

But the clue was much earlier:

cache key changed

If the model only sees:

hit rate to

it has lost the important cause.

Language often depends on information that is far away:

  • a subject introduced at the start of a sentence
  • a variable defined earlier in code
  • a service name mentioned in a previous log line
  • a negation that changes the meaning of a later phrase
  • a topic established several paragraphs ago

A fixed window cannot choose what matters. It just clips the context.

Bigger n creates a new problem

The obvious fix is to make n larger.

If a trigram is too small, use a 10-gram. If a 10-gram is too small, use a 50-gram.

That sounds reasonable until the counts disappear.

Suppose the model needs to estimate:

P(timeout | the grpc request from checkout service ended with)

That exact phrase may never have appeared in the training data. Even if every individual word is common, the exact long sequence is probably rare.

This is the sparsity problem.

short window -> enough counts, weak context
long window  -> better context, missing counts

N-gram models trade context length against statistical coverage.

The more words you condition on, the more specific the pattern becomes. Specific patterns are useful when they exist, but many useful phrases will not have exact matches.

It cannot generalize meaning well

N-gram models are literal.

These sentences mean similar things:

the request timed out
the call timed out
the query timed out
the operation timed out

An n-gram model mostly sees different surface forms. It can count them separately, but it does not naturally know that request, call, query, and operation are related in this context.

That hurts generalization.

If the training data has:

the request timed out

but the input says:

the operation timed

the model needs enough examples of that exact phrasing. It does not have a deep representation that says:

operation behaves like request here

Neural models improved this by learning dense representations. Similar words and contexts could share statistical strength instead of living as isolated counts.

It struggles with structure

Language is not just nearby word order.

Code makes the problem obvious:

if cache_hit:
    return cached_value
else:

A useful continuation depends on structure. The model should understand that else belongs to the if, and that the next line probably handles a cache miss.

An n-gram model can memorize common snippets, but it does not build a flexible representation of program structure.

The same issue appears in prose:

The model was small, but the context window was enormous, so the GPU

The likely continuation has to respect the contrast:

still ran out of memory

The important signal is not just the last few words. It is the relationship between small model, enormous context window, and GPU.

Smoothing helps, but does not change the shape

N-gram systems used smoothing to avoid assigning zero probability to unseen phrases.

The idea is simple: if an exact long pattern is missing, back off to shorter patterns.

try 5-gram
if missing, try 4-gram
if missing, try trigram
if missing, try bigram

That is useful engineering. It makes the model less brittle.

But it does not remove the core limit. Backing off means throwing away context. The model survives by becoming less specific.

The wall

N-gram models were a good answer to an early question:

Can we predict language from observed local statistics?

The answer was yes.

But the next question was harder:

Can we represent meaning, structure, and long-range context?

For n-grams, the answer was mostly no.

The wall had four parts:

  • the context window was fixed
  • long windows caused sparse counts
  • similar meanings did not share enough information
  • structure beyond local word order was hard to model

That is why the field moved toward neural language models and recurrent networks. They were attempts to replace brittle local counting with learned representations and memory.

The mental model

Think of an n-gram model as autocomplete with a short clipboard.

It is fast, simple, and often surprisingly decent when the phrase is familiar.

But it cannot decide that an earlier clue should stay important. It cannot build a flexible concept of similarity. It cannot reason over a long context. It can only ask what usually followed the exact local pattern it can see.

That local clipboard was the wall.

Quick check

Why does increasing n not fully solve the problem?

A. Larger n-grams make the model forget all punctuation
B. Larger n-grams create more exact patterns that may never appear in training data
C. Larger n-grams prevent the model from counting words
D. Larger n-grams only work for SQL

The best answer is B.

A larger window gives more context, but it also makes the pattern more specific. Specific patterns are more likely to be missing or rare.

Next step

The next move was to stop treating words as isolated count keys.

Neural language models learned representations, so related words and contexts could share information.

Next: Why RNNs Process Text Sequentially.