Self-attention is attention applied inside one sequence. Each token builds a better representation by comparing itself with the other tokens in the same input.

Self-attention lets tokens in the same sentence look at each other, score what matters, and mix useful context into each token representation.

Attention changed the sequence-model bottleneck. Instead of forcing every old clue through one running state, it let the model look back and score which earlier tokens mattered now.

Attention helped sequence models use distant context by letting the current step look back at relevant earlier tokens.

RNNs and LSTMs gave language models memory, but that memory still had to pass through one step after another. Long context made useful clues fade, compete, and arrive too slowly.

Sequence models struggled with long context because old information had to survive many updates inside a small running state.

Plain RNNs rewrote one hidden state at every step. LSTMs added gates and a cell state, giving the model a cleaner path for preserving useful information over longer spans.

An LSTM is still recurrent, but it controls memory more deliberately. Gates decide what to forget, what to write, and what to expose.

RNNs improved on local n-gram counting by carrying a hidden state through the text. The tradeoff was dependency: token 10 cannot be processed until token 9 has updated the state.

A recurrent neural network reads text step by step, updating a hidden state after every token. That gives it memory, but also makes the sequence hard to parallelize.

N-gram models were useful because they turned language into counts. They failed because a fixed local window cannot remember distant context, understand meaning, or generalize well to unseen phrases.

An n-gram model predicts from nearby words. That makes it simple, but the same simplicity creates sparsity, short memory, and brittle predictions.

Transformers were not the first attempt to model language. They arrived after rules, n-grams, recurrent networks, LSTMs, seq2seq models, and attention exposed the same bottleneck: sequence processing was too slow and too forgetful.

Before transformers, language systems moved from hand-written rules to statistical counts, then to neural models that read text one step at a time.

Before transformers, KV cache, and GPU serving, language models had a smaller job: look at context and predict what text should come next.

The core language-modeling problem is prediction. Given the text so far, estimate the next likely token, then repeat.

Speculative decoding speeds up LLM inference by letting a cheap draft model guess several tokens, then asking the expensive target model to verify those guesses in one parallel pass.

The trick is not making the large model smaller. It is making each expensive target-model pass confirm more than one token when the draft model guessed correctly.

If language models can be compiled directly into silicon, inference stops looking like software running on a GPU and starts looking like the computer itself.

The next serving bottleneck may not be bigger GPUs. It may be whether the model is still software at all.

Bigger GPUs help, but fast LLM serving is mostly a scheduling and KV cache problem. PagedAttention, block allocation, batching, and prefix caching decide how much useful work fits in memory.

The serving loop is constantly asking which requests can run, how many tokens fit, and whether there is enough KV cache memory.

VRAM planning for LLMs is not just about model weights. Long context windows can make the KV cache as expensive as the model itself.

A small model can still need a large GPU once you account for KV cache at long context length.