How Speculative Decoding Makes LLMs Feel Faster
Speculative decoding speeds up LLM inference by letting a cheap draft model guess several tokens, then asking the expensive target model to verify those guesses in one parallel pass.
The slow part is the big model loop
Large language models do not generate a paragraph in one shot.
They generate one token, append it to the context, then run another forward pass to generate the next token.
big model -> token 1
big model -> token 2
big model -> token 3
big model -> token 4
If the target model is huge, that loop is expensive. A trillion-parameter model needs a lot of memory bandwidth just to load weights for each step, and it needs enough VRAM to keep weights and KV cache available.
Normal decoding pays for one large-model pass per generated token.
That is the bottleneck speculative decoding attacks.
The intuition
Speculative decoding adds a small, cheap draft model in front of the expensive target model.
The draft model might be a smaller model, a quantized model, a model with extra draft heads, or another lightweight predictor. Its job is not to be perfect. Its job is to guess a few likely next tokens quickly.
Suppose the draft model proposes:
A, B, C, D
Now instead of asking the big model to generate A, then B, then C, then D one by one, the server asks the big model to score the whole candidate sequence in one verification pass.
The reason this works is a transformer detail: during a prefill-style pass, the model produces next-token distributions for multiple positions in parallel.
It can score:
after prompt -> is A likely?
after prompt + A -> is B likely?
after prompt + A B -> is C likely?
after prompt + A B C -> is D likely?
That verification pass is still expensive, but it can confirm multiple tokens at once.
The accept/reject step
The target model compares its probability for each draft token against the draft model's probability.
Using your example:
draft p(A) = 0.80
draft p(B) = 0.70
draft p(C) = 0.80
draft p(D) = 0.90
If the target model is at least as happy with a token as the draft model was, that token can be accepted.
For example:
A -> accepted
B -> accepted
C -> rejected
D -> discarded
The important correction is what happens at C.
The system does not keep C as-is after rejecting it. It samples a replacement token for that position from a corrected distribution based on the target model and draft model probabilities. Everything after the rejected token is thrown away because it depended on the rejected branch.
Accepted draft tokens become real output. The first rejected position is repaired. Later draft tokens are discarded.
Then the process repeats from the repaired prefix.
Why this preserves quality
The clever part of speculative decoding is that it can be exact.
With the right rejection-sampling rule, the final output distribution is the same as sampling directly from the target model token by token. The draft model only proposes candidates. The target model still decides what is valid.
That is why this technique is attractive in production inference: it can reduce latency without changing the target model's behavior.
Where the speedup comes from
Imagine the draft model proposes four tokens and the target accepts three of them.
Without speculation:
4 generated tokens = 4 target-model decode passes
With speculation:
draft 4 cheap tokens
verify them with 1 target-model pass
keep the accepted prefix
If the draft model is fast and the acceptance rate is high, the target model does fewer serial decode steps.
That is the win.
Speculative decoding turns some token-by-token latency into parallel verification work.
Why it does not always help
Speculative decoding is not free.
You now run a draft model too. You also run target-model verification over draft tokens that may be rejected. If the draft model is slow, or if it guesses badly, the overhead can eat the benefit.
The speedup depends on:
- how fast the draft model is
- how many draft tokens it proposes
- how many are accepted
- batch size and serving load
- target model architecture
- whether the workload is predictable enough
For dense models and predictable continuations, speculative decoding can be very effective. For harder generations, low acceptance rates reduce the gain.
The production takeaway
Speculative decoding is best understood as a serving trick, not a new model capability.
The target model is still the source of truth. The draft model is just a fast guesser.
The server is betting:
These next few tokens are probably obvious.
Let me guess them cheaply,
then ask the big model to verify the whole run.
When the bet is right, users see tokens arrive faster.
When the bet is wrong, the system falls back to the target model's corrected distribution and continues.
The magic is not that the small model is trusted. The magic is that the big model can verify several guesses in parallel.
Sources
Field notes
Sign in to leave a comment.
Sign in