Consider how human beings read a sentence like: 'The quick brown fox jumps over the lazy...' Your brain does not painstakingly decode every letter of the word 'dog' before understanding it; your subconscious mind anticipates the word before your eyes even reach it. Speculative decoding brings this exact cognitive mechanism to large language model inference.
The Single-Token Autoregressive Tax
Standard language models are strictly autoregressive: to generate token $N+1$, the model must inspect tokens $1$ through $N$. When generating a 500-word essay, an inference server must execute the full forward pass of a massive 70-billion-parameter neural network 500 consecutive times in lockstep.
Because each forward pass processes only a single token vector, the GPU's massive parallel arithmetic units are starved for work. Modern GPUs are designed to process thousands of mathematical calculations in parallel; forcing an H100 GPU to multiply a single vector against its weights is like using an 18-wheeler truck to deliver a single envelope across town.
[Standard Sequential Decoding: 5 Slow Forward Passes]
Step 1: Load 70B Weights ──► Emit 'The' (50ms)
Step 2: Load 70B Weights ──► Emit 'quick' (50ms)
Step 3: Load 70B Weights ──► Emit 'brown' (50ms)
Step 4: Load 70B Weights ──► Emit 'fox' (50ms)
Step 5: Load 70B Weights ──► Emit 'jumps' (50ms)
Total Latency: 250ms (GPU compute cores idling 90% of the time!)
[Speculative Decoding: Draft Fast, Verify in Parallel]
Step 1: Small 1B Draft Model guesses: ['The', 'quick', 'brown', 'fox', 'jumps'] (15ms)
Step 2: Large 70B Target Model checks all 5 tokens in ONE single forward pass (55ms)
Verification: ['The' ✓, 'quick' ✓, 'brown' ✓, 'fox' ✓, 'jumps' ✓]
Total Latency: 70ms (3.5x Faster with Zero Loss in Accuracy!)
The Mechanics of Speculative Decoding
Speculative decoding pairs a large, highly capable target model (e.g. 70B) with a tiny, ultra-fast draft model (e.g. 1B) or speculative draft heads:
- High-Speed Drafting: The lightweight 1B model rapidly autoregresses $K$ candidate tokens (for instance, predicting the next 5 words). Because the draft model is tiny, generating 5 tokens takes only 15 milliseconds.
- Batched Parallel Verification: The large 70B target model receives all 5 candidate tokens simultaneously. Because modern GPUs process small batches of tokens in parallel at virtually zero marginal memory bandwidth cost, verifying 5 tokens takes almost the exact same wall-clock time as generating a single token (55 milliseconds).
- Statistical Rejection Sampling: The target model evaluates the probability distribution for each candidate token. If the draft model's predictions align with the target model's probability thresholds, all 5 tokens are accepted simultaneously in a single step. The moment a speculative token diverges, generation falls back to the target model's true prediction, and the loop restarts.
Mathematical Rigor
The remarkable beauty of speculative decoding is that it is mathematically exact: the output probability distribution of the combined draft-and-verify system is provably identical to sampling directly from the large target model alone. There is zero degradation in reasoning, zero loss in creativity, and a 2x to 3x increase in user-facing streaming speed.