Speculative Decoding Simulator
Interactive LLM inference latency calculator & token verification tree simulator. Analyze speedup ratios, expected accepted tokens, and draft model tradeoffs offline.
[ 1. DECODING & MODEL PARAMETERS ]
Probability that target model accepts a draft token.
Number of candidate tokens speculated per iteration.
Time taken by draft model to generate 1 token.
Time taken by target model to verify 1 batch in parallel.
[ 2. REAL-TIME PERFORMANCE METRICS ]
[ ARCHITECTURE ] How Speculative Decoding Works
Speculative Decoding (Leviathan et al., Chen et al.) accelerates LLM inference by pairing a small, fast draft model with a large target model. The draft model speculatively generates $K$ tokens. The target model then verifies all $K$ tokens in a single parallel forward pass.
Mathematical Foundation
- Expected Tokens per Step: \( E[T] = \frac{1 - \alpha^{K+1}}{1 - \alpha} \). Higher acceptance rate \(\alpha\) yields more accepted tokens per verification pass.
- Step Execution Time: \( T_{step} = K \cdot t_{draft} + t_{target} \). Includes draft generation overhead plus 1 target verification pass.
- Speedup Formula: \( S = \frac{E[T] \cdot t_{target}}{K \cdot t_{draft} + t_{target}} \). Maximum speedup occurs when \(t_{draft} \ll t_{target}\) and acceptance rate \(\alpha > 70\%\).
Speculative Decoding, Medusa & S-LoRA Multiplexing Architecture
Accelerate multi-token prediction using Medusa architecture and learn how to multiplex hundreds of LoRA adapters on a single GPU.