Speculative Decoding Simulator

Interactive LLM inference latency calculator & token verification tree simulator. Analyze speedup ratios, expected accepted tokens, and draft model tradeoffs offline.

[ PARAMETERS ] Model Controls

Draft Acceptance Rate (α): 75%
Probability that target model accepts a draft token.
Lookahead Window (K): 5 tokens
Number of tokens guessed by draft model per step.
Draft Latency (t_draft): 5 ms
Time taken by draft model to generate 1 token.
Target Latency (t_target): 45 ms
Time taken by target model to verify 1 batch of tokens.

[ METRICS ] Real-Time Performance

Speedup Ratio (S)
2.34x
Expected Tokens / Step
3.16
Effective Latency / Token
19.2 ms
Time Saved
57.3%

[ TOKEN VERIFICATION STREAM ]

[ ARCHITECTURE ] How Speculative Decoding Works

Speculative Decoding (Leviathan et al., Chen et al.) accelerates LLM inference by pairing a small, fast draft model with a large target model. The draft model speculatively generates $K$ tokens. The target model then verifies all $K$ tokens in a single parallel forward pass.

Mathematical Foundation

  • Expected Tokens per Step: \( E[T] = \frac{1 - \alpha^{K+1}}{1 - \alpha} \). Higher acceptance rate \(\alpha\) yields more accepted tokens per verification pass.
  • Step Execution Time: \( T_{step} = K \cdot t_{draft} + t_{target} \). Includes draft generation overhead plus 1 target verification pass.
  • Speedup Formula: \( S = \frac{E[T] \cdot t_{target}}{K \cdot t_{draft} + t_{target}} \). Maximum speedup occurs when \(t_{draft} \ll t_{target}\) and acceptance rate \(\alpha > 70\%\).