sequential-speculative-decoding 0.1.1

Speculative decoding methods for efficient LLM inference: standard SD and hierarchical speculative decoding (HSD)

Sequential Speculative Decoding

A Python package implementing standard and sequential speculative decoding for accelerating LLM inference without changing the output distribution.

Speculative decoding uses a small, fast draft model to propose multiple tokens, then lets the larger target model verify them in a single forward pass. The result is the same text you'd get from the target model alone, but faster.

This package currently implements three speculative decoding methods:

• Standard SD — token-level verification (Leviathan et al., 2023)
• Naive HSD — sequence-level hierarchical verification (Zhou et al., 2026)
• Capped HSD — improved HSD with capped prefix ratios (Zhou et al., 2026)

Installation

First, install PyTorch for your hardware (see pytorch.org for the right CUDA). Then run:

pip install sequential-speculative-decoding

or:

git clone https://github.com/Yijin-Zeng/Sequential-Speculative-Decoding
cd Sequential-Speculative-Decoding
pip install -e .

Quick Start

from seqspecdecod import load_models, speculative_generate

tokenizer, draft_model, target_model = load_models(
    draft_name="Qwen/Qwen3-0.6B",
    target_name="Qwen/Qwen3-4B",
    device="cuda",
)

result = speculative_generate(
    draft_model=draft_model,
    target_model=target_model,
    tokenizer=tokenizer,
    prompt="Explain what machine learning is in one paragraph.",
    max_new_tokens=200,
    gamma=5,
    method="standard",
)

print(result["text"])
print(f"Speed: {result['tokens_per_sec']:.1f} tok/s")
print(f"Acceptance rate: {result['acceptance_rate']:.1%}")

Benchmarking

Use benchmark to compare speculative decoding against the target-only baseline across multiple prompts:

from seqspecdecod import load_models, benchmark

tokenizer, draft_model, target_model = load_models()

prompts = [
    "Give me a short introduction to large language models.",
    "Explain the difference between supervised and unsupervised learning.",
    "What is the transformer architecture and why is it so popular?",
    "What is gradient descent and how does it work?",
]

baseline_results, spec_results = benchmark(
    draft_model=draft_model,
    target_model=target_model,
    tokenizer=tokenizer,
    prompts=prompts,
    max_new_tokens=200,
    gamma=5,
    method="standard",
    verbose=True,
)

Benchmark Results

Measured on Qwen3-0.6B (draft) and Qwen3-4B (target), 4 prompts, 200 tokens each, Gamma = 5. Hardware: NVIDIA GeForce RTX 3070 (8 GB VRAM).

Method	Speed (tok/s)	Speedup	Acceptance Rate
Target only (baseline)	1.9	1.0×	—
Standard SD	3.2	1.7×	43%
Capped HSD	3.7	1.9×	52%
Naive HSD	1.8	0.9×	49%

Effect of Gamma (draft length):

Note: Naive HSD is slower than the baseline in practice because its resampling step requires multiple additional target model calls, so in practice, it is not recommended.

Methods

Standard Speculative Decoding

The draft model proposes Gamma tokens autoregressively. The target model evaluates them all in one forward pass. Tokens are accepted left-to-right with probability min(p/q, 1). The first rejection stops the chain and resamples from a residual distribution. Produces an extra bonus token if all proposed tokens are accepted.

Naive Hierarchical SD (HSD)

Verifies the entire drafted sequence at once using joint probability ratios. Scans backward to find the longest acceptable prefix, then resamples the rest using branch divergence probabilities. Each resampling step calls both models again, which is why it is slow in practice.

Capped HSD

A more efficient variant of HSD that caps prefix ratios to avoid the multi-step resampling. Achieves the best speed in our experiments.

API

# Load models
tokenizer, draft_model, target_model = load_models(
    draft_name="Qwen/Qwen3-0.6B",   # any HF model
    target_name="Qwen/Qwen3-4B",
    device="cuda",                   # or "cpu", "mps"
)

# Generate
result = speculative_generate(
    draft_model, target_model, tokenizer,
    prompt="...",
    max_new_tokens=200,
    gamma=5,                         # draft tokens per round
    method="standard",               # "standard" | "naive_hsd" | "capped_hsd"  
    draft_temperature=0.6,
    target_temperature=0.6,
)
# result keys: text, tokens_generated, tokens_per_sec, acceptance_rate, average_accepted_n

# Benchmark against target-only baseline
baseline_results, spec_results = benchmark(
    draft_model, target_model, tokenizer,
    prompts=[...],
    max_new_tokens=200,
    gamma=5,
    method="standard",
    verbose=True,
)

See examples/ for runnable scripts.

Project Structure

src/seqspecdecod/   # the source code for the package, including generation, sampling, benchmarking, model loading
notebooks/          # toy experiments, including several simple experiments for each method and gamma comparison
examples/           # runnable scripts showing basic usage and method comparison
docs/               # figures generated from the notebooks

References

1. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML.
2. Zhou, Y. et al. (2026). Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding.

License

MIT