Benchmark draft models

A speculative decoding claim needs two numbers: how much faster the target decodes, and evidence that its outputs did not change. This page benchmarks eight ways of serving Qwen3-8B on identical GPUs, covering every draft model family in these tutorials, and checks both.

Compare eight configurations

Configuration	What drafts	Training
`baseline`	Nothing. The target decodes one token per forward pass.	None
`ngram`	Matching n-grams copied from the prompt	None
`suffix`	A suffix tree over the prompt and earlier responses	None
`eagle3-public`	Red Hat’s general-purpose EAGLE-3 head	Pretrained
`eagle3-finetuned`	The head from Train an EAGLE-3 draft head	Fine-tuned here
`dflash-public`	Red Hat’s general-purpose DFlash head	Pretrained
`dflash-finetuned`	The drafter from Train a DFlash block drafter	Fine-tuned here
`dspark-trained`	The drafter from Train a DSpark drafter	From scratch here

The two draft-free methods matter as baselines. On repetitive traffic, they can match or beat a learned drafter without any training, so a learned drafter has to earn its place.

compare trains a drafter of every family, then benchmarks all eight configurations:

compare.py
                
                    
                
            
                
            

                
                    
                
            
@driver_env.task
async def compare(
    num_chat: int = 4000,
    num_agent: int = 4000,
    num_shards: int = 4,
    max_steps: int | None = None,
    n_bench: int = 78,
) -> File:
    """Train (or reuse) a drafter of every family, then benchmark them all."""
    dataset = await build_dataset(num_chat, num_agent, num_shards)
    with flyte.group("train"):
        eagle3, dflash, dspark = await asyncio.gather(
            *(
                train_drafter(data=dataset.files, method=m, max_steps=max_steps)
                for m in ("eagle3", "dflash", "dspark")
            )
        )
    prompts = await prepare_benchmark_prompts(n_bench)
    configs = [
        # No speculation, and two methods that need no training.
        ServerConfig(name="baseline"),
        ServerConfig(name="ngram", method="ngram", num_speculative_tokens=4),
        ServerConfig(name="suffix", method="suffix", num_speculative_tokens=16),
        # Each family's public head, and the one trained on this dataset.
        drafter_config("eagle3-public", "eagle3"),
        drafter_config("eagle3-finetuned", "eagle3", eagle3),
        drafter_config("dflash-public", "dflash"),
        drafter_config("dflash-finetuned", "dflash", dflash),
        drafter_config("dspark-trained", "dspark", dspark),
    ]
    return await run_benchmarks(configs, prompts)

train_drafter is cached. If you already ran pipeline.py, dflash.py, and dspark.py, compare calls it with the same data and arguments, reuses all three drafters, and goes straight to the benchmarks. Otherwise it trains the three in parallel, each on its own GPU.

Each configuration is a ServerConfig. A drafter configuration takes its draft length from its family’s recipe:

benchmark.py
                
                    
                
            
                
            

                
                    
                
            
@dataclass
class ServerConfig:
    """One way of serving the target model."""

    name: str
    method: str = ""  # "", "ngram", "suffix", or a drafter: "eagle3", "dflash", "dspark"
    num_speculative_tokens: int = NUM_SPECULATIVE_TOKENS
    drafter_hf: str = ""  # A Hugging Face draft model...
    drafter: Dir | None = None  # ...or one trained by this pipeline.

    def speculative_config(self, drafter_path: str | None) -> dict | None:
        if not self.method:
            return None
        config = {"method": self.method, "num_speculative_tokens": self.num_speculative_tokens}
        if self.method == "ngram":
            config |= {"prompt_lookup_min": 2, "prompt_lookup_max": 5}
        if self.method in RECIPES:
            config["model"] = drafter_path or self.drafter_hf
        return config

def drafter_config(name: str, method: str, drafter: Dir | None = None, hf: str = "") -> ServerConfig:
    """A configuration that serves a draft model, with its recipe's draft length."""
    recipe = RECIPES[method]
    return ServerConfig(
        name=name,
        method=method,
        num_speculative_tokens=recipe.num_speculative_tokens,
        drafter=drafter,
        drafter_hf=hf or (recipe.init_from if drafter is None else ""),
    )

Measure on held-out prompts

prepare_benchmark_prompts draws three workloads that the training data never touched:

benchmark.py
                
                    
                
            
                
            

                
                    
                
            
@data_env.task(cache="auto")
async def prepare_benchmark_prompts(n_per_workload: int = 78) -> File:
    """Held-out prompts for three workloads. None of them appear in the training data.

    The benchmark splits each workload across its concurrency levels, so 78 prompts give 26
    unseen prompts per level.
    """
    agent = load_agent_tasks("heldout", n_per_workload)
    prompts = {
        "chat": [{"messages": m} for m in load_mt_bench()[:n_per_workload]],
        "agent": [
            {
                "messages": [
                    {"role": "system", "content": "Complete the request by calling the provided tools."},
                    {"role": "user", "content": t["request"]},
                ],
                "tools": t["tools"],
            }
            for t in agent
        ],
        "math": load_gsm8k(n_per_workload),
    }
    path = "/tmp/benchmark-prompts.json"
    with open(path, "w") as f:
        json.dump(prompts, f)
    return await File.from_local(path)

chat: first-turn MT-Bench prompts, the standard for reporting acceptance length.
agent: held-out ToolACE requests with their tool schemas. The response is usually a JSON tool call.
math: GSM8K test questions, which also have checkable answers.

Run one configuration per GPU

benchmark_server starts vLLM with one configuration, warms it up, and replays every workload at concurrency 1, 4, and 16:

benchmark.py
                
                    
                
            
                
            

                
                    
                
            
@bench_env.task(cache="auto", report=True)
async def benchmark_server(
    config: ServerConfig,
    prompts: File,
    concurrencies: list[int] = [1, 4, 16],
    max_tokens: int = 256,
) -> File:
    """Serve the target with one configuration and replay every workload."""
    import httpx

    drafter_path = await config.drafter.download() if config.drafter else None
    server = start_server(
        speculative_config=config.speculative_config(drafter_path),
        extra_args=[*TOOL_ARGS, "--max-model-len", "8192", "--gpu-memory-utilization", "0.9"],
    )
    with open(await prompts.download()) as f:
        workloads = json.load(f)

    report: dict = {"config": config.name, "workloads": {}}

    async def measure(client, items: list[dict], concurrency: int) -> dict:
        semaphore = asyncio.Semaphore(concurrency)

        async def one(p):
            async with semaphore:
                return await _stream_one(client, server.base_url, p, max_tokens)

        before = read_counters(server.base_url)
        start = time.perf_counter()
        results = await asyncio.gather(*(one(p) for p in items))
        wall = time.perf_counter() - start
        spec = read_counters(server.base_url) - before
        return {
            **_summarize(results, wall),
            "acceptance_length": spec.acceptance_length,
            "acceptance_rate": spec.acceptance_rate,
            # Fraction of verification steps that accepted the token at position k.
            "acceptance_by_position": {
                str(k): v / spec.drafts for k, v in sorted(spec.accepted_per_pos.items())
            } if spec.drafts else {},
            "outputs": [r.output for r in results],
        }

    try:
        async with httpx.AsyncClient() as client:
            # Warm up CUDA graphs with a prompt outside every workload, so nothing measured
            # has been seen before.
            await asyncio.gather(*(_stream_one(client, server.base_url, WARMUP, 32) for _ in range(4)))
            for name, items in workloads.items():
                # Each concurrency level gets its own slice of prompts. A method that caches
                # earlier responses (suffix decoding) must never be measured on a prompt it
                # has already answered.
                runs = {}
                for i, c in enumerate(concurrencies):
                    runs[str(c)] = await measure(client, items[i :: len(concurrencies)], c)
                    print(f"{config.name} {name} c={c}: {runs[str(c)]['per_request_tok_s']:.1f} tok/s/request, AL={runs[str(c)]['acceptance_length']:.2f}")
                if config.method == "":
                    # The noise floor: the baseline, which caches nothing, replays its
                    # concurrency-1 prompts at the highest concurrency.
                    rerun = await measure(client, items[0 :: len(concurrencies)], concurrencies[-1])
                    runs["noise_floor_outputs"] = rerun["outputs"]
                report["workloads"][name] = runs
            if "math" in workloads:
                report["math_answers"] = [p["answer"] for p in workloads["math"][0 :: len(concurrencies)]]
    finally:
        server.stop()

    await flyte.report.replace.aio(server_report(report, config.speculative_config(drafter_path)))
    await flyte.report.flush.aio()
    path = f"/tmp/bench-{config.name}.json"
    with open(path, "w") as f:
        json.dump(report, f)
    return await File.from_local(path)

Each request streams, so the task records time to first token (TTFT), time per output token (TPOT), and decode speed per request. Before and after each pass, it reads vLLM’s speculative decoding counters and takes the difference:

vllm_server.py
                
                    
                
            
                
            

                
                    
                
            
@dataclass
class SpecDecodeCounters:
    """Cumulative speculative decoding counters from vLLM's `/metrics` endpoint."""

    drafts: float = 0.0
    draft_tokens: float = 0.0
    accepted_tokens: float = 0.0
    accepted_per_pos: dict[int, float] = field(default_factory=dict)

    def __sub__(self, other: "SpecDecodeCounters") -> "SpecDecodeCounters":
        return SpecDecodeCounters(
            drafts=self.drafts - other.drafts,
            draft_tokens=self.draft_tokens - other.draft_tokens,
            accepted_tokens=self.accepted_tokens - other.accepted_tokens,
            accepted_per_pos={
                k: v - other.accepted_per_pos.get(k, 0.0) for k, v in self.accepted_per_pos.items()
            },
        )

    @property
    def acceptance_length(self) -> float:
        """Mean tokens produced per verification step: 1 + accepted / drafts."""
        return 1.0 + self.accepted_tokens / self.drafts if self.drafts else 1.0

    @property
    def acceptance_rate(self) -> float:
        """Fraction of proposed draft tokens the target accepted."""
        return self.accepted_tokens / self.draft_tokens if self.draft_tokens else 0.0

_METRIC = re.compile(r'^(vllm:spec_decode_[a-z_]+?)(?:_total)?(\{[^}]*\})?\s+([0-9.eE+-]+)$')

def read_counters(base_url: str) -> SpecDecodeCounters:
    """Scrape vLLM's Prometheus endpoint, summing across label sets."""
    counters = SpecDecodeCounters()
    for line in httpx.get(f"{base_url}/metrics", timeout=10).text.splitlines():
        match = _METRIC.match(line)
        if not match:
            continue
        name, labels, value = match.group(1), match.group(2) or "", float(match.group(3))
        if name == "vllm:spec_decode_num_drafts":
            counters.drafts += value
        elif name == "vllm:spec_decode_num_draft_tokens":
            counters.draft_tokens += value
        elif name == "vllm:spec_decode_num_accepted_tokens":
            counters.accepted_tokens += value
        elif name == "vllm:spec_decode_num_accepted_tokens_per_pos":
            pos = re.search(r'position="(\d+)"', labels)
            if pos:
                k = int(pos.group(1))
                counters.accepted_per_pos[k] = counters.accepted_per_pos.get(k, 0.0) + value
    return counters

Mean acceptance length is 1 + accepted / drafts: the tokens each target forward pass produces, counting the one token the target always contributes. It is the headline draft quality metric because it does not depend on how many tokens the drafter proposes. Acceptance rate, accepted / proposed, falls as you propose more tokens even when the drafter improves.

Never replay a prompt a caching method has seen

Suffix decoding caches earlier responses and drafts from them. Replaying the same prompts at each concurrency level lets it draft the exact greedy answer it already produced. In an early version of this benchmark, that inflated its acceptance length from about 1.5 to 8.5. Each concurrency level here gets its own slice of prompts, and warm-up uses a prompt outside every workload.

run_benchmarks runs all eight configurations at once, each on its own GPU, and compares them in a report:

compare.py
                
@driver_env.task(report=True)
async def run_benchmarks(configs: list[ServerConfig], prompts: File) -> File:
    """Benchmark every configuration in parallel, then compare them in a report."""
    with flyte.group("benchmarks"):
        files = await asyncio.gather(*(benchmark_server(c, prompts) for c in configs))

    results = []
    for f in files:
        with open(await f.download()) as fh:
            results.append(json.load(fh))
    summary = summarize(results)

    await flyte.report.replace.aio(render_html(summary, title="Speculative decoding: every draft model family"))
    await flyte.report.flush.aio()

    path = "/tmp/summary.json"
    with open(path, "w") as fh:
        json.dump(summary, fh, indent=2)
    return await File.from_local(path)

        
python compare.py --smoke   # 20 training steps, 12 prompts per workload
python compare.py           # the full run

In the run, the three trainings sit in a train group and the eight benchmarks in a benchmarks group. Each benchmark has its own report, with decode speed and throughput by concurrency for each workload:

The benchmark_server report for dflash-finetuned: tokens per second and acceptance length per workload, by concurrency

run_benchmarks renders the comparison: the best speedup per workload, the GSM8K accuracy range, and speedup charts for every configuration:

The run_benchmarks report comparing all eight configurations

Verify that outputs are unchanged

Speculative sampling is lossless in theory: every accepted token is one the target would have produced. In practice, three checks confirm it:

report.py
                
                    
                
            
                
            

                
                    
                
            
def summarize(results: list[dict]) -> dict:
    """Compare every configuration against the no-speculation baseline."""
    by_name = {r["config"]: r for r in results}
    base = by_name["baseline"]
    summary: dict = {"configs": [r["config"] for r in results], "workloads": {}}

    for workload, base_runs in base["workloads"].items():
        rows = {}
        base_c1 = base_runs["1"]
        concurrencies = sorted((c for c in base_runs if c.isdigit()), key=int)
        for name, result in by_name.items():
            runs = result["workloads"][workload]
            row = {
                "acceptance_length": runs["1"]["acceptance_length"],
                "speedup": {
                    c: runs[c]["per_request_tok_s"] / base_runs[c]["per_request_tok_s"]
                    for c in concurrencies
                },
                "throughput_tok_s": {c: runs[c]["throughput_tok_s"] for c in concurrencies},
                "tpot_p50_ms": runs["1"]["tpot_p50_ms"],
                "ttft_p50_s": runs["1"]["ttft_p50_s"],
                # Losslessness: greedy outputs should match the baseline's. Over a long
                # output, one floating-point difference flips a token and the texts diverge,
                # so also measure how much of the output matches before that happens.
                "exact_match": _match_rate(runs["1"]["outputs"], base_c1["outputs"]),
                "prefix_agreement": _prefix_agreement(runs["1"]["outputs"], base_c1["outputs"]),
            }
            if workload == "math":
                answers = result["math_answers"]
                row["gsm8k_accuracy"] = sum(
                    gsm8k_correct(o, a) for o, a in zip(runs["1"]["outputs"], answers)
                ) / len(answers)
            rows[name] = row
        # The noise floor: the baseline against itself, same prompts, at a different batch
        # size. Batched kernels are not bit-identical across batch sizes, speculation or not.
        summary["workloads"][workload] = {
            "rows": rows,
            "noise_floor_exact_match": _match_rate(base_runs["noise_floor_outputs"], base_c1["outputs"]),
            "noise_floor_prefix_agreement": _prefix_agreement(base_runs["noise_floor_outputs"], base_c1["outputs"]),
            "concurrencies": concurrencies,
        }
    return summary

Agreement with the baseline. At temperature 0, each configuration’s outputs are compared with the baseline’s, prompt by prompt: whether they match exactly, and how much of each output matches before the first differing character.
The noise floor. The baseline replays its own prompts at concurrency 16 and is compared with itself the same way. Batched GPU kernels are not bit-identical across batch sizes, and over a few hundred tokens one rounding difference flips a token and the texts diverge. A lossless configuration agrees with the baseline about as well as the baseline agrees with itself, which on most GPUs is well below 100%.
Task accuracy. GSM8K accuracy should be the same within noise for every configuration. This is the check that matters most, because two greedy paths that diverge after a rounding difference are usually both correct, while a broken drafter or verifier costs accuracy.

Bit-exact comparisons need batch-invariant kernels

vLLM’s VLLM_BATCH_INVARIANT=1 makes outputs independent of batch size on Hopper and newer GPUs, at a throughput cost. Without it, expect exact match well below 100% for every configuration, including the baseline against itself.

Results

These numbers come from one full run of compare.py: the drafters trained on 11,728 rows regenerated by Qwen3-8B, and each configuration ran on one NVIDIA L40S. There were 78 prompts per workload, 26 at each concurrency, with up to 256 output tokens at temperature 0 and thinking turned off. Without speculation, Qwen3-8B decodes about 46 tokens per second per request.

Speedup is decode speed per request, relative to baseline at the same concurrency.

Chat (MT-Bench)

Configuration	Acceptance length	Speedup, c=1	c=4	c=16
`ngram`	1.12	1.06x	1.12x	1.06x
`suffix`	1.30	1.05x	1.15x	1.11x
`eagle3-public`	2.25	1.92x	1.98x	1.78x
`eagle3-finetuned`	2.39	2.03x	2.11x	1.88x
`dflash-public`	3.22	3.00x	2.81x	2.33x
`dflash-finetuned`	3.32	3.04x	2.87x	2.33x
`dspark-trained`	2.04	1.74x	1.73x	1.60x

Agent (ToolACE)

Configuration	Acceptance length	Speedup, c=1	c=4	c=16
`ngram`	2.21	1.95x	1.70x	1.49x
`suffix`	2.83	2.24x	1.98x	1.72x
`eagle3-public`	2.82	2.15x	1.85x	1.72x
`eagle3-finetuned`	3.24	2.40x	2.05x	1.76x
`dflash-public`	7.72	4.63x	3.05x	2.53x
`dflash-finetuned`	12.62	6.74x	4.10x	2.99x
`dspark-trained`	5.49	4.08x	2.98x	2.36x

Math (GSM8K)

Configuration	Acceptance length	Speedup, c=1	c=4	c=16
`ngram`	1.28	1.21x	1.27x	1.22x
`suffix`	1.58	1.33x	1.47x	1.37x
`eagle3-public`	2.46	2.03x	2.09x	1.99x
`eagle3-finetuned`	2.66	2.18x	2.26x	2.11x
`dflash-public`	5.87	4.35x	4.02x	3.61x
`dflash-finetuned`	5.97	4.39x	4.07x	3.63x
`dspark-trained`	2.39	1.99x	2.03x	1.88x

Outputs and accuracy

Prefix agreement is the mean fraction of each baseline output that a configuration reproduces before the first differing character, at concurrency 1. GSM8K accuracy is over the 26 questions at concurrency 1.

Configuration	Chat	Agent	Math	GSM8K accuracy
Baseline against itself, at c=16	50%	99%	73%
`baseline`	100%	100%	100%	18/26
`ngram`	60%	98%	75%	18/26
`suffix`	63%	100%	69%	17/26
`eagle3-public`	51%	99%	68%	17/26
`eagle3-finetuned`	51%	99%	68%	17/26
`dflash-public`	60%	99%	72%	18/26
`dflash-finetuned`	60%	99%	72%	18/26
`dspark-trained`	55%	99%	72%	17/26

Every configuration agrees with the baseline about as well as the baseline agrees with itself at a different batch size, and GSM8K accuracy differs by at most one question. The divergences are the floating-point noise described above, not lost accuracy. Accuracy is below Qwen3-8B’s published GSM8K score because thinking is off and answers are cut at 256 tokens; what matters here is that it is the same for every configuration.

Read the results

DFlash is the fastest family on every workload. It drafts 15 tokens per step and keeps enough of them to reach 3.0x on chat and 4.4x on math at concurrency 1.
Fine-tuning pays off most where traffic differs from the public head’s training data. On chat and math, which resemble the public heads’ data, fine-tuning adds a few percent. On agent traffic, it raises DFlash’s acceptance length from 7.7 to 12.6 tokens per step, and its speedup from 4.6x to 6.7x. Serve and retrain draft models keeps that gap open as traffic changes.
Agent traffic drafts best. Tool calls repeat tool names and argument names from the schema, and JSON syntax is predictable. Draft-free methods help only here: suffix matches the public EAGLE-3 head on agent traffic with no training, and barely helps on chat.
A drafter trained from scratch needs more data. dspark-trained saw 11,728 rows for five epochs. It beats EAGLE-3 on agent traffic but trails it on chat and math, where the public heads trained on far larger datasets. Warm-starting from a public head, as the EAGLE-3 and DFlash pipelines do, is the cheaper path when one exists.
Speedup shrinks at high concurrency. At concurrency 1, the GPU is mostly idle while it waits on memory, so verifying draft tokens is nearly free. By concurrency 16, that idle compute fills with other requests. Block drafters, which verify the most tokens per step, lose the most: fine-tuned DFlash falls from 6.7x to 3.0x on agent traffic.
Compare families at the concurrency you serve. Pick the configuration that wins where your traffic runs, on prompts that look like your traffic.