← Writing

Speculative Decoding for Agent Workloads

October 6, 2026· Agents· Local LLMs· Speculative Decoding
A drafter predicts tokens; the target checks the block, corrects a rejection and completes a green sequence beside it.

Hi! The goal of this project was to see how much of a coding agent's output a smaller model can predict, and whether those predictions actually save generation time. The data is my own: traces from the local Qwen 27B agents in my agent loop, split into thinking, text, code, tool calls, and tool results.

The experiments use 1.5B and 0.5B targets, so they measure the idea on agent traffic, not a speedup of the 27B agents themselves.

How Speculative Decoding Works

A language model normally generates one token at a time. With speculative decoding, a cheap drafter guesses the next few tokens and the target checks the whole block in one call. Every accepted guess is a token we didn't have to generate one by one, but the guesses aren't free either.

The acceptance rule matters: a draft token x is kept with probability min(1, p(x)/q(x)), and the first rejection is replaced by a sample from what's left of the target distribution. That keeps the output distribution exactly the target's (Leviathan et al., 2023), so speculation changes speed, not what the model says.

draft 1draft 2draft 3
The drafter predicts three tokens and keeps the distribution that produced each one. The target checks the pending token and all three drafts in one call.
PositionTarget p(x)Draft q(x)Uniform uu × q(x)Result
Draft 10.400.300.600.180.18 < 0.40: keep
Draft 20.350.400.500.200.20 < 0.35: keep
Draft 30.120.300.700.210.21 ≥ 0.12: repair
A synthetic example with three drafts. These probabilities illustrate the acceptance rule.

I built the target in JAX (Qwen2 with a KV cache that can roll back rejected drafts), three kinds of drafters, a Go gateway with metrics, and a Ray sweep over drafters, draft lengths k, and temperatures.

1. Agent Traffic Is Very Predictable, but Not Evenly

The main metric is overlap (α), the share of the target's probability mass the drafter agrees with at each position. It's measured on 14 later runs held out from training (56,990 positions), with confidence intervals that resample whole runs:

0%20%40%60%80%100%N-gram32.3%Tiny distilled1.1%LoRA71.2%LoRA + DPO71.3%
DrafterOverlap95% intervalCoverage
N-gram32.28%30.74% to 33.89%73.9%
Tiny distilled1.14%1.09% to 1.20%100.0%
LoRA71.20%70.61% to 71.87%100.0%
LoRA + DPO71.35%70.78% to 72.01%100.0%
Qwen2.5-1.5B-Instruct target. All drafters use the same 700 held-out blocks, 14 runs, and 56,990 scored positions. Lines show 95% run-bootstrap intervals. Hover or focus a point to see coverage.

A 0.5B model fine-tuned with LoRA on the agent traces overlaps with the target 71.2% of the time. Plain n-gram lookup, which just copies what came earlier in the context, gets 32.3%. A tiny model distilled from scratch gets 1.1%, so it's basically useless.

Tool calls are the most predictable part, which makes sense with all the repeated JSON and commands. Thinking and prose are the hardest:

ContentSupportN-gramTiny distilledLoRALoRA + DPO
Text14 runs
2,149 positions
19.8%
13.6% to 26.8%
1.9%
1.6% to 2.2%
62.8%
60.2% to 66.0%
63.1%
60.6% to 66.3%
Thinking13 runs
18,273 positions
16.3%
14.6% to 18.2%
1.6%
1.5% to 1.7%
64.3%
63.5% to 65.2%
64.7%
63.9% to 65.6%
Tool results13 runs
12,634 positions
27.3%
23.9% to 30.8%
1.2%
1.1% to 1.3%
72.0%
70.5% to 73.3%
72.1%
70.6% to 73.4%
Bash calls13 runs
22,279 positions
47.7%
45.8% to 49.6%
0.7%
0.6% to 0.7%
76.4%
75.6% to 77.3%
76.4%
75.6% to 77.4%
Edit calls4 runs
1,348 positions
56.9%
53.9% to 61.4%
0.6%
0.6% to 0.7%
81.7%
77.1% to 83.2%
81.6%
77.1% to 83.1%
Read calls5 runs
307 positions
56.0%
45.0% to 61.0%
0.5%
0.3% to 0.7%
77.4%
66.2% to 83.0%
77.4%
66.5% to 83.0%
Darker cells mean a higher estimate; 95% intervals appear below. Read calls have five supporting runs and Edit calls have four.

2. Predicting Isn't the Same as Being Fast

Overlap only says how often a guess is right. Whether that saves time depends on what the guesses cost, so I benchmarked each configuration head to head against plain decoding, in one process, on the same 34 held-out prompts, with the order interleaved across repetitions:

0×0.5×1×1.5×0.5B · N-gram k=81.38×1.5B · N-gram k=81.50×1.5B · LoRA k=21.58×1.5B · LoRA k=41.25×1.5B · LoRA k=4 T=0.71.38×
N-gramLoRA
Configurationtok/sTarget-only tok/sSpeedupAcceptanceTokens / step
0.5B · N-gram k=88.926.471.38×13.2%1.26
1.5B · N-gram k=82.851.901.50×20.2%1.36
1.5B · LoRA k=22.991.901.58×45.5%1.86
1.5B · LoRA k=42.381.901.25×30.9%2.14
1.5B · LoRA k=4 T=0.72.621.911.38×37.4%2.39
Wall-clock speedup vs target-only decoding on the same 34 held-out prompts, 32 tokens each, one process after compile warm-up, configurations interleaved over 3 repetitions. Lines: range across repetitions. CPU only.

With the 1.5B target, LoRA drafting at k = 2 is 1.58× faster and n-grams at k = 8 are 1.50× faster. The interesting part is that longer LoRA drafts get slower: at k = 4 each step emits more tokens (2.14 vs 1.86), but every extra draft token costs a full forward pass of the 0.5B drafter, so the speedup drops to 1.25×. N-grams cost almost nothing to draft, so they can afford long guesses even at 20% acceptance.

These are CPU numbers. On a GPU the target forward is memory-bound, so drafting is relatively cheaper and the best k should move up.

3. The Measurement Bugs Mattered More Than the Model

The first version of this report had numbers that looked great and were wrong. An audit of the code found six bugs, and each one changed how the results had to be read:

  • One token off. The evaluator compared token i with the logit row that predicts token i + 1. (7a84891)
  • Sampling wasn't lossless. Drafts were picked greedily but scored with soft probabilities, which inflated acceptance at temperature 0.7 and pulled samples toward greedy output. (1e9f5d3)
  • Error bars about 100× too narrow. The bootstrap had its dimensions swapped, so a "significant" DPO gain disappeared once it was fixed. (0158d68)
  • The cache, the timing, and the stop token. The drafter decoded a token twice, speed came from concurrent sweep workers, and serving ran past the end-of-sequence token. (ae371e3, 5f98414, 48cf15b)

The code fixes come with regression tests that fail on the old code, and the project's REPORT.md is now checked by a script that recomputes every number in it from the saved results. A blind LLM judge also found no quality difference between speculative and plain output (43% win rate, CI [33%, 54%]), which is what the acceptance rule guarantees.

What Didn't Work

  • DPO on top of LoRA did nothing measurable: 71.35% vs 71.20% overlap, with overlapping intervals. The pair builder also has a known bug, so this one needs a rebuild before it means much.
  • The distilled tiny drafter. Its validation loss improved, but at 1.1% overlap it's well below the size where imitation starts to work.
  • Recorded text overstates live acceptance. LoRA's 71% overlap on recorded traces becomes about 63% acceptance while generating, since the model's own output drifts away from what the drafter was trained on.

Future Improvement

  • Run the GPU path (a 7B target, vLLM speculative decoding, and load tests through the gateway)
  • Rebuild the DPO pairs and compare against an un-adapted 0.5B drafter
  • Try n-gram and LoRA together, using the cheap guess when it's available

Data and Code

Code, results, and the full report (every number traced to results/): github.com/michael951021/speculative-agent-workloads. The standalone report has the code diffs for each fix, and the evidence JSON and metrics CSV hold the aggregates. None of them contain prompts or transcript text.