Hi! The goal of this project was to see how much of a coding agent's output a smaller model can predict, and whether those predictions actually save generation time. The data is my own: traces from the local Qwen 27B agents in my agent loop, split into thinking, text, code, tool calls, and tool results.
The experiments use 1.5B and 0.5B targets, so they measure the idea on agent traffic, not a speedup of the 27B agents themselves.
How Speculative Decoding Works
A language model normally generates one token at a time. With speculative decoding, a cheap drafter guesses the next few tokens and the target checks the whole block in one call. Every accepted guess is a token we didn't have to generate one by one, but the guesses aren't free either.
The acceptance rule matters: a draft token x is kept with probability min(1, p(x)/q(x)), and the first rejection is replaced by a sample from what's left of the target distribution. That keeps the output distribution exactly the target's (Leviathan et al., 2023), so speculation changes speed, not what the model says.
| Position | Target p(x) | Draft q(x) | Uniform u | u × q(x) | Result |
|---|---|---|---|---|---|
| Draft 1 | 0.40 | 0.30 | 0.60 | 0.18 | 0.18 < 0.40: keep |
| Draft 2 | 0.35 | 0.40 | 0.50 | 0.20 | 0.20 < 0.35: keep |
| Draft 3 | 0.12 | 0.30 | 0.70 | 0.21 | 0.21 ≥ 0.12: repair |
I built the target in JAX (Qwen2 with a KV cache that can roll back rejected drafts), three kinds of drafters, a Go gateway with metrics, and a Ray sweep over drafters, draft lengths k, and temperatures.
1. Agent Traffic Is Very Predictable, but Not Evenly
The main metric is overlap (α), the share of the target's probability mass the drafter agrees with at each position. It's measured on 14 later runs held out from training (56,990 positions), with confidence intervals that resample whole runs:
| Drafter | Overlap | 95% interval | Coverage |
|---|---|---|---|
| N-gram | 32.28% | 30.74% to 33.89% | 73.9% |
| Tiny distilled | 1.14% | 1.09% to 1.20% | 100.0% |
| LoRA | 71.20% | 70.61% to 71.87% | 100.0% |
| LoRA + DPO | 71.35% | 70.78% to 72.01% | 100.0% |
A 0.5B model fine-tuned with LoRA on the agent traces overlaps with the target 71.2% of the time. Plain n-gram lookup, which just copies what came earlier in the context, gets 32.3%. A tiny model distilled from scratch gets 1.1%, so it's basically useless.
Tool calls are the most predictable part, which makes sense with all the repeated JSON and commands. Thinking and prose are the hardest:
| Content | Support | N-gram | Tiny distilled | LoRA | LoRA + DPO |
|---|---|---|---|---|---|
| Text | 14 runs 2,149 positions | 19.8% 13.6% to 26.8% | 1.9% 1.6% to 2.2% | 62.8% 60.2% to 66.0% | 63.1% 60.6% to 66.3% |
| Thinking | 13 runs 18,273 positions | 16.3% 14.6% to 18.2% | 1.6% 1.5% to 1.7% | 64.3% 63.5% to 65.2% | 64.7% 63.9% to 65.6% |
| Tool results | 13 runs 12,634 positions | 27.3% 23.9% to 30.8% | 1.2% 1.1% to 1.3% | 72.0% 70.5% to 73.3% | 72.1% 70.6% to 73.4% |
| Bash calls | 13 runs 22,279 positions | 47.7% 45.8% to 49.6% | 0.7% 0.6% to 0.7% | 76.4% 75.6% to 77.3% | 76.4% 75.6% to 77.4% |
| Edit calls | 4 runs 1,348 positions | 56.9% 53.9% to 61.4% | 0.6% 0.6% to 0.7% | 81.7% 77.1% to 83.2% | 81.6% 77.1% to 83.1% |
| Read calls | 5 runs 307 positions | 56.0% 45.0% to 61.0% | 0.5% 0.3% to 0.7% | 77.4% 66.2% to 83.0% | 77.4% 66.5% to 83.0% |
2. Predicting Isn't the Same as Being Fast
Overlap only says how often a guess is right. Whether that saves time depends on what the guesses cost, so I benchmarked each configuration head to head against plain decoding, in one process, on the same 34 held-out prompts, with the order interleaved across repetitions:
| Configuration | tok/s | Target-only tok/s | Speedup | Acceptance | Tokens / step |
|---|---|---|---|---|---|
| 0.5B · N-gram k=8 | 8.92 | 6.47 | 1.38× | 13.2% | 1.26 |
| 1.5B · N-gram k=8 | 2.85 | 1.90 | 1.50× | 20.2% | 1.36 |
| 1.5B · LoRA k=2 | 2.99 | 1.90 | 1.58× | 45.5% | 1.86 |
| 1.5B · LoRA k=4 | 2.38 | 1.90 | 1.25× | 30.9% | 2.14 |
| 1.5B · LoRA k=4 T=0.7 | 2.62 | 1.91 | 1.38× | 37.4% | 2.39 |
With the 1.5B target, LoRA drafting at k = 2 is 1.58× faster and n-grams at k = 8 are 1.50× faster. The interesting part is that longer LoRA drafts get slower: at k = 4 each step emits more tokens (2.14 vs 1.86), but every extra draft token costs a full forward pass of the 0.5B drafter, so the speedup drops to 1.25×. N-grams cost almost nothing to draft, so they can afford long guesses even at 20% acceptance.
These are CPU numbers. On a GPU the target forward is memory-bound, so drafting is relatively cheaper and the best k should move up.
3. The Measurement Bugs Mattered More Than the Model
The first version of this report had numbers that looked great and were wrong. An audit of the code found six bugs, and each one changed how the results had to be read:
- One token off. The evaluator compared token
iwith the logit row that predicts tokeni + 1. (7a84891) - Sampling wasn't lossless. Drafts were picked greedily but scored with soft probabilities, which inflated acceptance at temperature 0.7 and pulled samples toward greedy output. (1e9f5d3)
- Error bars about 100× too narrow. The bootstrap had its dimensions swapped, so a "significant" DPO gain disappeared once it was fixed. (0158d68)
- The cache, the timing, and the stop token. The drafter decoded a token twice, speed came from concurrent sweep workers, and serving ran past the end-of-sequence token. (ae371e3, 5f98414, 48cf15b)
The code fixes come with regression tests that fail on the old code, and the project's REPORT.md is now checked by a script that recomputes every number in it from the saved results. A blind LLM judge also found no quality difference between speculative and plain output (43% win rate, CI [33%, 54%]), which is what the acceptance rule guarantees.
What Didn't Work
- DPO on top of LoRA did nothing measurable: 71.35% vs 71.20% overlap, with overlapping intervals. The pair builder also has a known bug, so this one needs a rebuild before it means much.
- The distilled tiny drafter. Its validation loss improved, but at 1.1% overlap it's well below the size where imitation starts to work.
- Recorded text overstates live acceptance. LoRA's 71% overlap on recorded traces becomes about 63% acceptance while generating, since the model's own output drifts away from what the drafter was trained on.
Future Improvement
- Run the GPU path (a 7B target, vLLM speculative decoding, and load tests through the gateway)
- Rebuild the DPO pairs and compare against an un-adapted 0.5B drafter
- Try n-gram and LoRA together, using the cheap guess when it's available
Data and Code
Code, results, and the full report (every number traced to results/): github.com/michael951021/speculative-agent-workloads. The standalone report has the code diffs for each fix, and the evidence JSON and metrics CSV hold the aggregates. None of them contain prompts or transcript text.
