Speculative Decoding for Agent Workloads / Speculative decoding October 6 results, after the fixes
Speculative Decoding for Agent Workloads
A smaller model drafts several tokens and a larger one checks them in one call. The held-out results show how much the models agree; the controlled speed comparison is still pending.
Workload Local coding-agent trafficMain target Qwen2.5-1.5B-InstructEvaluation 14 held-out runs
71.2%
Overlap between the LoRA drafter and target. 95% interval across runs: 70.61–71.87%. 56,990 token positions scored.
Tool calls have more overlap.Even a lookup table can predict some of their repeated syntax.
The speed comparison is pending.The corrected sweep includes unfinished cells.
On October 6, the sampling, cache handling, and confidence intervals were corrected. This page uses the newer held-out alpha results and judge rerun, along with the unfinished sweep. The earlier speedup claims need a new controlled comparison, so they are left out here.
01
The Idea
The goal of this project was to see how much of a coding agent’s output a smaller model can predict, and whether those predictions can save generation time. The experiments use traces from the local agent loop.
A language model normally generates one token at a time, where a token can be part of a word, some punctuation, or a piece of code. Each new token depends on the previous ones. With speculative decoding, a cheaper model drafts a few tokens and the target checks the block in one call. We get more tokens out of that call when the drafts are accepted, although we also have to pay for drafting them.
Our local Qwen 27B coding agents supply the traces, including thinking, text, code, and tool traffic. The experiments here use 1.5B and 0.5B targets, so they do not measure a speedup of the original 27B agents.
How the pieces fit together
The experiments measure acceptance and generation. The serving path exposes the decoder through an API.
01 / COLLECTAgent traces → corpus
Split the traces into content types, remove duplicates, tokenize, and keep each run in one data split.
02 / DRAFTLookup or learned model
Try n-grams, a tiny distilled model, LoRA, and LoRA with DPO to predict the next few tokens.
03 / VERIFYJAX target + decoder
Check the drafts together, keep the accepted prefix, replace the first rejected token, and roll back the cache.
04 / MEASUREAlpha, sweep & judge
Measure how much the models overlap, what happens during generation, and whether the output still behaves correctly.
Qwen2 runs in JAX, with grouped-query attention, rotary positions, RMS normalization, a gated MLP, and a fixed-size key/value cache. Adapters trained in PyTorch are merged into the JAX weights for generation.
Serving & visibility
Requests go through a Go gateway that handles API keys, rate limits, routing, and a circuit breaker. It streams output and exposes Prometheus metrics. Grafana displays latency, token counts, acceptance, and backend health.
Experiment tooling
Ray runs the sweep configurations. Saved JSON and Parquet results feed a React explorer, while a separate benchmark compares configurations in one process after compilation warm-up.
Deployment tooling
Dockerfiles, a Helm chart, a host-only kind check, and Terraform/GKE configuration are included in the repo. This snapshot has no production deployment or GPU performance results.
02
Speculative Decoding
The draft has to be checked in a way that preserves the target’s output distribution. Otherwise the smaller model changes what we generate.
Call the target’s next-token distribution p, and the distribution that produced the draft q. A proposed token x is accepted with probability min(1, p(x)/q(x)). At the first rejection, we sample a replacement from the normalized max(0, p − q) and discard the rest of the block. If every draft passes, we sample a bonus token from the target.
At temperature zero, we keep a draft if it matches the target’s top choice. For sampling, the acceptance and replacement rule preserves the target distribution, so two runs can produce different text. The rule comes from Leviathan et al., 2023 and Chen et al., 2023.
One step, with three drafts
These are made-up probabilities to show how the rule works.
Proposed suffixdraft 1draft 2draft 3
The drafter predicts three tokens in sequence and keeps the distribution that produced each one. The target then checks the pending token and all three drafts in one call.
Position
Target p(x)
Draft q(x)
Uniform u
u × q(x)
Test
Draft 1
0.40
0.30
0.60
0.18
0.18 < 0.40 → keep
Draft 2
0.35
0.40
0.50
0.20
0.20 < 0.35 → keep
Draft 3
0.12
0.30
0.70
0.21
0.21 ≥ 0.12 → repair
Drafts after a rejection were predicted using a token we have discarded, so they are discarded too. A fully accepted block can emit k + 1 tokens, subject to the output limit.
The cache, and why one token stays pending
The target cache holds all the committed tokens except the newest one. That newest token stays pending, so the next call sends [pending_token] + drafts together. Logit row j predicts draft j, and the last row predicts the bonus. After accepting A drafts, we keep the pending token and accepted prefix in the cache, then the replacement or bonus becomes the next pending token.
Rollback changes the cache’s logical length without recomputing the prefix. Old values can stay in the buffers as long as the attention mask hides them, and the drafter’s cache must match the committed history too. The pending-token setup removed a second target call that used to process the bonus separately. See 1e9f5d3.
Triton verify and alpha kernels have PyTorch references. The alpha evaluator can use the fused kernel on CUDA; the JAX decoder currently verifies drafts in its own loop.
03
The Data
The corpus is split into thinking, text, fenced code, tool calls, and tool results. Duplicate blocks are removed, and blocks are capped at 4,096 tokens. Each whole run stays in one split, which keeps adjacent parts of a session from crossing between training and evaluation.
The first validation and test splits each had three runs, with one run contributing most of their tokens. The newer hold-out adds 14,974 blocks and 4.24 million tokens across 14 later runs. It excludes runs already present and exact (kind, text) duplicates from the older splits.
Where the training tokens come from
Most are thinking and tool output. This is the original training split, counted by tokens.
Source: results/corpus_stats.json. Tool results are things the agent reads, so being able to predict them does not directly save time generating the agent’s own output.
Split
Runs
Blocks
Tokens
Role in this edition
Measuring overlap
The alpha evaluation uses 700 held-out blocks and scores 56,990 token positions. Blocks are selected across runs, capped at 256 tokens, and given context from preceding selected blocks within a 1,024-token budget. At each position we give both models the recorded prefix and compare their predictions. This is teacher forcing, where the recorded tokens determine the history.
Alpha measures the overlap between their probability distributions:
α = Σₓ min(p(x), q(x)) = 1 − TV(p, q)
What 0.71 means
An alpha of 0.71 means about 71% of the probability mass overlaps on average across these positions. With correctly sampled proposals at those contexts, it gives expected acceptance. It is different from next-token accuracy against the recorded trace, and does not predict how much of every live draft block will pass.
How much it varies
The saved estimates average 1,000 bootstrap replicates. Each replicate samples whole runs with replacement and pools their token counts, keeping related parts of a session together. Larger runs still contribute more tokens.
Alpha here uses unscaled softmax, or temperature 1. The sweep uses temperatures 0 and 0.7, a different target, and generated continuations. The selected alpha blocks also contain no code row. These results apply to the sampled agent traffic.
04
Held-out Results
LoRA reaches 71.2% overlap with the 1.5B target. N-gram lookup gets 32.28%, and the tiny distilled model gets 1.14%.
Coverage means how often the drafter has a proposal. For n-grams, a position with no proposal counts as zero overlap, while the model drafters cover almost every valid position. Source: results/alpha/{ngram,lora,dpo,tiny}.json.
DPO adds 0.153 percentage points to the saved estimate. Its interval overlaps with LoRA’s, but separate intervals cannot settle whether the paired difference is statistically clear. The saved results lack that run-level comparison.
Tool calls are easier to predict
Overlap by content type, with a 95% interval under each number.
Content kind
Runs / positions
N-gram
Tiny
LoRA
LoRA + DPO
Darker cells mean more overlap. The models use the same scored positions. Read calls have five supporting runs and Edit calls have four; their intervals are wider. On a phone, choose a drafter to see its column.
Tool calls have more overlap than text or thinking, which is consistent with repeated syntax helping prediction. To separate the effect of training on agent traffic, we would need an unadapted 0.5B baseline and a matched comparison on non-agent text. Those comparisons are missing here.
05
Training the Drafters
Strategy
How it proposes
Why it is interesting
N-gram lookup
Look for a recent sequence of one to four tokens in earlier committed context, then propose what followed it.
A cheap way to use repetition, with no neural model call and no access to future text.
Tiny distilled
Train a small JAX student with 4 layers and width 256, using the same token vocabulary and its own embeddings.
Only 43.8M parameters. For distillation, save the teacher’s top-64 probabilities and the remaining tail mass, which is much smaller than saving every logit.
LoRA
Adapt a Qwen2.5-0.5B base model to the agent traffic, with rank 16 and scaling 32 on the attention and MLP projections.
Train small matrices that update the base weights. There is FSDP training code, along with a QLoRA path that needs a GPU.
LoRA + DPO
Use target-corrected tokens as preferred examples and rejected drafts as the other choice, with TRL DPO and β = 0.1.
Try to improve acceptance directly, although the code building those pairs still needs fixing (below).
Training the tiny student
Validation KL over 1,200 training steps. This measures a difference between distributions, where lower is better.
Standard initializationProjected teacher embedding initialization
The standard run went from 3.788 to 2.629, a 30.6% decrease. Initializing with projected teacher embeddings ended at 2.638. These curves show the earlier training runs; held-out overlap is measured separately. Sources: results/distill_curve.json, results/distill_curve_tinit.json.
The student’s validation KL falls by 30.6%, while its held-out overlap is only 1.14%. Its small capacity, training duration, and block-only training context could contribute to that gap. The experiment did not vary them separately.
For LoRA, the sweep merges the saved adapter into the JAX model with W ← W + (scaling/rank) · B · A, updating the attention and MLP weights. We can train in PyTorch and generate in JAX without a separate PyTorch drafter in the decoding loop.
DPO still has a problem to fix
The DPO pair builder chooses d = argmax(q), then applies an acceptance test using the soft probability q[d] and residual p − q. Since the token was chosen greedily, those probabilities do not describe how it was produced. The code also prefills a sequence and feeds its last token again when starting cached generation. These findings come from speculative_agent_workloads/drafters/dpo.py, which the October 6 decoder fix did not change. Training has not been rerun, so the pairs need checking and rebuilding before the DPO increase can support a claim about this method.
06
Generation and Timing
The corrected sweep generates 32 tokens after short held-out prompts, with draft lengths k ∈ {1, 2, 4, 8} and temperatures 0 and 0.7. Its Qwen2.5-0.5B-Instruct target is about the same size as the learned drafter. The grid measures decoder behavior; a larger target would give a better comparison for the cost of a smaller drafter.
Each finished prompt is saved to a sidecar file and folded into Parquet when its cell finishes or restarts. The report reads both formats and removes duplicates by configuration and prompt index.
The corrected sweep, including unfinished cells
0.5B-Instruct target · 32 generated tokens per prompt · 115 prompts in a complete cell
Drafter
k = 1
k = 2
k = 4
k = 8
26 complete8 partial8 pending42 configurations, including two target-only baselines
Tokens per step = emitted tokens / target steps. Finished cells use the same prompt set, while partial cells use different subsets. Striped cells are unfinished, and a pending cell has no measurement yet.
Sources: results/sweep.parquet and its 8 saved sidecars. The target-only baselines, 1.5B sweep, and controlled speed benchmark are still pending in this snapshot, so we do not have a measured speedup to report yet.
Once generation diverges from the recorded trace, the next prediction sees a different history. Alpha on recorded prefixes cannot describe all of those later contexts. The accepted/drafted ratio also counts drafts after the first rejection, even though they cannot be committed, so changing k can change that ratio while first-token overlap stays the same.
The identity control uses the target itself as the drafter, where acceptance should be one. At k = 8, a full step emits nine tokens, but the 32-token output limit requires four steps. The average is therefore 32 / 4 = 8 tokens per step.
Acceptance, draft cost, and draft length
A simplified cost model. The controls change assumptions, with no measured speed data behind them.
The model assumes constant, independent acceptance, a proposal at every position, and block verification costing the same as one target decode. It excludes prefill, output limits, EOS, compilation, cache work, and resource contention. Leviathan et al., generated-token and wall-time analysis.
tools/bench_speedup.py warms up the model shapes, alternates configurations over repeated passes, and compares each with target-only decoding on the same prompts. Its result file, results/speedup_bench.json, was absent from this snapshot. Timings from concurrent sweep workers cannot establish the runtime saving.
07
Checking the Output
The tests compare greedy output with the target, check the cache against a fresh prefill, check sampled distributions, and count target calls per step. Newer sampling tests use an identity drafter and check the second-token distribution of a mismatched drafter. A first-token test can miss incorrect state carried into the next step.
The judge rerun uses held-out prompts on the fixed engine, with a 0.5B-Instruct target, DPO drafter, k = 4, temperature 0.7, and 16 output tokens. The local qwen-agent judge receives blind comparisons in both orders, with cached responses and a 150-call cap. These are short-output preference checks, which do not measure completion of coding tasks.
What the judge preferred after the fix
Wins, ties, and losses for the speculative output.
Spec winsTieSpec loses
Source: results/judge.json, generated October 6. There were 150 calls used and 18 invalid responses, and the chart only counts included verdicts. The file’s order-disagreement rate includes invalid orders in its denominator; the counts separately show prompts with both orders valid.
The speculative-versus-target win-rate interval includes 50%, which does not establish equivalence or non-inferiority. Verdicts from both orders judge the same prompt and are correlated, while the saved Wilson intervals treat them as independent. Eight of the 39 prompts with both orders valid received conflicting judgments.
Distribution preservation depends on the acceptance rule and its implementation checks. The judge can help catch an obvious quality change in the short continuations.
What we mean by “lossless”
At temperature zero, output should match the target’s greedy tokens. At positive temperature, its distribution should match target-only sampling within numerical tolerance. Statistical tests check a finite sample. The commit adding the new checks records that they fail on the old implementation; this report uses saved evidence rather than a fresh run of the full test suite.
08
Fixing the Measurements
The commit history records several bugs that changed how the results had to be read.
The evaluator compared token i with logit row i, which predicts token i + 1. The low overlap came from comparing the wrong positions.
The fix lines up logits with next-token labels and skips the first position when there is no preceding context. Tests compare cross-entropy with Hugging Face’s shifted loss and check repeated versus shuffled text.
The drafter decoded the last committed token twice, and its position marker failed to advance correctly when every draft passed.
The fix corrected proposal positions and rollback. Tests compare the drafter’s cache and next proposal with a fresh prefill after rejection and full acceptance. The October 6 change then moved to the pending-token setup described earlier.
Drafts were chosen greedily but reported with soft probabilities and without matching temperature. The acceptance test used a distribution that had not produced the token, biasing sampled output toward greedy choices.
Model drafts now come from softmax(logits / T), and the verifier receives that distribution. At temperature zero, the proposal distribution is one-hot at the greedy token. The decoder sets the drafter’s temperature and seed. This change also removed the extra target call.
The bootstrap dimensions were swapped: n_runs replicates with n_boot draws each. Drawing so many times from the same few runs made their representation almost even in each replicate, collapsing the intervals around the estimate.
The corrected version makes n_boot replicates, each with n_runs session draws. A regression test uses two runs with very different acceptance rates and requires the interval to span both.
The first splits had few independent sessions. Selecting the first N prompts could concentrate a content type in one session, and the old speedup claims used timings from concurrent sweep workers.
The changes add a later hold-out, select prompts round-robin across runs, increase generation from 16 to 32 tokens, and add a 1.5B target option. The separate benchmark alternates configurations and includes target-only decoding.
Serving kept generating after end-of-sequence tokens and returned finish_reason: stop even at the length limit.
It now stops at the tokenizer’s EOS markers, excludes them from completion text, and returns stop or length accordingly. A no-drafter model is also available for comparisons.
09
Future Improvement
Question
Evidence needed
Snapshot status
Does speculation save wall time?
Run the fixed engine against target-only decoding on the same prompts, with a controlled timing comparison.
Benchmark results still pending.
Does a smaller drafter help a larger target?
Run the 1.5B sweep with the 0.5B adapters, then compare the timing.
1.5B sweep still pending.
Does DPO improve acceptance?
Check and rebuild the preference pairs, compare within runs, and include the unadapted model.
A small observed increase; still needs checking.
Do corpus scores predict generation?
Compare the fixed engine’s generated histories at the same positions and temperature.
The saved trajectories are from before the fixes.
Does the GPU path improve latency?
Benchmark the kernels and serving on a GPU, comparing target-only with speculative decoding.
The host script is there; no GPU results yet.
Can a new developer reproduce it?
Check a fresh clone, explicitly skip missing data or models, and run the full verification checks.
An open task lists four test failures from missing data.
10
Data and Code
The results come from revision e874bd1 and the files saved when the original report was built. Downloads include the revision, source hashes, confidence intervals, and sweep counts. They contain no prompts, transcript text, token sequences, or run identifiers. The standalone report includes its charts and data for offline use.
Data snapshot: . This page keeps those saved results even as new experiments finish.
Rebuilding the original data report
# From the Speculative Decoding for Agent Workloads project, with its Python dependencies available:
python tools/build_presentation.py
# Outputs:
# report/index.html # portable report
# report/data.json # aggregate evidence, metadata and commit diffs
# report/metrics.csv # full-precision metrics, intervals and status
# Optional fresh data report, at another path:
python tools/build_presentation.py --publish ../../reports/speculative-agent-workloads-report-fresh.html
The builder reads the saved results, checks the alpha support, and checks whether the pending benchmark files have appeared. These commands rebuild its original presentation; this rewrite is saved separately.
Commands for the measurements
These are the commands that produce the measurements. The report uses saved files, so writing it did not rerun training, generation, or judging.
Recoverable partial sweep cell; read without modifying it
09db99171a10
results/sweep.parquet.part-lora-2-0.7.jsonl
Recoverable partial sweep cell; read without modifying it
7bd28a066b29
results/sweep.parquet.part-lora-4-0.0.jsonl
Recoverable partial sweep cell; read without modifying it
713d3f1129da
results/sweep.parquet.part-lora-4-0.7.jsonl
Recoverable partial sweep cell; read without modifying it
1123dcb21ea3
results/sweep.parquet.part-lora-8-0.0.jsonl
Recoverable partial sweep cell; read without modifying it
67d1af2b1ed2
results/sweep.parquet.part-lora-8-0.7.jsonl
Recoverable partial sweep cell; read without modifying it
0db87d2ca673
results/sweep.parquet.part-lora_dpo-1-0.0.jsonl
Recoverable partial sweep cell; read without modifying it
54f4914cebd9
results/sweep.parquet.part-lora_dpo-1-0.7.jsonl
Recoverable partial sweep cell; read without modifying it
cc0712dd3042
results/judge.json
Corrected held-out judge rerun
5e6f7777e661
results/distill_curve.json
Recorded distillation training curve
90784039de12
results/distill_curve_tinit.json
Recorded distillation training curve
8443bb1b7392
ckpt/lora/adapter_config.json
Implementation or outstanding work used for this report
6439004389d3
ckpt/dpo/adapter_config.json
Implementation or outstanding work used for this report
4cf534ec2e63
speculative_agent_workloads/jaxlm/spec.py
Implementation or outstanding work used for this report
1171a5da035e
speculative_agent_workloads/jaxlm/model.py
Implementation or outstanding work used for this report
9a4218ae0d7b
speculative_agent_workloads/eval/alpha.py
Implementation or outstanding work used for this report
8d4a9e9146a2
speculative_agent_workloads/data/corpus.py
Implementation or outstanding work used for this report
eb1ea87d5969
speculative_agent_workloads/drafters/dpo.py
Implementation or outstanding work used for this report
96fbc3e9d5f6
speculative_agent_workloads/drafters/lora.py
Implementation or outstanding work used for this report
8f6d2ee01ff8
speculative_agent_workloads/drafters/distill.py
Implementation or outstanding work used for this report
4b1716c2da98
speculative_agent_workloads/kernels/verify.py
Implementation or outstanding work used for this report
036c1e6e5d46
speculative_agent_workloads/eval/judge.py
Implementation or outstanding work used for this report
85fca0709679
speculative_agent_workloads/serve/app.py
Implementation or outstanding work used for this report
d51e2d288b0e
speculative_agent_workloads/sweep.py
Implementation or outstanding work used for this report
ff6886f87297
gateway/metrics.go
Implementation or outstanding work used for this report
2812c9aa9afc
TODO.md
Implementation or outstanding work used for this report
feed15cd0c06
Source paths use the current package name. Hashes refer to the saved snapshot; the evidence JSON retains its original paths and full hashes. The held-out Parquet file stays local, and only aggregate columns were read to count the corpus.
serve: stop at EOS, real finish_reason, '+none' drafter
You can expand each change to see its code and test diff. These preserve the original implementation and paths at that commit, which later changes may have updated.
diff --git a/draftlab/jaxlm/spec.py b/draftlab/jaxlm/spec.py
index 255289a..a24a63e 100644
--- a/draftlab/jaxlm/spec.py
+++ b/draftlab/jaxlm/spec.py
@@ -4,9 +4,8 @@ Drafter protocol: ``propose(ctx, k) -> (tokens, probs)`` — up to k draft
tokens plus a (V,) float32 draft distribution for each; ``rollback(n_kept)``
undoes draft-side state (KV cache) added for the drafted suffix beyond the
-n_kept drafts that were committed. The engine scores all m drafted tokens plus
-the bonus position with one target call; accepts while u < p/q, then either
-resamples from max(0, p - q) at the first rejection or draws the bonus from p.
-Target-cache rollback is truncating the positional cache's `len`; a
-fully-accepted step decodes the bonus in.
+n_kept drafts that were committed. The engine scores the pending token plus all m drafted
+tokens with one target call; accepts while u < p/q, then either resamples
+from max(0, p - q) at the first rejection or draws the bonus from p.
+Target-cache rollback is truncating the positional cache's `len`.
"""
@@ -84,36 +83,65 @@ class NGramDrafter:
-class ModelDrafter:
- """A second JAX Qwen2 proposes its own greedy continuation.
+class NullDrafter:
+ """No drafting: plain autoregressive target decoding."""
+
+ def propose(self, ctx: list[int], k: int) -> tuple[list[int], list[np.ndarray]]:
+ return [], []
- Keeps its own KV cache, extended from committed tokens; a partial
- acceptance restores the pre-draft snapshot so the committed suffix is
- re-decoded at the next propose.
+ def rollback(self, n_kept: int) -> None:
+ pass
+
+
+class ModelDrafter:
+ """A second JAX Qwen2 proposes its own continuation.
+
+ T=0: argmax drafts with one-hot q. T>0: drafts sampled from
+ softmax(logits/T) and q is exactly that distribution, so accept/reject is
+ lossless (SpecDecoder sets T and the seed via `configure`). Keeps its own
+ KV cache; rollback truncates it to the committed drafts (no re-decode).
+ The last draft is never decoded here: on full acceptance the next propose
+ decodes it together with the bonus token.
"""
- def __init__(self, dcfg: M.Qwen2Cfg, dparams: dict, max_len: int) -> None:
+ def __init__(self, dcfg: M.Qwen2Cfg, dparams: dict, max_len: int,
+ temperature: float = 0.0, seed: int = 0) -> None:
self.dcfg = dcfg
self.dparams = dparams
self.cache = M.empty_cache(dcfg, max_len)
self._last = 0
- self._undo: tuple[dict, int] | None = None
+ self._undo: tuple[int, int] | None = None
+ self.configure(temperature, seed)
+
+ def configure(self, temperature: float, seed: int) -> None:
+ self.temperature = temperature
+ self._rng = np.random.default_rng(seed)
+
+ def _pick(self, row) -> tuple[int, np.ndarray]:
+ if self.temperature == 0.0:
+ t = int(np.argmax(np.asarray(row)))
+ q = np.zeros(row.shape[-1], np.float32)
+ q[t] = 1.0
+ return t, q
+ q = np.asarray(jax.nn.softmax(row.astype(jnp.float32) / self.temperature), np.float64)
+ q /= q.sum()
+ t = int(self._rng.choice(q.size, p=q))
+ return t, q.astype(np.float32)
def propose(self, ctx: list[int], k: int) -> tuple[list[int], list[np.ndarray]]:
- # ctx grows by >=1 committed token between proposes (first call: prompt),
- # so extending the cache to len(ctx) gives the dist over the FIRST draft
- # token on its last row; each draft then decodes the previous one.
+ # extending the cache to len(ctx) gives the dist over the FIRST draft on
+ # its last row; drafts 1..k-1 each decode the previous draft
if len(ctx) <= self._last:
raise RuntimeError("propose called without new committed tokens")
lg, self.cache = M.decode(self.dcfg, self.dparams, self.cache, ctx[self._last :])
self._last = len(ctx)
- self._undo = (self.cache, self._last, k)
toks: list[int] = []
probs: list[np.ndarray] = []
- for _ in range(k):
- p = np.asarray(jax.nn.softmax(lg[-1]), np.float32)
- t = int(np.argmax(p))
+ for j in range(k):
+ t, q = self._pick(lg[-1])
toks.append(t)
- probs.append(p)
- lg, self.cache = M.decode(self.dcfg, self.dparams, self.cache, [t])
+ probs.append(q)
+ if j < k - 1:
+ lg, self.cache = M.decode(self.dcfg, self.dparams, self.cache, [t])
+ self._undo = (self._last, len(toks))
return toks, probs
@@ -121,10 +149,9 @@ class ModelDrafter:
if self._undo is None:
return
- cache, last, m = self._undo
+ last, m = self._undo
self._undo = None
- if n_kept == m: # all drafts committed; the cache already holds them
- self._last = last + m # only the bonus is new; extend from after the drafts
- return
- self.cache, self._last = cache, last
+ keep = min(n_kept, m - 1) # cached drafts that were committed
+ self._last = last + keep
+ self.cache = {"k": self.cache["k"], "v": self.cache["v"], "len": jnp.int32(self._last)}
@@ -145,8 +172,9 @@ class SpecDecoder:
"""Speculative sampling over one jitted target model.
- temperature > 0 scales the TARGET sampling distribution (softmax(lg/T));
- draft probs stay unscaled. Accept/reject + residual max(0, p - q) is
- lossless for any draft q, so output still follows exactly the target's
- T-scaled distribution.
+ temperature > 0 samples the target's softmax(lg/T); the drafter is
+ configured to draft at the same T and report its true q, so accept/reject
+ + residual max(0, p - q) is lossless. One target forward per step: the
+ cache holds every committed token but the newest (`pend`), and each step
+ decodes [pend] + drafts, so row j scores draft j and row m is the bonus.
"""
@@ -168,4 +196,6 @@ class SpecDecoder:
self.temperature = temperature
self.rng = jax.random.PRNGKey(seed)
+ if hasattr(drafter, "configure"):
+ drafter.configure(temperature, seed + 1)
def _steps(self, prompt: list[int], n_tokens: int):
@@ -176,8 +206,7 @@ class SpecDecoder:
rng = self.rng
ctx = list(prompt)
- pre_lg, cache = M.prefill(cfg, params, ctx, self.max_len)
- prev_lg = pre_lg[-1] # dist over position L (first generated token)
+ _, cache = M.prefill(cfg, params, ctx, self.max_len)
+ cache = {"k": cache["k"], "v": cache["v"], "len": jnp.int32(len(ctx) - 1)}
out: list[int] = []
- stats: list[StepStats] = []
while len(out) < n_tokens:
rem = n_tokens - len(out)
@@ -190,51 +219,30 @@ class SpecDecoder:
dtoks, dprobs = drafter.propose(ctx, k)
m = len(dtoks)
- if m == 0: # no draft: draw from the held last row, then one decode
- if T0:
- tok = int(np.argmax(np.asarray(prev_lg)))
- else:
- tok = _categorical(step_rng, jax.nn.softmax(prev_lg / T))
- lg, cache = M.decode(cfg, params, cache, [tok])
- prev_lg = lg[-1]
- emitted, A = [tok], 0
+ rows, cache2 = M.decode(cfg, params, cache, [ctx[-1]] + dtoks)
+ if T0:
+ A = 0
+ while A < m and dtoks[A] == int(np.argmax(np.asarray(rows[A]))):
+ A += 1
+ bonus = int(np.argmax(np.asarray(rows[A])))
else:
- # one target call over the m drafts: row r is the dist over the
- # token at position L+r+1, i.e. draft r+1... so draft j's dist is
- # prev (j=0) or row j-1, and row m-1 is the bonus dist
- scored, cache2 = M.decode(cfg, params, cache, dtoks)
- Pd = [prev_lg if j == 0 else scored[j - 1] for j in range(m)]
- if T0:
- A = 0
- while A < m and dtoks[A] == int(np.argmax(np.asarray(Pd[A]))):
+ A = 0
+ for i, q in enumerate(dprobs):
+ p_i = np.asarray(jax.nn.softmax(rows[i] / T), np.float32)
+ u = float(jax.random.uniform(step_rng))
+ step_rng = jax.random.fold_in(step_rng, i + 1)
+ if u * q[dtoks[i]] < p_i[dtoks[i]]:
A += 1
- bonus = int(np.argmax(np.asarray(Pd[A] if A < m else scored[m - 1])))
- else:
- P = [np.asarray(jax.nn.softmax(row / T), np.float32) for row in Pd]
- A = 0
- for i, q in enumerate(dprobs):
- u = float(jax.random.uniform(step_rng))
- step_rng = jax.random.fold_in(step_rng, i + 1)
- if u * q[dtoks[i]] < P[i][dtoks[i]]:
- A += 1
- else:
- break
- if A == m:
- bonus = _categorical(step_rng, jax.nn.softmax(scored[m - 1] / T))
else:
- resid = np.maximum(0.0, P[A] - dprobs[A])
- bonus = _categorical(step_rng, resid / resid.sum())
- emitted = dtoks[:A] + [bonus]
- E = len(emitted) if A < m else min(rem, m + 1)
- emitted = emitted[:E]
- # Roll back the target cache to the A committed drafts, then
- # commit the bonus (its key/value only exists after this decode;
- # it was not part of the scoring call).
- trunc = {"k": cache2["k"], "v": cache2["v"], "len": jnp.int32(len(ctx) + A)}
- prev2, cache_b = M.decode(cfg, params, trunc, [bonus])
- if A == m and rem >= m + 1:
- cache, prev_lg = cache_b, prev2[-1]
- else:
- cache = {"k": cache_b["k"], "v": cache_b["v"], "len": jnp.int32(len(ctx) + E)}
- prev_lg = prev2[-1] if len(out) + E < n_tokens else None
+ resid = np.maximum(0.0, p_i - q)
+ break
+ if A == m:
+ bonus = _categorical(step_rng, jax.nn.softmax(rows[m] / T))
+ elif resid.sum() > 0:
+ bonus = _categorical(step_rng, resid / resid.sum())
+ else: # p == q on the draft's support: residual is p itself
+ bonus = _categorical(step_rng, p_i)
+ emitted = (dtoks[:A] + [bonus])[:rem]
+ # keep [pend] + the A accepted drafts; the bonus becomes the new pend
+ cache = {"k": cache2["k"], "v": cache2["v"], "len": jnp.int32(len(ctx) + A)}
drafter.rollback(A)
ctx.extend(emitted)
@@ -242,9 +250,8 @@ class SpecDecoder:
dt_ms = (time.perf_counter() - t0) * 1e3
st = StepStats(m, A, [x < A for x in range(m)], dt_ms)
- stats.append(st)
yield emitted, st
self.rng = rng
self.last_ctx = ctx # prompt + generated
- self.last_cache = cache
+ self.last_cache = cache # holds last_ctx[:-1] (the newest token is pending)
def generate(self, prompt: list[int], n_tokens: int) -> tuple[list[int], list[StepStats]]:
diff --git a/draftlab/tests/test_spec.py b/draftlab/tests/test_spec.py
index b8ad5f0..f9f510b 100644
--- a/draftlab/tests/test_spec.py
+++ b/draftlab/tests/test_spec.py
@@ -118,6 +118,6 @@ def test_rollback_cache_equals_fresh_prefill(kind):
_, ref_cache = M.prefill(cfg, params, full, max_len)
c = dec.last_cache
- assert int(c["len"]) == len(full)
- n = len(full)
+ n = len(full) - 1 # the newest token is pending (decoded by the next step)
+ assert int(c["len"]) == n
for key in ("k", "v"):
diff = float(abs(np.asarray(c[key])[:, :, :n] - np.asarray(ref_cache[key])[:, :, :n]).max())
@@ -167,11 +167,11 @@ def test_model_drafter_full_accept_steps_keep_cache_in_sync():
assert out == _greedy(cfg, params, prompt, 20)
assert all(s.drafted == 1 and s.accepted == 1 for s in st) # strict, no rejections
- # The drafter ends one token behind the final bonus: its cache must equal a
- # fresh prefill of that prefix (a desynced rollback re-decodes already
- # cached drafts and the KV diverges from the first duplicated slot on).
+ # The drafter's cache must equal a fresh prefill of the committed prefix it
+ # claims to hold (a desynced rollback re-decodes cached drafts and the KV
+ # diverges from the first duplicated slot on).
full = prompt + out
- n = len(full) - 1
d = dec.drafter
- assert int(d.cache["len"]) == n and d._last == n
+ n = d._last
+ assert int(d.cache["len"]) == n and 0 < n < len(full)
_, ref = M.prefill(d.dcfg, d.dparams, full[:n], 84)
for key in ("k", "v"):
@@ -198,2 +198,78 @@ def test_model_drafter_first_proposal_position():
jax.numpy.asarray(prompt + [toks[0]], jax.numpy.int32))
assert toks[1] == int(np.argmax(np.asarray(lg2[-1])))
+
+
+def _probs(cfg, params, ids, T=1.0):
+ lg, _ = M.step(cfg, params, M.empty_cache(cfg, len(ids) + 1),
+ jax.numpy.asarray(ids, jax.numpy.int32))
+ return np.asarray(jax.nn.softmax(lg[-1].astype(np.float64) / T), np.float64)
+
+
+def test_model_drafter_lossless_second_token_chi_square():
+ """T=0.8, model drafter != target, k=3: the marginal of the SECOND emitted
+ token equals sum_x p(x) p(y|x) exactly (covers draft rows, rejection +
+ residual, and the pending-token cache layout across steps)."""
+ cfg, params = _tiny16(1)
+ dcfg, dparams = _tiny16(42)
+ T = 0.8
+ prompt = [i % 7 for i in range(20)]
+ p1 = _probs(cfg, params, prompt, T)
+ exp_p = sum(p1[x] * _probs(cfg, params, prompt + [x], T) for x in range(cfg.vocab_size))
+ N = 6000
+ obs = np.zeros(cfg.vocab_size, np.int64)
+ acc = drafted = 0
+ for i in range(N):
+ dec = SpecDecoder(cfg, params, ModelDrafter(dcfg, dparams, len(prompt) + 8),
+ max_len=len(prompt) + 8, k=3, temperature=T, seed=i)
+ out, st = dec.generate(prompt, 2)
+ obs[out[1]] += 1
+ acc += sum(s.accepted for s in st)
+ drafted += sum(s.drafted for s in st)
+ assert 0 < acc < drafted, "need both accepts and rejects"
+ exp = N * exp_p
+ chi2 = float(((obs - exp) ** 2 / exp).sum())
+ pval = float(stats.chi2.sf(chi2, df=cfg.vocab_size - 1))
+ assert pval > 0.01, f"chi2={chi2:.1f}, p={pval:.4g} (second-token marginal shifted)"
+
+
+def test_identity_drafter_at_temperature_chi_square():
+ """Regression: argmax drafts + soft q made drafter==target accept the argmax
+ at T<1 every time (output collapsed to greedy). The first token must follow
+ softmax(lg/0.7) when the drafter is the target itself."""
+ cfg, params = _tiny16(1)
+ prompt = [i % 7 for i in range(32)]
+ pT = _probs(cfg, params, prompt, 0.7)
+ N = 4000
+ obs = np.zeros(cfg.vocab_size, np.int64)
+ for i in range(N):
+ dec = SpecDecoder(cfg, params, ModelDrafter(cfg, params, len(prompt) + 8),
+ max_len=len(prompt) + 8, k=2, temperature=0.7, seed=i)
+ out, st = dec.generate(prompt, 1)
+ obs[out[0]] += 1
+ exp = N * pT
+ chi2 = float(((obs - exp) ** 2 / exp).sum())
+ pval = float(stats.chi2.sf(chi2, df=cfg.vocab_size - 1))
+ assert pval > 0.01, f"chi2={chi2:.1f}, p={pval:.4g} (identity drafter not sampling at T)"
+
+
+@pytest.mark.parametrize("kind", ["ngram", "model", "none"])
+def test_one_target_forward_per_step(kind, monkeypatch):
+ from draftlab.jaxlm import spec as S
+
+ cfg, params = _tiny16(2)
+ calls = []
+ real = M.decode
+
+ def counting(c, p, cache, ids):
+ if p is params:
+ calls.append(len(ids))
+ return real(c, p, cache, ids)
+
+ monkeypatch.setattr(S.M, "decode", counting)
+ prompt = [i % 7 for i in range(30)]
+ d = S.NullDrafter() if kind == "none" else _make_drafter(kind, cfg, 80)
+ dec = SpecDecoder(cfg, params, d, max_len=80, k=3, temperature=1.0, seed=1)
+ out, st = dec.generate(prompt, 25)
+ assert len(out) == 25
+ assert len(calls) == len(st)
+ assert [c - 1 for c in calls] == [s.drafted for s in st]
0158d68 alpha: fix run bootstrap (CIs were ~100x too narrow)
diff --git a/draftlab/eval/alpha.py b/draftlab/eval/alpha.py
index b0c260f..139486b 100644
--- a/draftlab/eval/alpha.py
+++ b/draftlab/eval/alpha.py
@@ -271,5 +271,6 @@ def _bootstrap(
sums_v = np.array([np.concatenate([r[3] for r in by_run[rn]]).sum() for rn in runs])
cnts = np.array([sum(r[0].size for r in by_run[rn]) for rn in runs])
- w = rng.multinomial(n_boot, np.full(len(runs), 1.0 / len(runs)), size=len(runs))
+ # (n_boot, n_runs): each replicate draws len(runs) runs with replacement
+ w = rng.multinomial(len(runs), np.full(len(runs), 1.0 / len(runs)), size=n_boot)
# per-run rates over the drafter's valid (proposed) positions: this is
# what the sweep's accept = n_acc/n_draft measures (spec.py only scores
@@ -282,6 +283,6 @@ def _bootstrap(
w @ sums_gc / (w @ cnts),
w @ sums_v / (w @ cnts),
- w @ gv / n_boot,
- w @ av / n_boot,
+ w @ gv / len(runs),
+ w @ av / len(runs),
int(cnts.sum()),
len(runs),
diff --git a/draftlab/tests/test_alpha.py b/draftlab/tests/test_alpha.py
index 5d9f914..fb475a6 100644
--- a/draftlab/tests/test_alpha.py
+++ b/draftlab/tests/test_alpha.py
@@ -99,2 +99,21 @@ def test_identical_models_alpha_one():
assert valid2.all()
assert a2.min() >= 1.0 - 1e-6 and g2.all()
+
+
+def test_bootstrap_resamples_runs():
+ """Regression: weights were (n_runs, n_runs) with n_boot draws per row, so
+ every 'replicate' was ~equal weights and CIs collapsed to the point
+ estimate. Two runs at alpha 0.2 / 0.8: n_boot replicates, CI spans both."""
+ from draftlab.eval.alpha import _bootstrap
+
+ def rec(run, a):
+ x = np.full(100, a)
+ return (run, "text", x, x > 0.5, x > 0.5, np.ones(100, bool))
+
+ out = _bootstrap([rec("r1", 0.2), rec("r2", 0.8)], n_boot=2000, seed=0)
+ b_alpha = out["overall"][0]
+ assert b_alpha.shape == (2000,)
+ lo, hi = np.percentile(b_alpha, [2.5, 97.5])
+ assert lo == pytest.approx(0.2) and hi == pytest.approx(0.8)
+ assert out["overall"][5].shape == (2000,) # alpha over valid positions, per replicate
+ assert np.mean(out["overall"][5]) == pytest.approx(0.5, abs=0.02)