Speculative Decoding for Agent WorkloadsProject report · October 6, 2026

Speculative Decoding for Agent Workloads / Speculative decoding October 6 results, after the fixes

Speculative Decoding
for Agent Workloads

A smaller model drafts several tokens and a larger one checks them in one call. The held-out results show how much the models agree; the controlled speed comparison is still pending.

Workload Local coding-agent trafficMain target Qwen2.5-1.5B-InstructEvaluation 14 held-out runs
71.2%

Overlap between the LoRA drafter and target.
95% interval across runs: 70.61–71.87%.
56,990 token positions scored.

Tool calls have more overlap.Even a lookup table can predict some of their repeated syntax.

The speed comparison is pending.The corrected sweep includes unfinished cells.

On October 6, the sampling, cache handling, and confidence intervals were corrected. This page uses the newer held-out alpha results and judge rerun, along with the unfinished sweep. The earlier speedup claims need a new controlled comparison, so they are left out here.
01

The Idea

The goal of this project was to see how much of a coding agent’s output a smaller model can predict, and whether those predictions can save generation time. The experiments use traces from the local agent loop.

A language model normally generates one token at a time, where a token can be part of a word, some punctuation, or a piece of code. Each new token depends on the previous ones. With speculative decoding, a cheaper model drafts a few tokens and the target checks the block in one call. We get more tokens out of that call when the drafts are accepted, although we also have to pay for drafting them.

Our local Qwen 27B coding agents supply the traces, including thinking, text, code, and tool traffic. The experiments here use 1.5B and 0.5B targets, so they do not measure a speedup of the original 27B agents.

How the pieces fit together

The experiments measure acceptance and generation. The serving path exposes the decoder through an API.

01 / COLLECTAgent traces → corpus

Split the traces into content types, remove duplicates, tokenize, and keep each run in one data split.

02 / DRAFTLookup or learned model

Try n-grams, a tiny distilled model, LoRA, and LoRA with DPO to predict the next few tokens.

03 / VERIFYJAX target + decoder

Check the drafts together, keep the accepted prefix, replace the first rejected token, and roll back the cache.

04 / MEASUREAlpha, sweep & judge

Measure how much the models overlap, what happens during generation, and whether the output still behaves correctly.

Serving pathAPI client→Go gateway→FastAPI engine / optional vLLM backend→Streamed completion

Implementation: speculative_agent_workloads/data/, speculative_agent_workloads/drafters/, speculative_agent_workloads/jaxlm/, speculative_agent_workloads/eval/, speculative_agent_workloads/sweep.py, gateway/.

Model runtime
Qwen2 runs in JAX, with grouped-query attention, rotary positions, RMS normalization, a gated MLP, and a fixed-size key/value cache. Adapters trained in PyTorch are merged into the JAX weights for generation.
Serving & visibility
Requests go through a Go gateway that handles API keys, rate limits, routing, and a circuit breaker. It streams output and exposes Prometheus metrics. Grafana displays latency, token counts, acceptance, and backend health.
Experiment tooling
Ray runs the sweep configurations. Saved JSON and Parquet results feed a React explorer, while a separate benchmark compares configurations in one process after compilation warm-up.
Deployment tooling
Dockerfiles, a Helm chart, a host-only kind check, and Terraform/GKE configuration are included in the repo. This snapshot has no production deployment or GPU performance results.
02

Speculative Decoding

The draft has to be checked in a way that preserves the target’s output distribution. Otherwise the smaller model changes what we generate.

Call the target’s next-token distribution p, and the distribution that produced the draft q. A proposed token x is accepted with probability min(1, p(x)/q(x)). At the first rejection, we sample a replacement from the normalized max(0, p − q) and discard the rest of the block. If every draft passes, we sample a bonus token from the target.

At temperature zero, we keep a draft if it matches the target’s top choice. For sampling, the acceptance and replacement rule preserves the target distribution, so two runs can produce different text. The rule comes from Leviathan et al., 2023 and Chen et al., 2023.

One step, with three drafts

These are made-up probabilities to show how the rule works.

Proposed suffixdraft 1draft 2draft 3
The drafter predicts three tokens in sequence and keeps the distribution that produced each one. The target then checks the pending token and all three drafts in one call.
PositionTarget p(x)Draft q(x)Uniform uu × q(x)Test
Draft 10.400.300.600.180.18 < 0.40 → keep
Draft 20.350.400.500.200.20 < 0.35 → keep
Draft 30.120.300.700.210.21 ≥ 0.12 → repair

Drafts after a rejection were predicted using a token we have discarded, so they are discarded too. A fully accepted block can emit k + 1 tokens, subject to the output limit.

The cache, and why one token stays pending

The target cache holds all the committed tokens except the newest one. That newest token stays pending, so the next call sends [pending_token] + drafts together. Logit row j predicts draft j, and the last row predicts the bonus. After accepting A drafts, we keep the pending token and accepted prefix in the cache, then the replacement or bonus becomes the next pending token.

Rollback changes the cache’s logical length without recomputing the prefix. Old values can stay in the buffers as long as the attention mask hides them, and the drafter’s cache must match the committed history too. The pending-token setup removed a second target call that used to process the bonus separately. See 1e9f5d3.

Triton verify and alpha kernels have PyTorch references. The alpha evaluator can use the fused kernel on CUDA; the JAX decoder currently verifies drafts in its own loop.

03

The Data

The corpus is split into thinking, text, fenced code, tool calls, and tool results. Duplicate blocks are removed, and blocks are capped at 4,096 tokens. Each whole run stays in one split, which keeps adjacent parts of a session from crossing between training and evaluation.

The first validation and test splits each had three runs, with one run contributing most of their tokens. The newer hold-out adds 14,974 blocks and 4.24 million tokens across 14 later runs. It excludes runs already present and exact (kind, text) duplicates from the older splits.

Where the training tokens come from

Most are thinking and tool output. This is the original training split, counted by tokens.

Source: results/corpus_stats.json. Tool results are things the agent reads, so being able to predict them does not directly save time generating the agent’s own output.

SplitRunsBlocksTokensRole in this edition

Measuring overlap

The alpha evaluation uses 700 held-out blocks and scores 56,990 token positions. Blocks are selected across runs, capped at 256 tokens, and given context from preceding selected blocks within a 1,024-token budget. At each position we give both models the recorded prefix and compare their predictions. This is teacher forcing, where the recorded tokens determine the history.

Alpha measures the overlap between their probability distributions:

α = Σₓ min(p(x), q(x)) = 1 − TV(p, q)

What 0.71 means

An alpha of 0.71 means about 71% of the probability mass overlaps on average across these positions. With correctly sampled proposals at those contexts, it gives expected acceptance. It is different from next-token accuracy against the recorded trace, and does not predict how much of every live draft block will pass.

How much it varies

The saved estimates average 1,000 bootstrap replicates. Each replicate samples whole runs with replacement and pools their token counts, keeping related parts of a session together. Larger runs still contribute more tokens.

Alpha here uses unscaled softmax, or temperature 1. The sweep uses temperatures 0 and 0.7, a different target, and generated continuations. The selected alpha blocks also contain no code row. These results apply to the sampled agent traffic.

04

Held-out Results

LoRA reaches 71.2% overlap with the 1.5B target. N-gram lookup gets 32.28%, and the tiny distilled model gets 1.14%.

Overlap with the target

Qwen2.5-1.5B-Instruct target · same 700 selected blocks · 14 runs · 95% run-bootstrap intervals

DrafterOverlap α95% intervalProposal coverage
N-gram32.28%30.74–33.89%73.9%
LoRA71.20%70.61–71.87%100.0%
LoRA + DPO71.35%70.78–72.01%100.0%
Tiny distilled1.14%1.09–1.20%100.0%

Coverage means how often the drafter has a proposal. For n-grams, a position with no proposal counts as zero overlap, while the model drafters cover almost every valid position. Source: results/alpha/{ngram,lora,dpo,tiny}.json.

DPO adds 0.153 percentage points to the saved estimate. Its interval overlaps with LoRA’s, but separate intervals cannot settle whether the paired difference is statistically clear. The saved results lack that run-level comparison.

Tool calls are easier to predict

Overlap by content type, with a 95% interval under each number.

Content kindRuns / positionsN-gramTinyLoRALoRA + DPO

Darker cells mean more overlap. The models use the same scored positions. Read calls have five supporting runs and Edit calls have four; their intervals are wider. On a phone, choose a drafter to see its column.

Tool calls have more overlap than text or thinking, which is consistent with repeated syntax helping prediction. To separate the effect of training on agent traffic, we would need an unadapted 0.5B baseline and a matched comparison on non-agent text. Those comparisons are missing here.

05

Training the Drafters

StrategyHow it proposesWhy it is interesting
N-gram lookupLook for a recent sequence of one to four tokens in earlier committed context, then propose what followed it.A cheap way to use repetition, with no neural model call and no access to future text.
Tiny distilledTrain a small JAX student with 4 layers and width 256, using the same token vocabulary and its own embeddings.Only 43.8M parameters. For distillation, save the teacher’s top-64 probabilities and the remaining tail mass, which is much smaller than saving every logit.
LoRAAdapt a Qwen2.5-0.5B base model to the agent traffic, with rank 16 and scaling 32 on the attention and MLP projections.Train small matrices that update the base weights. There is FSDP training code, along with a QLoRA path that needs a GPU.
LoRA + DPOUse target-corrected tokens as preferred examples and rejected drafts as the other choice, with TRL DPO and β = 0.1.Try to improve acceptance directly, although the code building those pairs still needs fixing (below).
Training the tiny student

Validation KL over 1,200 training steps. This measures a difference between distributions, where lower is better.

Standard initializationProjected teacher embedding initialization

The standard run went from 3.788 to 2.629, a 30.6% decrease. Initializing with projected teacher embeddings ended at 2.638. These curves show the earlier training runs; held-out overlap is measured separately. Sources: results/distill_curve.json, results/distill_curve_tinit.json.

The student’s validation KL falls by 30.6%, while its held-out overlap is only 1.14%. Its small capacity, training duration, and block-only training context could contribute to that gap. The experiment did not vary them separately.

For LoRA, the sweep merges the saved adapter into the JAX model with W ← W + (scaling/rank) · B · A, updating the attention and MLP weights. We can train in PyTorch and generate in JAX without a separate PyTorch drafter in the decoding loop.

DPO still has a problem to fix

The DPO pair builder chooses d = argmax(q), then applies an acceptance test using the soft probability q[d] and residual p − q. Since the token was chosen greedily, those probabilities do not describe how it was produced. The code also prefills a sequence and feeds its last token again when starting cached generation. These findings come from speculative_agent_workloads/drafters/dpo.py, which the October 6 decoder fix did not change. Training has not been rerun, so the pairs need checking and rebuilding before the DPO increase can support a claim about this method.

06

Generation and Timing

The corrected sweep generates 32 tokens after short held-out prompts, with draft lengths k ∈ {1, 2, 4, 8} and temperatures 0 and 0.7. Its Qwen2.5-0.5B-Instruct target is about the same size as the learned drafter. The grid measures decoder behavior; a larger target would give a better comparison for the cost of a smaller drafter.

Each finished prompt is saved to a sidecar file and folded into Parquet when its cell finishes or restarts. The report reads both formats and removes duplicates by configuration and prompt index.

The corrected sweep, including unfinished cells

0.5B-Instruct target · 32 generated tokens per prompt · 115 prompts in a complete cell

Drafterk = 1k = 2k = 4k = 8
26 complete8 partial8 pending42 configurations, including two target-only baselines

Tokens per step = emitted tokens / target steps. Finished cells use the same prompt set, while partial cells use different subsets. Striped cells are unfinished, and a pending cell has no measurement yet.

Sources: results/sweep.parquet and its 8 saved sidecars. The target-only baselines, 1.5B sweep, and controlled speed benchmark are still pending in this snapshot, so we do not have a measured speedup to report yet.

Once generation diverges from the recorded trace, the next prediction sees a different history. Alpha on recorded prefixes cannot describe all of those later contexts. The accepted/drafted ratio also counts drafts after the first rejection, even though they cannot be committed, so changing k can change that ratio while first-token overlap stays the same.

The identity control uses the target itself as the drafter, where acceptance should be one. At k = 8, a full step emits nine tokens, but the 32-token output limit requires four steps. The average is therefore 32 / 4 = 8 tokens per step.

Acceptance, draft cost, and draft length

A simplified cost model. The controls change assumptions, with no measured speed data behind them.

1.98×

Estimated relative speed at this k

2.77 tokens / target call

Best k in this model: 4

E[tokens / call] = 1 + α + α² + … + αᵏ Ideal relative speed = E[tokens / call] / (1 + k · c)

The model assumes constant, independent acceptance, a proposal at every position, and block verification costing the same as one target decode. It excludes prefill, output limits, EOS, compilation, cache work, and resource contention. Leviathan et al., generated-token and wall-time analysis.

tools/bench_speedup.py warms up the model shapes, alternates configurations over repeated passes, and compares each with target-only decoding on the same prompts. Its result file, results/speedup_bench.json, was absent from this snapshot. Timings from concurrent sweep workers cannot establish the runtime saving.

07

Checking the Output

The tests compare greedy output with the target, check the cache against a fresh prefill, check sampled distributions, and count target calls per step. Newer sampling tests use an identity drafter and check the second-token distribution of a mismatched drafter. A first-token test can miss incorrect state carried into the next step.

The judge rerun uses held-out prompts on the fixed engine, with a 0.5B-Instruct target, DPO drafter, k = 4, temperature 0.7, and 16 output tokens. The local qwen-agent judge receives blind comparisons in both orders, with cached responses and a 150-call cap. These are short-output preference checks, which do not measure completion of coding tasks.

What the judge preferred after the fix

Wins, ties, and losses for the speculative output.

Spec winsTieSpec loses

Source: results/judge.json, generated October 6. There were 150 calls used and 18 invalid responses, and the chart only counts included verdicts. The file’s order-disagreement rate includes invalid orders in its denominator; the counts separately show prompts with both orders valid.

The speculative-versus-target win-rate interval includes 50%, which does not establish equivalence or non-inferiority. Verdicts from both orders judge the same prompt and are correlated, while the saved Wilson intervals treat them as independent. Eight of the 39 prompts with both orders valid received conflicting judgments.

Distribution preservation depends on the acceptance rule and its implementation checks. The judge can help catch an obvious quality change in the short continuations.

What we mean by “lossless”

At temperature zero, output should match the target’s greedy tokens. At positive temperature, its distribution should match target-only sampling within numerical tolerance. Statistical tests check a finite sample. The commit adding the new checks records that they fail on the old implementation; this report uses saved evidence rather than a fresh run of the full test suite.

08

Fixing the Measurements

The commit history records several bugs that changed how the results had to be read.

One token off

The evaluator compared token i with logit row i, which predicts token i + 1. The low overlap came from comparing the wrong positions.

The fix lines up logits with next-token labels and skips the first position when there is no preceding context. Tests compare cross-entropy with Hugging Face’s shifted loss and check repeated versus shuffled text.

The cache can still be wrong

The drafter decoded the last committed token twice, and its position marker failed to advance correctly when every draft passed.

The fix corrected proposal positions and rollback. Tests compare the drafter’s cache and next proposal with a fresh prefill after rejection and full acceptance. The October 6 change then moved to the pending-token setup described earlier.

Sampling the drafts

Drafts were chosen greedily but reported with soft probabilities and without matching temperature. The acceptance test used a distribution that had not produced the token, biasing sampled output toward greedy choices.

Model drafts now come from softmax(logits / T), and the verifier receives that distribution. At temperature zero, the proposal distribution is one-hot at the greedy token. The decoder sets the drafter’s temperature and seed. This change also removed the extra target call.

Suspiciously small error bars

The bootstrap dimensions were swapped: n_runs replicates with n_boot draws each. Drawing so many times from the same few runs made their representation almost even in each replicate, collapsing the intervals around the estimate.

The corrected version makes n_boot replicates, each with n_runs session draws. A regression test uses two runs with very different acceptance rates and requires the interval to span both.

More tokens do not always mean more evidence

The first splits had few independent sessions. Selecting the first N prompts could concentrate a content type in one session, and the old speedup claims used timings from concurrent sweep workers.

The changes add a later hold-out, select prompts round-robin across runs, increase generation from 16 to 32 tokens, and add a 1.5B target option. The separate benchmark alternates configurations and includes target-only decoding.

Knowing when to stop

Serving kept generating after end-of-sequence tokens and returned finish_reason: stop even at the length limit.

It now stops at the tokenizer’s EOS markers, excludes them from completion text, and returns stop or length accordingly. A no-drafter model is also available for comparisons.

09

Future Improvement

QuestionEvidence neededSnapshot status
Does speculation save wall time?Run the fixed engine against target-only decoding on the same prompts, with a controlled timing comparison.Benchmark results still pending.
Does a smaller drafter help a larger target?Run the 1.5B sweep with the 0.5B adapters, then compare the timing.1.5B sweep still pending.
Does DPO improve acceptance?Check and rebuild the preference pairs, compare within runs, and include the unadapted model.A small observed increase; still needs checking.
Do corpus scores predict generation?Compare the fixed engine’s generated histories at the same positions and temperature.The saved trajectories are from before the fixes.
Does the GPU path improve latency?Benchmark the kernels and serving on a GPU, comparing target-only with speculative decoding.The host script is there; no GPU results yet.
Can a new developer reproduce it?Check a fresh clone, explicitly skip missing data or models, and run the full verification checks.An open task lists four test failures from missing data.
10

Data and Code

The results come from revision e874bd1 and the files saved when the original report was built. Downloads include the revision, source hashes, confidence intervals, and sweep counts. They contain no prompts, transcript text, token sequences, or run identifiers. The standalone report includes its charts and data for offline use.

Data snapshot: . This page keeps those saved results even as new experiments finish.

Rebuilding the original data report

# From the Speculative Decoding for Agent Workloads project, with its Python dependencies available:
python tools/build_presentation.py

# Outputs:
# report/index.html   # portable report
# report/data.json    # aggregate evidence, metadata and commit diffs
# report/metrics.csv  # full-precision metrics, intervals and status

# Optional fresh data report, at another path:
python tools/build_presentation.py --publish ../../reports/speculative-agent-workloads-report-fresh.html

The builder reads the saved results, checks the alpha support, and checks whether the pending benchmark files have appeared. These commands rebuild its original presentation; this rewrite is saved separately.

Commands for the measurements

These are the commands that produce the measurements. The report uses saved files, so writing it did not rerun training, generation, or judging.

python -m speculative_agent_workloads.eval.alpha --split heldout --blocks 700 --drafter ngram
python -m speculative_agent_workloads.eval.alpha --split heldout --blocks 700 --drafter hf --draft-model ckpt/lora
python -m speculative_agent_workloads.eval.alpha --split heldout --blocks 700 --drafter hf --draft-model ckpt/dpo
python -m speculative_agent_workloads.eval.alpha --split heldout --blocks 700 --drafter hf --draft-model ckpt/tiny/export
python -m speculative_agent_workloads.sweep --split heldout --num-cpus 12 --cpus-per-cell 4
python -m speculative_agent_workloads.sweep --split heldout --target models/qwen2.5-1.5b-instruct --drafters ngram lora lora_dpo
python tools/bench_speedup.py
python -m speculative_agent_workloads.eval.judge --split heldout
make check
Source files and hashes
Project-relative pathUsed forSHA-256 prefix
results/alpha/ngram.jsonHeld-out distribution overlapae8577962fed
results/alpha/lora.jsonHeld-out distribution overlap7f6295f6f5ac
results/alpha/dpo.jsonHeld-out distribution overlape3edb088d1d4
results/alpha/tiny.jsonHeld-out distribution overlapebff23ea1323
results/corpus_stats.jsonOriginal corpus composition04367bcad7bf
data/corpus/heldout.parquetTemporal hold-out size; aggregate columns only0f6018450ec3
results/sweep.parquetCorrected 0.5B decoding sweep, completed cells1d8b94049b5e
results/sweep.parquet.part-lora-2-0.0.jsonlRecoverable partial sweep cell; read without modifying it09db99171a10
results/sweep.parquet.part-lora-2-0.7.jsonlRecoverable partial sweep cell; read without modifying it7bd28a066b29
results/sweep.parquet.part-lora-4-0.0.jsonlRecoverable partial sweep cell; read without modifying it713d3f1129da
results/sweep.parquet.part-lora-4-0.7.jsonlRecoverable partial sweep cell; read without modifying it1123dcb21ea3
results/sweep.parquet.part-lora-8-0.0.jsonlRecoverable partial sweep cell; read without modifying it67d1af2b1ed2
results/sweep.parquet.part-lora-8-0.7.jsonlRecoverable partial sweep cell; read without modifying it0db87d2ca673
results/sweep.parquet.part-lora_dpo-1-0.0.jsonlRecoverable partial sweep cell; read without modifying it54f4914cebd9
results/sweep.parquet.part-lora_dpo-1-0.7.jsonlRecoverable partial sweep cell; read without modifying itcc0712dd3042
results/judge.jsonCorrected held-out judge rerun5e6f7777e661
results/distill_curve.jsonRecorded distillation training curve90784039de12
results/distill_curve_tinit.jsonRecorded distillation training curve8443bb1b7392
ckpt/lora/adapter_config.jsonImplementation or outstanding work used for this report6439004389d3
ckpt/dpo/adapter_config.jsonImplementation or outstanding work used for this report4cf534ec2e63
speculative_agent_workloads/jaxlm/spec.pyImplementation or outstanding work used for this report1171a5da035e
speculative_agent_workloads/jaxlm/model.pyImplementation or outstanding work used for this report9a4218ae0d7b
speculative_agent_workloads/eval/alpha.pyImplementation or outstanding work used for this report8d4a9e9146a2
speculative_agent_workloads/data/corpus.pyImplementation or outstanding work used for this reporteb1ea87d5969
speculative_agent_workloads/drafters/dpo.pyImplementation or outstanding work used for this report96fbc3e9d5f6
speculative_agent_workloads/drafters/lora.pyImplementation or outstanding work used for this report8f6d2ee01ff8
speculative_agent_workloads/drafters/distill.pyImplementation or outstanding work used for this report4b1716c2da98
speculative_agent_workloads/kernels/verify.pyImplementation or outstanding work used for this report036c1e6e5d46
speculative_agent_workloads/eval/judge.pyImplementation or outstanding work used for this report85fca0709679
speculative_agent_workloads/serve/app.pyImplementation or outstanding work used for this reportd51e2d288b0e
speculative_agent_workloads/sweep.pyImplementation or outstanding work used for this reportff6886f87297
gateway/metrics.goImplementation or outstanding work used for this report2812c9aa9afc
TODO.mdImplementation or outstanding work used for this reportfeed15cd0c06

Source paths use the current package name. Hashes refer to the saved snapshot; the evidence JSON retains its original paths and full hashes. The held-out Parquet file stays local, and only aggregate columns were read to count the corpus.

The changes in code

You can expand each change to see its code and test diff. These preserve the original implementation and paths at that commit, which later changes may have updated.

7a84891 alpha: fix off-by-one logit alignment, known-answer tests, 1.5B default target

2026-09-30 · speculative_agent_workloads/eval/alpha.py, speculative_agent_workloads/tests/test_alpha_known.py

diff --git a/draftlab/eval/alpha.py b/draftlab/eval/alpha.py
index 316693f..89ae749 100644
--- a/draftlab/eval/alpha.py
+++ b/draftlab/eval/alpha.py
@@ -7,8 +7,10 @@ At every position of a block, with context = the run's recent preceding blocks
     greedy_corpus = [argmax q == actual token]  (what results/ngram.json measures)
 "Batched" = one teacher-forced forward per model per block (all positions at once),
-top-k-free. Blocks are selected round-robin across runs; per-kind aggregation uses
-bootstrap CIs over runs (units = runs, so no run leaks). Expected tokens per target
-call for k drafts: (1 - a^(k+1))/(1 - a); speedup divides by 1 + k*c, c = measured
-draft/target forward cost ratio.
+top-k-free. Alignment: logits row i predicts token i+1, so ids[j] is scored by row
+len(ctx)+j-1; with empty ctx there is no row for ids[0] and position 0 is invalid
+(valid[0] = False; no fabricated BOS). Blocks are selected round-robin across runs;
+per-kind aggregation uses bootstrap CIs over runs (units = runs, so no run leaks).
+Expected tokens per target call for k drafts: (1 - a^(k+1))/(1 - a); speedup divides
+by 1 + k*c, c = measured draft/target forward cost ratio.
 
 CLI: python -m draftlab.eval.alpha [--drafter ngram|jax|hf] [--draft-model PATH]
@@ -29,5 +31,5 @@ import numpy as np
 ROOT = Path(__file__).resolve().parents[2]
 CORPUS_DIR = ROOT / "data" / "corpus"
-TARGET_DIR = ROOT / "models" / "qwen2.5-0.5b"
+TARGET_DIR = ROOT / "models" / "qwen2.5-1.5b-instruct"  # target T (TODO.md model roles)
 RESULTS_DIR = ROOT / "results" / "alpha"
 
@@ -118,6 +120,16 @@ class JaxModel:
         logits, _ = self.M.prefill(self.cfg, self.params, full, self.max_len)
         n = len(ids)
-        rows = np.asarray(logits)[len(ctx) : len(ctx) + n]  # slice before softmax
-        return _softmax(rows), np.ones(n, dtype=bool)
+        # Row i predicts token i+1: ids[j] is scored by row len(ctx)+j-1 (slice
+        # before softmax/transfer). Empty ctx: no row predicts ids[0] -> dummy
+        # row 0, position 0 invalid (we do not fabricate a BOS).
+        L = np.asarray(logits)
+        valid = np.ones(n, dtype=bool)
+        if ctx:
+            rows = L[len(ctx) - 1 : len(ctx) + n - 1]
+        else:
+            valid[0] = False
+            rows = L[: n - 1]
+            rows = np.vstack([L[:1], rows]) if n > 1 else L[:1]
+        return _softmax(rows), valid
 
 
@@ -146,6 +158,15 @@ class HFDrafter:
         n = len(ids)
         with self.torch.no_grad():
-            logits = self.model(input_ids=self.torch.tensor([ctx + ids])).logits[0, -n:]
-        return _softmax(logits.float().numpy()), np.ones(n, dtype=bool)
+            logits = self.model(input_ids=self.torch.tensor([ctx + ids])).logits[0]
+        # Same alignment as JaxModel: ids[j] is scored by row len(ctx)+j-1;
+        # empty ctx -> dummy row 0, position 0 invalid.
+        valid = np.ones(n, dtype=bool)
+        if ctx:
+            rows = logits[len(ctx) - 1 : len(ctx) + n - 1]
+        else:
+            valid[0] = False
+            rows = logits[: n - 1]
+            rows = self.torch.cat([logits[:1], rows]) if n > 1 else logits[:1]
+        return _softmax(rows.float().numpy()), valid
 
 
diff --git a/draftlab/tests/test_alpha_known.py b/draftlab/tests/test_alpha_known.py
new file mode 100644
index 0000000..e62812a
--- /dev/null
+++ b/draftlab/tests/test_alpha_known.py
@@ -0,0 +1,97 @@
+"""Known answers for the alpha metric (CLAUDE.md "Known answers"), on the real
+0.5B checkpoint. Skipped only if models/ has not been fetched."""
+
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from draftlab.eval.alpha import JaxModel, NGramDrafter, alpha_and_greedy, load_blocks
+
+MODEL = Path(__file__).resolve().parents[2] / "models" / "qwen2.5-0.5b"
+HAS_MODEL = (MODEL / "config.json").exists()
+needs_model = pytest.mark.skipif(not HAS_MODEL, reason="models/ not fetched")
+
+
+@pytest.fixture(scope="module")
+def tok():
+    from transformers import AutoTokenizer
+
+    return AutoTokenizer.from_pretrained(MODEL)
+
+
+@pytest.fixture(scope="module")
+def target():
+    return JaxModel.from_path(MODEL, max_len=512)
+
+
+@needs_model
+def test_per_position_ce_matches_hf_labels_loss(target, tok):
+    """JaxModel rows must be aligned (row i predicts token i+1): per-position CE
+    from dists == HF per-token NLL, and their mean == model(ids, labels=ids).loss."""
+    import torch
+    from torch.nn import functional as F
+    from transformers import AutoModelForCausalLM
+
+    hf = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float32).eval()
+    for b in load_blocks("val", max_tokens=256, n_blocks=2):
+        ids = torch.tensor([tok.encode(b["text"], add_special_tokens=False)])
+        with torch.no_grad():
+            out = hf(ids, labels=ids)
+            nll = F.cross_entropy(hf(ids).logits[0, :-1], ids[0][1:], reduction="none").numpy()
+        p, valid = target.dists([], ids[0].tolist())
+        assert not valid[0] and valid[1:].all()
+        arr = ids[0].numpy()[1:]  # ce_j = -log p[j, ids[j]] for j >= 1
+        ce = -np.log(np.maximum(p.astype(np.float64)[np.arange(1, len(arr) + 1), arr], 1e-30))
+        assert np.max(np.abs(ce - nll)) < 1e-3, np.max(np.abs(ce - nll))
+        assert abs(float(ce.mean()) - out.loss.item()) < 1e-3
+
+
+@needs_model
+def test_target_top1_on_val_in_sanity_range(target, tok):
+    """Qwen2.5-0.5B top-1 next-token accuracy on corpus val blocks is 40-70%
+    (57.5% measured on 372 tok); under 20% means an alignment bug."""
+    hits = tot = 0
+    for b in load_blocks("val", max_tokens=256, n_blocks=40):
+        ids = np.asarray(tok.encode(b["text"], add_special_tokens=False))
+        p, valid = target.dists([], ids.tolist())
+        ok = valid & (p.argmax(axis=-1) == ids)
+        hits += int(ok.sum())
+        tot += int(valid.sum())
+    acc = hits / tot
+    assert 0.40 <= acc <= 0.70, f"top-1 {acc:.3f} outside sanity range 0.40-0.70"
+
+
+@needs_model
+def test_uniform_q_alpha_exact(target, tok):
+    """q uniform -> alpha = sum_x min(p, 1/V) = 1 - TV(p, u) = 1 - 0.5*sum|p - 1/V|
+    (checked via the independent total-variation identity, not the same min-sum)."""
+    ids = np.asarray(tok.encode(load_blocks("val", 256, 1)[0]["text"], add_special_tokens=False))
+    p, valid = target.dists([], ids.tolist())
+    v = p.shape[1]
+    a, _ = alpha_and_greedy(p, np.full_like(p, 1.0 / v))
+    expected = 1.0 - 0.5 * np.abs(p.astype(np.float64) - 1.0 / v).sum(axis=-1)
+    assert np.max(np.abs(a - expected)) < 1e-6
+    assert (a >= 1.0 / v - 1e-6).all() and (a <= 1.0).all()  # TV bounds
+
+
+def test_ngram_repeated_block_planted_effect():
+    """Planted effect: a block repeated verbatim after itself -> ngram acceptance
+    on the copy ~= 1 (all but the first position, which has no prior context)."""
+    x = list(range(1, 65))  # no internal n-gram repeats, so every proposal is exact
+    d = NGramDrafter(vocab_size=1000)
+    q, valid = d.dists(x, x)
+    match = valid & (q.argmax(axis=-1) == np.asarray(x))
+    assert match.sum() == len(x) - 1 and match[1:].all()
+
+
+def test_ngram_shuffled_block_null_control():
+    """Null control: the same tokens shuffled within the block -> ngram acceptance
+    collapses (proposals were learned on the original order)."""
+    x = list(range(1, 65))
+    rng = np.random.default_rng(0)
+    y = [x[i] for i in rng.permutation(len(x))]
+    d = NGramDrafter(vocab_size=1000)
+    q, valid = d.dists(x, y)
+    acc = float((valid & (q.argmax(axis=-1) == np.asarray(y))).mean())
+    assert acc < 0.5, f"shuffled acceptance {acc:.3f} did not collapse"
ae371e3 spec: fix ModelDrafter propose double-decode and full-accept rollback; regression tests

2026-09-30 · speculative_agent_workloads/jaxlm/spec.py, speculative_agent_workloads/tests/test_spec.py

diff --git a/draftlab/jaxlm/spec.py b/draftlab/jaxlm/spec.py
index 29a8fa1..255289a 100644
--- a/draftlab/jaxlm/spec.py
+++ b/draftlab/jaxlm/spec.py
@@ -143,5 +143,11 @@ class StepStats:
 
 class SpecDecoder:
-    """Speculative sampling over one jitted target model."""
+    """Speculative sampling over one jitted target model.
+
+    temperature > 0 scales the TARGET sampling distribution (softmax(lg/T));
+    draft probs stay unscaled. Accept/reject + residual max(0, p - q) is
+    lossless for any draft q, so output still follows exactly the target's
+    T-scaled distribution.
+    """
 
     def __init__(
@@ -167,4 +173,5 @@ class SpecDecoder:
         cfg, params, drafter = self.cfg, self.params, self.drafter
         T0 = self.temperature == 0.0
+        T = self.temperature if not T0 else 1.0
         rng = self.rng
         ctx = list(prompt)
@@ -187,5 +194,5 @@ class SpecDecoder:
                     tok = int(np.argmax(np.asarray(prev_lg)))
                 else:
-                    tok = _categorical(step_rng, jax.nn.softmax(prev_lg))
+                    tok = _categorical(step_rng, jax.nn.softmax(prev_lg / T))
                 lg, cache = M.decode(cfg, params, cache, [tok])
                 prev_lg = lg[-1]
@@ -203,5 +210,5 @@ class SpecDecoder:
                     bonus = int(np.argmax(np.asarray(Pd[A] if A < m else scored[m - 1])))
                 else:
-                    P = [np.asarray(jax.nn.softmax(row), np.float32) for row in Pd]
+                    P = [np.asarray(jax.nn.softmax(row / T), np.float32) for row in Pd]
                     A = 0
                     for i, q in enumerate(dprobs):
@@ -213,5 +220,5 @@ class SpecDecoder:
                             break
                     if A == m:
-                        bonus = _categorical(step_rng, jax.nn.softmax(scored[m - 1]))
+                        bonus = _categorical(step_rng, jax.nn.softmax(scored[m - 1] / T))
                     else:
                         resid = np.maximum(0.0, P[A] - dprobs[A])
diff --git a/draftlab/tests/test_spec.py b/draftlab/tests/test_spec.py
index ce6d277..b8ad5f0 100644
--- a/draftlab/tests/test_spec.py
+++ b/draftlab/tests/test_spec.py
@@ -78,4 +78,30 @@ def test_lossless_chi_square():
 
 
+def test_temperature_scaling_chi_square():
+    """T=0.7: first-token distribution == softmax(logits/0.7), chi2 p > 0.01,
+    and distinguishable from the unscaled softmax (so the test fails if the
+    temperature is ignored)."""
+    from scipy.special import logsumexp
+
+    cfg, params = _tiny16(1)
+    prompt = [i % 7 for i in range(32)]
+    lg, _ = M.prefill(cfg, params, prompt, len(prompt) + 8)
+    lg = np.asarray(lg[-1], np.float64)
+    pT = np.exp(lg / 0.7 - logsumexp(lg / 0.7))
+    p1 = np.exp(lg - logsumexp(lg))
+    assert float(np.abs(pT - p1).sum()) > 0.05, "guard: T=0.7 must change the distribution"
+    N = 8000
+    obs = np.zeros(cfg.vocab_size, np.int64)
+    for i in range(N):
+        dec = SpecDecoder(cfg, params, NGramDrafter(cfg.vocab_size, max_n=4),
+                          max_len=len(prompt) + 8, k=2, temperature=0.7, seed=i)
+        out, _ = dec.generate(prompt, 1)
+        obs[out[0]] += 1
+    exp = N * pT
+    chi2 = float(((obs - exp) ** 2 / exp).sum())
+    pval = float(stats.chi2.sf(chi2, df=cfg.vocab_size - 1))
+    assert pval > 0.01, f"chi2={chi2:.1f}, p={pval:.4g} (not sampling softmax(lg/0.7))"
+
+
 @pytest.mark.parametrize("kind", ["ngram", "model"])
 def test_rollback_cache_equals_fresh_prefill(kind):
1e9f5d3 spec: lossless T>0 model drafting, one target forward per step

2026-10-06 · speculative_agent_workloads/jaxlm/spec.py, speculative_agent_workloads/tests/test_spec.py

diff --git a/draftlab/jaxlm/spec.py b/draftlab/jaxlm/spec.py
index 255289a..a24a63e 100644
--- a/draftlab/jaxlm/spec.py
+++ b/draftlab/jaxlm/spec.py
@@ -4,9 +4,8 @@ Drafter protocol: ``propose(ctx, k) -> (tokens, probs)`` — up to k draft
 tokens plus a (V,) float32 draft distribution for each; ``rollback(n_kept)``
 undoes draft-side state (KV cache) added for the drafted suffix beyond the
-n_kept drafts that were committed. The engine scores all m drafted tokens plus
-the bonus position with one target call; accepts while u < p/q, then either
-resamples from max(0, p - q) at the first rejection or draws the bonus from p.
-Target-cache rollback is truncating the positional cache's `len`; a
-fully-accepted step decodes the bonus in.
+n_kept drafts that were committed. The engine scores the pending token plus all m drafted
+tokens with one target call; accepts while u < p/q, then either resamples
+from max(0, p - q) at the first rejection or draws the bonus from p.
+Target-cache rollback is truncating the positional cache's `len`.
 """
 
@@ -84,36 +83,65 @@ class NGramDrafter:
 
 
-class ModelDrafter:
-    """A second JAX Qwen2 proposes its own greedy continuation.
+class NullDrafter:
+    """No drafting: plain autoregressive target decoding."""
+
+    def propose(self, ctx: list[int], k: int) -> tuple[list[int], list[np.ndarray]]:
+        return [], []
 
-    Keeps its own KV cache, extended from committed tokens; a partial
-    acceptance restores the pre-draft snapshot so the committed suffix is
-    re-decoded at the next propose.
+    def rollback(self, n_kept: int) -> None:
+        pass
+
+
+class ModelDrafter:
+    """A second JAX Qwen2 proposes its own continuation.
+
+    T=0: argmax drafts with one-hot q. T>0: drafts sampled from
+    softmax(logits/T) and q is exactly that distribution, so accept/reject is
+    lossless (SpecDecoder sets T and the seed via `configure`). Keeps its own
+    KV cache; rollback truncates it to the committed drafts (no re-decode).
+    The last draft is never decoded here: on full acceptance the next propose
+    decodes it together with the bonus token.
     """
 
-    def __init__(self, dcfg: M.Qwen2Cfg, dparams: dict, max_len: int) -> None:
+    def __init__(self, dcfg: M.Qwen2Cfg, dparams: dict, max_len: int,
+                 temperature: float = 0.0, seed: int = 0) -> None:
         self.dcfg = dcfg
         self.dparams = dparams
         self.cache = M.empty_cache(dcfg, max_len)
         self._last = 0
-        self._undo: tuple[dict, int] | None = None
+        self._undo: tuple[int, int] | None = None
+        self.configure(temperature, seed)
+
+    def configure(self, temperature: float, seed: int) -> None:
+        self.temperature = temperature
+        self._rng = np.random.default_rng(seed)
+
+    def _pick(self, row) -> tuple[int, np.ndarray]:
+        if self.temperature == 0.0:
+            t = int(np.argmax(np.asarray(row)))
+            q = np.zeros(row.shape[-1], np.float32)
+            q[t] = 1.0
+            return t, q
+        q = np.asarray(jax.nn.softmax(row.astype(jnp.float32) / self.temperature), np.float64)
+        q /= q.sum()
+        t = int(self._rng.choice(q.size, p=q))
+        return t, q.astype(np.float32)
 
     def propose(self, ctx: list[int], k: int) -> tuple[list[int], list[np.ndarray]]:
-        # ctx grows by >=1 committed token between proposes (first call: prompt),
-        # so extending the cache to len(ctx) gives the dist over the FIRST draft
-        # token on its last row; each draft then decodes the previous one.
+        # extending the cache to len(ctx) gives the dist over the FIRST draft on
+        # its last row; drafts 1..k-1 each decode the previous draft
         if len(ctx) <= self._last:
             raise RuntimeError("propose called without new committed tokens")
         lg, self.cache = M.decode(self.dcfg, self.dparams, self.cache, ctx[self._last :])
         self._last = len(ctx)
-        self._undo = (self.cache, self._last, k)
         toks: list[int] = []
         probs: list[np.ndarray] = []
-        for _ in range(k):
-            p = np.asarray(jax.nn.softmax(lg[-1]), np.float32)
-            t = int(np.argmax(p))
+        for j in range(k):
+            t, q = self._pick(lg[-1])
             toks.append(t)
-            probs.append(p)
-            lg, self.cache = M.decode(self.dcfg, self.dparams, self.cache, [t])
+            probs.append(q)
+            if j < k - 1:
+                lg, self.cache = M.decode(self.dcfg, self.dparams, self.cache, [t])
+        self._undo = (self._last, len(toks))
         return toks, probs
 
@@ -121,10 +149,9 @@ class ModelDrafter:
         if self._undo is None:
             return
-        cache, last, m = self._undo
+        last, m = self._undo
         self._undo = None
-        if n_kept == m:  # all drafts committed; the cache already holds them
-            self._last = last + m  # only the bonus is new; extend from after the drafts
-            return
-        self.cache, self._last = cache, last
+        keep = min(n_kept, m - 1)  # cached drafts that were committed
+        self._last = last + keep
+        self.cache = {"k": self.cache["k"], "v": self.cache["v"], "len": jnp.int32(self._last)}
 
 
@@ -145,8 +172,9 @@ class SpecDecoder:
     """Speculative sampling over one jitted target model.
 
-    temperature > 0 scales the TARGET sampling distribution (softmax(lg/T));
-    draft probs stay unscaled. Accept/reject + residual max(0, p - q) is
-    lossless for any draft q, so output still follows exactly the target's
-    T-scaled distribution.
+    temperature > 0 samples the target's softmax(lg/T); the drafter is
+    configured to draft at the same T and report its true q, so accept/reject
+    + residual max(0, p - q) is lossless. One target forward per step: the
+    cache holds every committed token but the newest (`pend`), and each step
+    decodes [pend] + drafts, so row j scores draft j and row m is the bonus.
     """
 
@@ -168,4 +196,6 @@ class SpecDecoder:
         self.temperature = temperature
         self.rng = jax.random.PRNGKey(seed)
+        if hasattr(drafter, "configure"):
+            drafter.configure(temperature, seed + 1)
 
     def _steps(self, prompt: list[int], n_tokens: int):
@@ -176,8 +206,7 @@ class SpecDecoder:
         rng = self.rng
         ctx = list(prompt)
-        pre_lg, cache = M.prefill(cfg, params, ctx, self.max_len)
-        prev_lg = pre_lg[-1]  # dist over position L (first generated token)
+        _, cache = M.prefill(cfg, params, ctx, self.max_len)
+        cache = {"k": cache["k"], "v": cache["v"], "len": jnp.int32(len(ctx) - 1)}
         out: list[int] = []
-        stats: list[StepStats] = []
         while len(out) < n_tokens:
             rem = n_tokens - len(out)
@@ -190,51 +219,30 @@ class SpecDecoder:
                 dtoks, dprobs = drafter.propose(ctx, k)
             m = len(dtoks)
-            if m == 0:  # no draft: draw from the held last row, then one decode
-                if T0:
-                    tok = int(np.argmax(np.asarray(prev_lg)))
-                else:
-                    tok = _categorical(step_rng, jax.nn.softmax(prev_lg / T))
-                lg, cache = M.decode(cfg, params, cache, [tok])
-                prev_lg = lg[-1]
-                emitted, A = [tok], 0
+            rows, cache2 = M.decode(cfg, params, cache, [ctx[-1]] + dtoks)
+            if T0:
+                A = 0
+                while A < m and dtoks[A] == int(np.argmax(np.asarray(rows[A]))):
+                    A += 1
+                bonus = int(np.argmax(np.asarray(rows[A])))
             else:
-                # one target call over the m drafts: row r is the dist over the
-                # token at position L+r+1, i.e. draft r+1... so draft j's dist is
-                # prev (j=0) or row j-1, and row m-1 is the bonus dist
-                scored, cache2 = M.decode(cfg, params, cache, dtoks)
-                Pd = [prev_lg if j == 0 else scored[j - 1] for j in range(m)]
-                if T0:
-                    A = 0
-                    while A < m and dtoks[A] == int(np.argmax(np.asarray(Pd[A]))):
+                A = 0
+                for i, q in enumerate(dprobs):
+                    p_i = np.asarray(jax.nn.softmax(rows[i] / T), np.float32)
+                    u = float(jax.random.uniform(step_rng))
+                    step_rng = jax.random.fold_in(step_rng, i + 1)
+                    if u * q[dtoks[i]] < p_i[dtoks[i]]:
                         A += 1
-                    bonus = int(np.argmax(np.asarray(Pd[A] if A < m else scored[m - 1])))
-                else:
-                    P = [np.asarray(jax.nn.softmax(row / T), np.float32) for row in Pd]
-                    A = 0
-                    for i, q in enumerate(dprobs):
-                        u = float(jax.random.uniform(step_rng))
-                        step_rng = jax.random.fold_in(step_rng, i + 1)
-                        if u * q[dtoks[i]] < P[i][dtoks[i]]:
-                            A += 1
-                        else:
-                            break
-                    if A == m:
-                        bonus = _categorical(step_rng, jax.nn.softmax(scored[m - 1] / T))
                     else:
-                        resid = np.maximum(0.0, P[A] - dprobs[A])
-                        bonus = _categorical(step_rng, resid / resid.sum())
-                emitted = dtoks[:A] + [bonus]
-                E = len(emitted) if A < m else min(rem, m + 1)
-                emitted = emitted[:E]
-                # Roll back the target cache to the A committed drafts, then
-                # commit the bonus (its key/value only exists after this decode;
-                # it was not part of the scoring call).
-                trunc = {"k": cache2["k"], "v": cache2["v"], "len": jnp.int32(len(ctx) + A)}
-                prev2, cache_b = M.decode(cfg, params, trunc, [bonus])
-                if A == m and rem >= m + 1:
-                    cache, prev_lg = cache_b, prev2[-1]
-                else:
-                    cache = {"k": cache_b["k"], "v": cache_b["v"], "len": jnp.int32(len(ctx) + E)}
-                    prev_lg = prev2[-1] if len(out) + E < n_tokens else None
+                        resid = np.maximum(0.0, p_i - q)
+                        break
+                if A == m:
+                    bonus = _categorical(step_rng, jax.nn.softmax(rows[m] / T))
+                elif resid.sum() > 0:
+                    bonus = _categorical(step_rng, resid / resid.sum())
+                else:  # p == q on the draft's support: residual is p itself
+                    bonus = _categorical(step_rng, p_i)
+            emitted = (dtoks[:A] + [bonus])[:rem]
+            # keep [pend] + the A accepted drafts; the bonus becomes the new pend
+            cache = {"k": cache2["k"], "v": cache2["v"], "len": jnp.int32(len(ctx) + A)}
             drafter.rollback(A)
             ctx.extend(emitted)
@@ -242,9 +250,8 @@ class SpecDecoder:
             dt_ms = (time.perf_counter() - t0) * 1e3
             st = StepStats(m, A, [x < A for x in range(m)], dt_ms)
-            stats.append(st)
             yield emitted, st
         self.rng = rng
         self.last_ctx = ctx  # prompt + generated
-        self.last_cache = cache
+        self.last_cache = cache  # holds last_ctx[:-1] (the newest token is pending)
 
     def generate(self, prompt: list[int], n_tokens: int) -> tuple[list[int], list[StepStats]]:
diff --git a/draftlab/tests/test_spec.py b/draftlab/tests/test_spec.py
index b8ad5f0..f9f510b 100644
--- a/draftlab/tests/test_spec.py
+++ b/draftlab/tests/test_spec.py
@@ -118,6 +118,6 @@ def test_rollback_cache_equals_fresh_prefill(kind):
     _, ref_cache = M.prefill(cfg, params, full, max_len)
     c = dec.last_cache
-    assert int(c["len"]) == len(full)
-    n = len(full)
+    n = len(full) - 1  # the newest token is pending (decoded by the next step)
+    assert int(c["len"]) == n
     for key in ("k", "v"):
         diff = float(abs(np.asarray(c[key])[:, :, :n] - np.asarray(ref_cache[key])[:, :, :n]).max())
@@ -167,11 +167,11 @@ def test_model_drafter_full_accept_steps_keep_cache_in_sync():
     assert out == _greedy(cfg, params, prompt, 20)
     assert all(s.drafted == 1 and s.accepted == 1 for s in st)  # strict, no rejections
-    # The drafter ends one token behind the final bonus: its cache must equal a
-    # fresh prefill of that prefix (a desynced rollback re-decodes already
-    # cached drafts and the KV diverges from the first duplicated slot on).
+    # The drafter's cache must equal a fresh prefill of the committed prefix it
+    # claims to hold (a desynced rollback re-decodes cached drafts and the KV
+    # diverges from the first duplicated slot on).
     full = prompt + out
-    n = len(full) - 1
     d = dec.drafter
-    assert int(d.cache["len"]) == n and d._last == n
+    n = d._last
+    assert int(d.cache["len"]) == n and 0 < n < len(full)
     _, ref = M.prefill(d.dcfg, d.dparams, full[:n], 84)
     for key in ("k", "v"):
@@ -198,2 +198,78 @@ def test_model_drafter_first_proposal_position():
                     jax.numpy.asarray(prompt + [toks[0]], jax.numpy.int32))
     assert toks[1] == int(np.argmax(np.asarray(lg2[-1])))
+
+
+def _probs(cfg, params, ids, T=1.0):
+    lg, _ = M.step(cfg, params, M.empty_cache(cfg, len(ids) + 1),
+                   jax.numpy.asarray(ids, jax.numpy.int32))
+    return np.asarray(jax.nn.softmax(lg[-1].astype(np.float64) / T), np.float64)
+
+
+def test_model_drafter_lossless_second_token_chi_square():
+    """T=0.8, model drafter != target, k=3: the marginal of the SECOND emitted
+    token equals sum_x p(x) p(y|x) exactly (covers draft rows, rejection +
+    residual, and the pending-token cache layout across steps)."""
+    cfg, params = _tiny16(1)
+    dcfg, dparams = _tiny16(42)
+    T = 0.8
+    prompt = [i % 7 for i in range(20)]
+    p1 = _probs(cfg, params, prompt, T)
+    exp_p = sum(p1[x] * _probs(cfg, params, prompt + [x], T) for x in range(cfg.vocab_size))
+    N = 6000
+    obs = np.zeros(cfg.vocab_size, np.int64)
+    acc = drafted = 0
+    for i in range(N):
+        dec = SpecDecoder(cfg, params, ModelDrafter(dcfg, dparams, len(prompt) + 8),
+                          max_len=len(prompt) + 8, k=3, temperature=T, seed=i)
+        out, st = dec.generate(prompt, 2)
+        obs[out[1]] += 1
+        acc += sum(s.accepted for s in st)
+        drafted += sum(s.drafted for s in st)
+    assert 0 < acc < drafted, "need both accepts and rejects"
+    exp = N * exp_p
+    chi2 = float(((obs - exp) ** 2 / exp).sum())
+    pval = float(stats.chi2.sf(chi2, df=cfg.vocab_size - 1))
+    assert pval > 0.01, f"chi2={chi2:.1f}, p={pval:.4g} (second-token marginal shifted)"
+
+
+def test_identity_drafter_at_temperature_chi_square():
+    """Regression: argmax drafts + soft q made drafter==target accept the argmax
+    at T<1 every time (output collapsed to greedy). The first token must follow
+    softmax(lg/0.7) when the drafter is the target itself."""
+    cfg, params = _tiny16(1)
+    prompt = [i % 7 for i in range(32)]
+    pT = _probs(cfg, params, prompt, 0.7)
+    N = 4000
+    obs = np.zeros(cfg.vocab_size, np.int64)
+    for i in range(N):
+        dec = SpecDecoder(cfg, params, ModelDrafter(cfg, params, len(prompt) + 8),
+                          max_len=len(prompt) + 8, k=2, temperature=0.7, seed=i)
+        out, st = dec.generate(prompt, 1)
+        obs[out[0]] += 1
+    exp = N * pT
+    chi2 = float(((obs - exp) ** 2 / exp).sum())
+    pval = float(stats.chi2.sf(chi2, df=cfg.vocab_size - 1))
+    assert pval > 0.01, f"chi2={chi2:.1f}, p={pval:.4g} (identity drafter not sampling at T)"
+
+
+@pytest.mark.parametrize("kind", ["ngram", "model", "none"])
+def test_one_target_forward_per_step(kind, monkeypatch):
+    from draftlab.jaxlm import spec as S
+
+    cfg, params = _tiny16(2)
+    calls = []
+    real = M.decode
+
+    def counting(c, p, cache, ids):
+        if p is params:
+            calls.append(len(ids))
+        return real(c, p, cache, ids)
+
+    monkeypatch.setattr(S.M, "decode", counting)
+    prompt = [i % 7 for i in range(30)]
+    d = S.NullDrafter() if kind == "none" else _make_drafter(kind, cfg, 80)
+    dec = SpecDecoder(cfg, params, d, max_len=80, k=3, temperature=1.0, seed=1)
+    out, st = dec.generate(prompt, 25)
+    assert len(out) == 25
+    assert len(calls) == len(st)
+    assert [c - 1 for c in calls] == [s.drafted for s in st]
0158d68 alpha: fix run bootstrap (CIs were ~100x too narrow)

2026-10-06 · speculative_agent_workloads/eval/alpha.py, speculative_agent_workloads/tests/test_alpha.py

diff --git a/draftlab/eval/alpha.py b/draftlab/eval/alpha.py
index b0c260f..139486b 100644
--- a/draftlab/eval/alpha.py
+++ b/draftlab/eval/alpha.py
@@ -271,5 +271,6 @@ def _bootstrap(
         sums_v = np.array([np.concatenate([r[3] for r in by_run[rn]]).sum() for rn in runs])
         cnts = np.array([sum(r[0].size for r in by_run[rn]) for rn in runs])
-        w = rng.multinomial(n_boot, np.full(len(runs), 1.0 / len(runs)), size=len(runs))
+        # (n_boot, n_runs): each replicate draws len(runs) runs with replacement
+        w = rng.multinomial(len(runs), np.full(len(runs), 1.0 / len(runs)), size=n_boot)
         # per-run rates over the drafter's valid (proposed) positions: this is
         # what the sweep's accept = n_acc/n_draft measures (spec.py only scores
@@ -282,6 +283,6 @@ def _bootstrap(
             w @ sums_gc / (w @ cnts),
             w @ sums_v / (w @ cnts),
-            w @ gv / n_boot,
-            w @ av / n_boot,
+            w @ gv / len(runs),
+            w @ av / len(runs),
             int(cnts.sum()),
             len(runs),
diff --git a/draftlab/tests/test_alpha.py b/draftlab/tests/test_alpha.py
index 5d9f914..fb475a6 100644
--- a/draftlab/tests/test_alpha.py
+++ b/draftlab/tests/test_alpha.py
@@ -99,2 +99,21 @@ def test_identical_models_alpha_one():
     assert valid2.all()
     assert a2.min() >= 1.0 - 1e-6 and g2.all()
+
+
+def test_bootstrap_resamples_runs():
+    """Regression: weights were (n_runs, n_runs) with n_boot draws per row, so
+    every 'replicate' was ~equal weights and CIs collapsed to the point
+    estimate. Two runs at alpha 0.2 / 0.8: n_boot replicates, CI spans both."""
+    from draftlab.eval.alpha import _bootstrap
+
+    def rec(run, a):
+        x = np.full(100, a)
+        return (run, "text", x, x > 0.5, x > 0.5, np.ones(100, bool))
+
+    out = _bootstrap([rec("r1", 0.2), rec("r2", 0.8)], n_boot=2000, seed=0)
+    b_alpha = out["overall"][0]
+    assert b_alpha.shape == (2000,)
+    lo, hi = np.percentile(b_alpha, [2.5, 97.5])
+    assert lo == pytest.approx(0.2) and hi == pytest.approx(0.8)
+    assert out["overall"][5].shape == (2000,)  # alpha over valid positions, per replicate
+    assert np.mean(out["overall"][5]) == pytest.approx(0.5, abs=0.02)
5f98414 eval: temporal hold-out split, head-to-head speed bench, 1.5B sweep

2026-10-06 · speculative_agent_workloads/data/corpus.py, tools/bench_speedup.py

diff --git a/draftlab/data/corpus.py b/draftlab/data/corpus.py
index 600aeb7..7438c29 100644
--- a/draftlab/data/corpus.py
+++ b/draftlab/data/corpus.py
@@ -193,4 +193,46 @@ def build(raw_path: Path = RAW_PATH, out_dir: Path = OUT_DIR, tok=None) -> dict[
 
 
+HELDOUT_RAW = ROOT / "data/raw/heldout_turns.jsonl"
+
+
+def build_heldout(raw_path: Path = HELDOUT_RAW, out_dir: Path = OUT_DIR, tok=None) -> int:
+    """Temporal hold-out: runs exported after the corpus (tools/export_traces.py
+    --since) -> data/corpus/heldout.parquet. Drops runs already in a split and
+    blocks whose (kind, text) occurs in any split, so no drafter saw them."""
+    import pyarrow as pa
+    import pyarrow.parquet as pq
+
+    if tok is None:
+        from transformers import AutoTokenizer
+
+        tok = AutoTokenizer.from_pretrained(TOKENIZER_DIR)
+    old_runs: set[str] = set()
+    seen: set[tuple[str, str]] = set()
+    for name in ("train", "val", "test"):
+        t = pq.read_table(out_dir / f"{name}.parquet", columns=["run", "kind", "text"]).to_pydict()
+        old_runs.update(t["run"])
+        seen.update(zip(t["kind"], t["text"], strict=True))
+    rs = []
+    for r in expand(load_raw(raw_path)):
+        if r["run"] in old_runs:
+            continue
+        ids = tok.encode(r["text"])
+        if len(ids) < MIN_TOKENS:
+            continue
+        text = tok.decode(ids[:MAX_TOKENS]) if len(ids) > MAX_TOKENS else r["text"]
+        key = (r["kind"], text)
+        if key in seen:
+            continue
+        seen.add(key)
+        rs.append({**r, "text": text, "n_tokens": min(len(ids), MAX_TOKENS)})
+    if not rs:
+        raise RuntimeError(f"no new blocks in {raw_path}")
+    cols = ["run", "project", "seq", "role", "kind", "tool", "text", "n_tokens"]
+    pq.write_table(pa.table({c: [r[c] for r in rs] for c in cols}), out_dir / "heldout.parquet")
+    print(f"heldout: {len(rs)} blocks, {len({r['run'] for r in rs})} runs "
+          f"-> {out_dir / 'heldout.parquet'}")
+    return len(rs)
+
+
 STATS_SQL = Path(__file__).resolve().parent / "stats.sql"
 RESULTS_DIR = ROOT / "results"
diff --git a/tools/bench_speedup.py b/tools/bench_speedup.py
new file mode 100644
index 0000000..d8176db
--- /dev/null
+++ b/tools/bench_speedup.py
@@ -0,0 +1,113 @@
+"""Head-to-head CPU speedup: each config vs `none` on the same prompts, one process,
+nothing else running. A warm-up pass compiles every shape; then `--reps` passes
+run the configs interleaved (none, A, B, ..., none, A, B, ...), so drift hits all
+configs alike. Speedup per rep = none wall / config wall over the same prompts.
+
+    python tools/bench_speedup.py            # -> results/speedup_bench.json
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from draftlab.jaxlm import model as M  # noqa: E402
+from draftlab.jaxlm import spec as S  # noqa: E402
+from draftlab.sweep import (  # noqa: E402
+    CKPT,
+    LORA_BASE,
+    MODELS,
+    RESULTS,
+    load_prompts,
+    merge_adapter,
+)
+
+# (target, drafter, k, temperature)
+CONFIGS = [
+    ("qwen2.5-0.5b-instruct", "ngram", 8, 0.0),
+    ("qwen2.5-1.5b-instruct", "ngram", 8, 0.0),
+    ("qwen2.5-1.5b-instruct", "lora", 2, 0.0),
+    ("qwen2.5-1.5b-instruct", "lora", 4, 0.0),
+    ("qwen2.5-1.5b-instruct", "lora", 4, 0.7),
+]
+
+
+def _drafter(name, tcfg, max_len, cache):
+    if name == "none":
+        return S.NullDrafter()
+    if name == "ngram":
+        return S.NGramDrafter(tcfg.vocab_size)
+    if name not in cache:
+        dcfg, dp = M.load_safetensors(LORA_BASE)
+        cache[name] = (dcfg, merge_adapter(dp, dcfg, CKPT / ("lora" if name == "lora" else "dpo")))
+    dcfg, dp = cache[name]
+    return S.ModelDrafter(dcfg, dp, max_len)
+
+
+def _pass(tcfg, tp, prompts, drafter, k, temp, n_gen, max_len, cache):
+    wall = drafted = acc = steps = toks = 0
+    for p in prompts:
+        dec = S.SpecDecoder(tcfg, tp, _drafter(drafter, tcfg, max_len, cache), max_len,
+                            k=k if drafter != "none" else 0, temperature=temp, seed=p["i"])
+        t0 = time.perf_counter()
+        out, st = dec.generate(p["ids"], n_gen)
+        wall += time.perf_counter() - t0
+        toks += len(out)
+        steps += len(st)
+        drafted += sum(s.drafted for s in st)
+        acc += sum(s.accepted for s in st)
+    return {"wall_s": wall, "tok_s": toks / wall, "tokens_per_step": toks / steps,
+            "accept": acc / drafted if drafted else None}
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--n-per-kind", type=int, default=3)
+    ap.add_argument("--n-gen", type=int, default=32)
+    ap.add_argument("--reps", type=int, default=3)
+    ap.add_argument("--out", default=str(RESULTS / "speedup_bench.json"))
+    a = ap.parse_args()
+    out: dict = {"n_gen": a.n_gen, "reps": a.reps, "split": "heldout", "targets": {}}
+    for target in dict.fromkeys(c[0] for c in CONFIGS):
+        tdir = MODELS / target
+        prompts = load_prompts("heldout", a.n_per_kind, 128, tdir)
+        max_len = max(len(p["ids"]) for p in prompts) + a.n_gen + 16
+        tcfg, tp = M.load_safetensors(tdir)
+        cache: dict = {}
+        cfgs = [c for c in CONFIGS if c[0] == target]
+        temps = sorted({c[3] for c in cfgs})
+        runs = [("none", 0, t) for t in temps] + [(d, k, t) for _, d, k, t in cfgs]
+        print(f"{target}: {len(prompts)} prompts, {len(runs)} configs", flush=True)
+        for d, k, t in runs:  # warm-up: compile every prompt/step shape
+            _pass(tcfg, tp, prompts, d, k, t, a.n_gen, max_len, cache)
+        reps = {f"{d}-k{k}-T{t}": [] for d, k, t in runs}
+        for r in range(a.reps):
+            for d, k, t in runs:
+                res = _pass(tcfg, tp, prompts, d, k, t, a.n_gen, max_len, cache)
+                reps[f"{d}-k{k}-T{t}"].append(res)
+                print(f"  rep {r} {d} k={k} T={t}: {res['tok_s']:.2f} tok/s", flush=True)
+        rows = {}
+        for d, k, t in runs:
+            key = f"{d}-k{k}-T{t}"
+            base = reps[f"none-k0-T{t}"]
+            sp = [b["wall_s"] / x["wall_s"] for b, x in zip(base, reps[key], strict=True)]
+            rows[key] = {
+                "drafter": d, "k": k, "temp": t,
+                "tok_s": float(np.mean([x["tok_s"] for x in reps[key]])),
+                "speedup": float(np.mean(sp)), "speedup_range": [float(min(sp)), float(max(sp))],
+                "accept": reps[key][0]["accept"],
+                "tokens_per_step": reps[key][0]["tokens_per_step"],
+            }
+        out["targets"][target] = {"n_prompts": len(prompts), "configs": rows}
+    Path(a.out).write_text(json.dumps(out, indent=1) + "\n")
+    print(json.dumps(out, indent=1))
+
+
+if __name__ == "__main__":
+    main()
48cf15b serve: stop at EOS, real finish_reason, '+none' drafter

2026-10-06 · speculative_agent_workloads/serve/app.py, speculative_agent_workloads/tests/test_serve.py

diff --git a/draftlab/serve/app.py b/draftlab/serve/app.py
index de10174..21cd299 100644
--- a/draftlab/serve/app.py
+++ b/draftlab/serve/app.py
@@ -2,5 +2,7 @@
 
 Model names select base + drafter + k: ``<base>`` (default ngram, k=5) or
-``<base>+<drafter>-k<K>`` (e.g. ``qwen0.5b+ngram-k4``). ``usage`` adds
+``<base>+<drafter>-k<K>`` (e.g. ``qwen0.5b+ngram-k4``); ``<base>+none`` decodes
+without drafting. Generation stops at the tokenizer's EOS (finish_reason
+"stop") or at max_tokens ("length"). ``usage`` adds
 ``draft_tokens`` (drafts proposed) and ``accepted_tokens`` (drafts accepted).
 """
@@ -21,5 +23,5 @@ from pydantic import BaseModel, Field
 
 from draftlab.jaxlm import model as M
-from draftlab.jaxlm.spec import ModelDrafter, NGramDrafter, SpecDecoder
+from draftlab.jaxlm.spec import ModelDrafter, NGramDrafter, NullDrafter, SpecDecoder
 
 DEFAULT_K = 5
@@ -43,4 +45,8 @@ class HFChatTokenizer:
 
         self.tok = AutoTokenizer.from_pretrained(path)
+        ids = {self.tok.eos_token_id}
+        ids |= {self.tok.convert_tokens_to_ids(t) for t in ("<|im_end|>", "<|endoftext|>")
+                if t in self.tok.get_vocab()}
+        self.eos_ids = frozenset(i for i in ids if i is not None)
 
     def apply_chat_template(self, messages: list[dict]) -> str:
@@ -59,4 +65,5 @@ class ToyTokenizer:
 
     vocab_size = 512
+    eos_ids: frozenset[int] = frozenset()
 
     def apply_chat_template(self, messages: list[dict]) -> str:
@@ -99,4 +106,6 @@ def parse_engine_spec(model: str) -> tuple[str, str, int]:
 
 def make_drafter(kind: str, engine: Engine, max_len: int):
+    if kind == "none":
+        return NullDrafter()
     if kind == "ngram":
         return NGramDrafter(vocab_size=engine.cfg.vocab_size, max_n=4)
@@ -157,5 +166,5 @@ def create_app(engines: dict[str, Engine]) -> FastAPI:
         except ValueError:
             return _err(request, 400, f"bad model name {body.model!r}", "invalid_model_name")
-        if drafter not in ("ngram", "model"):
+        if drafter not in ("ngram", "model", "none"):
             return _err(request, 400, f"unknown drafter {drafter!r}", "invalid_drafter")
         engine = engines.get(base)
@@ -182,14 +191,31 @@ def create_app(engines: dict[str, Engine]) -> FastAPI:
         created = int(time.time())
 
+        eos = getattr(engine.tokenizer, "eos_ids", frozenset())
+
+        def tokens():
+            """(token, new_step_stats_or_None) until EOS (excluded) or max_tokens."""
+            last_st = None
+            for tok, st in decoder.generate_stream(prompt_ids, body.max_tokens):
+                if tok in eos:
+                    return
+                yield tok, (st if st is not last_st else None)
+                last_st = st
+
         if not body.stream:
-            toks, stats = decoder.generate(prompt_ids, body.max_tokens)
-            content = engine.tokenizer.decode(toks)
-            usage = _usage(len(prompt_ids), len(toks),
-                           sum(s.drafted for s in stats), sum(s.accepted for s in stats))
+            toks: list[int] = []
+            draft_tokens = accepted_tokens = 0
+            for tok, st in tokens():
+                toks.append(tok)
+                if st is not None:
+                    draft_tokens += st.drafted
+                    accepted_tokens += st.accepted
+            finish = "length" if len(toks) == body.max_tokens else "stop"
+            usage = _usage(len(prompt_ids), len(toks), draft_tokens, accepted_tokens)
             return {
                 "id": cid, "object": "chat.completion", "created": created,
                 "model": body.model,
-                "choices": [{"index": 0, "message": {"role": "assistant", "content": content},
-                             "finish_reason": "stop", "logprobs": None}],
+                "choices": [{"index": 0, "message": {"role": "assistant",
+                                                     "content": engine.tokenizer.decode(toks)},
+                             "finish_reason": finish, "logprobs": None}],
                 "usage": usage,
             }
@@ -198,10 +224,8 @@ def create_app(engines: dict[str, Engine]) -> FastAPI:
             draft_tokens = accepted_tokens = 0
             completion_tokens = 0
-            last_st = None
-            for tok, st in decoder.generate_stream(prompt_ids, body.max_tokens):
-                if st is not last_st:  # one StepStats per target-call step
+            for tok, st in tokens():
+                if st is not None:  # one StepStats per target-call step
                     draft_tokens += st.drafted
                     accepted_tokens += st.accepted
-                    last_st = st
                 completion_tokens += 1
                 delta: dict[str, Any] = {"role": "assistant",
@@ -215,8 +239,9 @@ def create_app(engines: dict[str, Engine]) -> FastAPI:
             usage = _usage(len(prompt_ids), completion_tokens,
                            draft_tokens, accepted_tokens)
+            finish = "length" if completion_tokens == body.max_tokens else "stop"
             final = {
                 "id": cid, "object": "chat.completion.chunk", "created": created,
                 "model": body.model,
-                "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
+                "choices": [{"index": 0, "delta": {}, "finish_reason": finish}],
                 "usage": usage,
             }
diff --git a/draftlab/tests/test_serve.py b/draftlab/tests/test_serve.py
index 8e83c87..621f121 100644
--- a/draftlab/tests/test_serve.py
+++ b/draftlab/tests/test_serve.py
@@ -44,5 +44,5 @@ def test_nonstream_usage(client):
     assert j["object"] == "chat.completion"
     ch = j["choices"][0]
-    assert ch["finish_reason"] == "stop"
+    assert ch["finish_reason"] == "length"  # toy tokenizer has no EOS
     assert ch["message"]["role"] == "assistant"
     assert len(ch["message"]["content"]) == 32
@@ -65,5 +65,5 @@ def test_stream_chunks_and_usage(client):
     assert all(c["choices"][0]["finish_reason"] is None for c in chunks[:-1])
     last = chunks[-1]
-    assert last["choices"][0]["finish_reason"] == "stop"
+    assert last["choices"][0]["finish_reason"] == "length"
     assert last["choices"][0]["delta"] == {}
     # deterministic (seed 0, T=0): stream must equal the non-stream result
@@ -98,2 +98,35 @@ def test_missing_fields_400(client):
 def test_unknown_model_404(client):
     assert client.post("/v1/chat/completions", json=_body(model="nope")).status_code == 404
+
+
+def test_eos_stops_generation():
+    """A token in tokenizer.eos_ids ends the completion (excluded, finish 'stop')."""
+    cfg = M.Qwen2Cfg.from_hf(tiny_config())
+    params = M.init_params(cfg, jax.random.PRNGKey(0))
+    tok = ToyTokenizer()
+    c = TestClient(create_app({"tiny": Engine("tiny", cfg, params, tok)}))
+    full = c.post("/v1/chat/completions", json=_body(model="tiny+none")).json()
+    text = full["choices"][0]["message"]["content"]
+    # ToyTokenizer.decode maps id -> chr(33 + id % 90); mark the 4th token's ids as EOS
+    stop_ch = text[3]
+    tok.eos_ids = frozenset(i for i in range(tok.vocab_size) if chr(33 + i % 90) == stop_ch)
+    cut = text.index(stop_ch)
+    for stream in (False, True):
+        r = c.post("/v1/chat/completions", json=_body(model="tiny+none", stream=stream))
+        if stream:
+            chunks = _sse_chunks(r)
+            content = "".join(ch["choices"][0]["delta"].get("content", "") for ch in chunks)
+            fin, usage = chunks[-1]["choices"][0]["finish_reason"], chunks[-1]["usage"]
+        else:
+            j = r.json()
+            content, fin, usage = j["choices"][0]["message"]["content"], j["choices"][0][
+                "finish_reason"], j["usage"]
+        assert content == text[:cut] and fin == "stop"
+        assert usage["completion_tokens"] == cut
+
+
+def test_none_drafter_drafts_nothing(client):
+    j = client.post("/v1/chat/completions", json=_body(model="tiny+none")).json()
+    assert j["usage"]["draft_tokens"] == 0 and j["usage"]["completion_tokens"] == 32
+    spec = client.post("/v1/chat/completions", json=_body()).json()
+    assert j["choices"][0]["message"]["content"] == spec["choices"][0]["message"]["content"]

Research references

Leviathan, Kalman & Matias (ICML 2023), Fast Inference from Transformers via Speculative Decoding. Sampling rule and analytic cost model.
Chen et al. (2023), Accelerating Large Language Model Decoding with Speculative Sampling. Distribution-preserving speculative sampling.