Learning proposal distributions — IGUANA & MSC

IGUANA = Importance-weighted Gradient Update with Advantage Normalization Algorithm · MSC = Markovian Score Climbing

Fine-tuning a language model to be a good proposal for posterior sampling under a constraint. Everything on this page uses plain importance sampling (IS): per prompt we draw M i.i.d. samples from the proposal qφ and weight them; when a posterior sample is needed, one resampling at the end suffices.

Why train a proposal?

The argument of the paper's introduction, in five steps.

Models make mistakes their applications cannot afford

Code that does not run, queries that do not parse, output that breaks the format a pipeline depends on. Fine-tuning makes such failures rarer but never impossible — and gives no way to tell, for a given output, whether this is one of them.

Conditioning can: write the requirement as a checker and sample the posterior

The model's own distribution, restricted to outputs that pass, renormalized. Adherence is exact by construction — acceptance is the checker passing — and the model's preferences among passing outputs are untouched, so its accuracy is kept rather than renegotiated rule by rule.

π(y|x) = p(y|x) r(x,y) / Z(x)

The price is draws, set by Z(x)

An exact posterior sample costs 1/Z(x) draws, where Z(x) is the chance a bare sample passes. Z(x) is small exactly when the requirement bites — cheap where you didn't need it, expensive where you did. With the prior as proposal and a 0/1 checker:

χ²(π‖p) = 1/Z(x) − 1, ESS ≈ M·Z(x)

Every sampler's guarantee is priced by one thing it cannot change

IS, SMC, rejection sampling, MCMC: bias of a resampled draw, effective sample size, draws-per-accept, mixing rate — all governed by the mismatch between the proposal q and the posterior π. A fixed proposal (the prior, a prompted variant, a masked decoder) freezes that mismatch, and on hard prompts freezes it at a hopeless level: on DS-1000 some prompts have Z(x) so small the base model essentially never passes, and no downstream estimator can recover strings that are not in the set.

So make the proposal the learned object

Fine-tune the model to be a better proposal: same target, same guarantee, smaller divergence between q and π, learned from the weighted samples inference already produces. Less divergence buys accuracy at a fixed budget and fewer draws for the same accuracy — the same bound read both ways.

minφ Ex KL(π(·|x) ‖ qφ(·|x))

The interaction the paper expects — a developer with a multi-page style guide, escalating:

BareFluent, idiomatic code that violates the guide throughout — nothing announced it.
AskAppend the guide. Salient rules improve; structural ones don't — corpus idiom outweighs an instruction block thousands of tokens back.
PleadRule by rule: “no lambdas”, “snake_case”. Each patched violation surfaces another; the prompt becomes a long, brittle artifact that still isn't compliant.
ConditionWrite the guide once as a checker, sample the posterior. Exact adherence, accuracy untouched — and the only cost is 1/Z(x) draws.

The escalation story is the paper's pre-registered protocol (H1–H3), not yet a measured result; the cost identity 1/Z(x) is exact.

Unlike prompting, rejection, or iterative repair — whose full cost recurs at every query — a learned proposal pays its cost once and folds it into the weights. The guarantee stays fixed while its cost falls with use. The rest of this page is about how to learn that proposal, what goes wrong, and what it buys.

Setup and objective

Given a prompt x and a potential r(x,y) ∈ [0,1], the target is the posterior that reshapes the language-model prior p (sampled at temperature 0.6 throughout):

π(y|x)  =  p(y|x) r(x,y)Z(x),  Z(x) = Ey∼p(·|x)[r(x,y)]

Objective

Train the proposal qφ to match the posterior, in the inclusive (mass-covering) direction, averaged over the prompt distribution D (here: uniform over the training prompts):

L(φ)  =  Ex∼D  KL( π(·|x) ∥ qφ(·|x) )  =  Ex∼D Σy π(y|x) log π(y|x)qφ(y|x)

Its gradient is an expectation under the posterior — the object every method here estimates:

∇φ L(φ)  =  − Ex∼D Ey∼π(·|x)[ ∇φ log qφ(y|x) ]

Change of measure: the exact per-prompt gradient

The posterior expectation is not sampleable, but rewriting it as an expectation under the proposal makes every ingredient computable — except one scale:

− ∇φ KL(π(·|x) ∥ qφ(·|x))  =  Ey∼π(·|x)[ ∇φ log qφ(y|x) ]  =  1Z(x) Ey∼qφ(·|x) [ p(y|x) r(x,y)qφ(y|x) ∇φ log qφ(y|x) ]

The score ∇φ log qφ(y|x) and the weight ratio p r / qφ are exact for any proposal draw; only the scale 1/Z(x) is intractable — a normalizer slot to be filled with an estimate Ẑ(x).

The design space is a triple

A gradient estimator in this family is a Monte Carlo average, over draws y ∼ qφ(·|x), of

1Ẑ(x) ( p(y|x) r(x,y)qφ(y|x) − b(x) ) ∇φ log qφ(y|x)

The design space is a triple, not a pair: the normalizer Ẑ(x), the baseline b(x), and the sampling distribution. IGUANA fills both statistics from previous training iterations — a normalizer pooled over the run's history, a per-prompt baseline b(x) = Ẑ(x) — so neither depends on the current batch. Conditional on that history Ẑ(x) is a constant, so the gradient is unbiased up to a positive per-prompt scale for ANY such estimator, and the baseline is mean-zero by the score identity.

Estimating Z(x)

Cross-iteration running mean. At visit K,  Ẑ(x) = Σk<K Σm uw(m,k)(K−1) M — the pooled average of all past raw weights for this prompt.

Shrinkage (what the runs below use). Early in training a prompt has few visits, so we start from ONE value shared by ALL prompts — the grand mean validity of the base model over the training set, μ0 = 0.5965 (measured from base rollouts) — and interpolate toward the per-prompt estimate as its own observations accumulate:

Ẑ(x) = n0 μpool + Sxn0 + nx,  μpool = m0 μ0 + Sallm0 + nall

where Sx, nx are the prompt's accumulated weight sum and count, Sall, nall the pooled ones, and n0 = 100, m0 = 50 pseudo-counts. A fresh prompt uses (nearly) the global value; a well-visited prompt uses (nearly) its own.

MSC vs IGUANA

IGUANA — weight-then-average

Importance-weighted Gradient Update with Advantage Normalization Algorithm

coefficient uw(m) − b(x)M Ẑ(x) for every m
baseline b(x) all M samples enter the gradient: above b(x) attract, below repel

All M i.i.d. samples enter the gradient with centered, normalized coefficients. Needs a normalizer estimate Ẑ(x); unbiased up to a positive per-prompt scale for any past-measurable choice.

MSC — sample-then-imitate

Markovian Score Climbing

select next retained ∝ uw, gradient −∇ log qφ(retained)
retained + M−1 fresh draws from q selected ∝ weight → next retained gradient: −∇ log qr(selected) weights only select; the gradient imitates that one sample

A conditional-IS kernel: the retained sample plus M−1 fresh draws; the next retained sample is selected with probability ∝ weight (leaves the posterior exactly invariant). No Ẑ(x) needed; samples are correlated across visits.

DS-1000: the task

Task, potential, and what “good” means

DS-1000 (Qwen2.5-Coder-3B, T = 0.6, M = 10). The potential is the RuntimeNoErrorPotential: r = 1 iff the completed program runs without raising an error — wrong answers are tolerated; correctness is never checked during training. What we ultimately care about is correctness on the hidden tests. For the BASE model it factors as

Pp(correct | x) = Z(x) · P(correct | valid, x)

Both factors are properties of the base model and the potential, so both are constants — Z(x) = Ey∼p[r(x,y)] does not depend on qφ, and neither does the posterior. Averaging the second factor over the benchmark — one prompt, one vote — gives the posterior accuracy (the macro skyline):

ŝ = 1N Σx s(x),   s(x) = #{y correct}#{y valid},  0/0 := 0   = 0.324

95% CI [0.285, 0.363], from 100 base rollouts per prompt on our 254 held-out prompts (genlm/rollouts per-sample table, qwen25-coder-3b at T = 0.6). Per prompt it is the accuracy of rejection sampling — keep drawing from the base model until r accepts; a prompt the base model never gets past the potential scores 0, and 2 of the 254 (0.8%) are in that state. The average is macro on purpose: held-out pass@1 is itself one-prompt-one-vote, so this is the quantity our curves can actually reach. Pooling rollouts instead would weight each prompt by Z(x) and answer a different question. Over all 845 non-Matplotlib problems the same estimator gives 0.310 [0.289, 0.333], matching the published genlm/rollouts figure.

What training moves is the proposal's own rates. For an indicator potential the posterior puts all its mass on valid samples, so a perfectly trained proposal would have validity 1 and its accuracy would equal the posterior accuracy — which is therefore the ceiling for pass@1, not a curiosity: base pass@1 is 0.195, so the headroom on DS-1000 is 0.129. And the split is empirical: P(correct | valid) stays ≈ 0.40 for every arm, so the pass@1 gain we measure is essentially all validity gain.

How much is there to learn? The Z(x) distribution

The same measurement as for LiveCodeBench below: Z(x) is the base model's pass rate on the potential, here “runs without error”. It decides in advance which prompts carry informative signal (§2 of the LCB section explains why a visit with no passing draw is a repulsion step, not a learning step). Train split: 100 offline base-model rollouts per prompt; held-out split: the 50-draw base-model eval cell. Same model (Qwen2.5-Coder-3B), temperature (0.6) and potential.

Histograms of base-model Z(x) over the DS-1000 train and held-out prompts

What we take from it. DS-1000 is the mirror image of LiveCodeBench. Almost nothing is dead: 4 train prompts and 5 held-out prompts have Z(x) exactly 0 (1% / 2%, vs 31% on LCB), and only 3% / 6% sit below 0.05. The mass is at the top: 67% of train prompts and 50% of held-out prompts already pass at least half the time (mean Z(x) 0.60 / 0.50). Two consequences seen elsewhere on this page follow directly: (i) anchor-free training survives here — the −1/M repulsion steps are rare and the rel-clip arm improved 80% of the prompts in the informative band without an anchor (§4 below); (ii) the headroom is modest — with the base model passing half the time, one extra checker call already recovers much of what training buys, which is why the base+checker curve of the compute-budget section is steep and the break-even volume large. A domain with more mass on the left (LCB) is where amortization has the most to save and where learning is hardest.

train (100 rollouts / prompt)held-out (50 draws / prompt)
librarypromptsmean Z(x)Z(x) ≥ .5promptsmean Z(x)Z(x) ≥ .5
Pandas2040.6373%870.5146%
Numpy1540.6775%660.6270%
Sklearn800.4649%350.3429%
Scipy740.5258%320.4538%
Pytorch480.5871%200.5470%
Tensorflow310.5655%140.4136%

The held-out split is harder than the train split on average (0.50 vs 0.60), mostly through Pandas and Sklearn; part of the gap may also be the two measurements (July offline rollouts vs the September eval cell, same potential and executor but different runs). Held-out Z(x) from 50 draws is quantized to 0.02, train from 100 draws to 0.01.

Offline training: distilling the rejection sampler at 30B

For a prompt x, draw N = 100 rollouts from a sampler pgen of the base model, keep the ones the checker accepts — r(x, y) = 1 when the program runs without error in the DS-1000 environment — and treat them as what they are, exact samples from the posterior:

y(i) ∼ pgen(·|x),  i = 1…N;    S(x) = { i : r(x, y(i)) = 1 },  k(x) = |S(x)|;    y(i) ∼ π(·|x) ∝ pgen(y|x) r(x,y)  for  i ∈ S(x)

Loff(φ)  =  Ex∼D Ey∼π(·|x)[ −log qφ(y|x) ]  ≈  Ex∼D 1k(x) Σi∈S(x) −log qφ(y(i)|x),     log qφ(y|x) = Σt log qφ(ct | x, c<t)

∇φ Loff(φ)  =  − Ex∼D 1k(x) Σi∈S(x) ∇φ log qφ(y(i)|x)

Two priors, one proposal. pgen is the sampler that produced the rollouts; it is the prior in every evaluation weight below. The proposal qφ is the same family in every arm — the adapted model sampled at T = 1 with no truncation, the same chat and few-shot prompt — which keeps its density exact and the arms comparable.

pgen — rollout sampler = priorqφ — proposalqφ at initialization
arms 1–2base model under its model-card decoding: T = 0.7, top-p 0.8, top-k 20, repetition penalty 1.05 (a truncated distribution) base model + LoRA adapter, T = 1, no top-p, no top-k, no penalty the base model at T = 1 (the adapter is zero): the untruncated relative of pgen, not pgen itself
arm 3base model at T = 1, no top-p, no top-k, no penalty exactly pgen

Model and training setup.

modelQwen3-Coder-30B-A3B-Instruct — mixture of experts, 30.5B total / 3.3B active parameters, BF16, trained through Hugging Face transformers + peft
adapterLoRA, r = 8, α = 16, dropout 0.05, on the attention projections q, k, v, o of all 48 layers (6.68M trainable parameters, 0.022%). Attention-only is deliberate: “all layers” on an MoE would put adapters on 128 experts × 48 layers, a different regime from the 3B recipe.
lossLoff above, summed over completion tokens (prompt tokens masked), completion capped at 400 tokens; plain cross-entropy at T = 1
optimizerAdam, constant learning rate 1e-5, no weight decay, no warm-up; gradients clipped to global norm 1.0
batch8 examples per optimizer step (micro-batch 2 × 4 accumulation steps), gradient checkpointing on
schedule3 epochs, checkpoint every 200 steps, seed 42
data699 training / 301 held-out prompts, split by prompt. Arms 1–2: 56,307 examples from 604 prompts (card rollouts; 95 prompts had no valid draw). Arm 3: 58,572 examples from 659 prompts (T = 1 rollouts; 40 prompts had none). Prompts with k(x) = 0 contribute nothing.
batchingarm 1: whole-dataset length sort (79% single-prompt batches — next subsection); arms 2–3: diverse batching
hardwarearm 1: one A100-80GB; arms 2–3: two L40S-48GB with the model sharded across both; ≈ 6 s per step, ≈ 36 h per arm

How much is there to learn at 30B? The Z(x) distribution

Z(x) is the probability that a base-model sample runs without error — the normalizer of the posterior, and the quantity that decides how much a prompt can teach. Measured from the 100 rollouts per prompt of the T = 1 sampler (arm 3’s prior), it looks nothing like the 3B picture above: the 30B model already solves most prompts outright.

How Z(x) is computed. For every prompt x we draw N = 100 independent completions from the untrained model with the arm’s own sampler (T = 1, no top-p, no top-k here; the model-card sampler for arms 1–2), run each one through the no-runtime-error check and take the fraction that pass:

Z(x)  =  Ey∼p(·|x)[ r(x,y) ]  =  Py∼p(·|x)( y runs without error )  ≈  1N Σi=1N r(x, y(i))  =  k(x)N,    y(i) ∼ p(·|x) i.i.d.

r(x,y) ∈ {0, 1} is the potential: the completion’s program (its last fenced code block) is executed inside the problem’s code context in the DS-1000 environment and counts as valid when it finishes without a non-assertion exception. So Z(x) is the normalizer of the posterior π(y|x) ∝ p(y|x) r(x,y), and k(x)/N is exactly the acceptance count the offline objective divides by. For training prompts these are the same 100 rollouts that form the training data; for held-out prompts the 100 rollouts serve only this measurement. Two consequences of the estimator: its resolution is 1/100, so “Z = 0” means 0 of 100 passed (true Z below about 1 %), not an unsolvable prompt, and its standard error √(Z(1−Z)/N) is at most 0.05. And Z(x) belongs to a sampler, not a model: truncation changes p, which is why the card sampler leaves 95 training prompts at zero where T = 1 leaves 40.

Histograms of base-model Z(x) under the T = 1 sampler over the 699 training and 301 held-out prompts
Same bins and bands as the 3B histogram above, but with a logarithmic count axis: 503 of the 699 training prompts sit in the top bin. Z(x) from 100 rollouts per prompt (Qwen3-Coder-30B-A3B-Instruct, T = 1, no truncation, no-error potential).

What we take from it. The distribution is bimodal. Of the 699 training prompts, 596 (85%) have Z > .5 and the median is Z = 1.0 — the base model never fails on them, so filtered-BC sees 100 near-identical accepted samples and has nothing to move. The informative band .05 ≤ Z < .5 holds only 47 training prompts (7%, against 30% at 3B), and 40 prompts have Z = 0 and contribute no data at all. Mean Z is 0.84 (train) / 0.82 (held-out), against 0.60 / 0.50 at 3B. Under the card sampler the picture is the same at the top and worse at the bottom: its truncation turns 95 training and 46 held-out prompts into Z = 0 (against 40 and 20 at T = 1), which is why its skyline is lower. The headroom for a correctness-blind objective at 30B is therefore small by construction, and concentrated in a few dozen prompts.

Batch composition: length sorting quietly breaks it

Distillation data has an unusual shape: ~93 accepted samples of each of 604 prompts. Every sample of a prompt carries an identical prompt prefix and a similar completion length, so sorting the dataset by length — the obvious way to cut padding — lines up all the samples of one prompt.

Batch diversity — distinct prompts per batch (batch = 8)

0246810ideal = 8whole-dataset length sort — click for how it forms batches1.46whole-datasetlength sortdiverse batching — click for how it forms batches6.34diverse(chunk + sort)pure random shuffle — click for how it forms batches7.96pure randomshuffle

Cost of that diversity — padding waste, % of batch tokens

010203040whole-dataset length sort — click for how it forms batches0%whole-datasetlength sortdiverse batching — click for how it forms batches3%diverse(chunk + sort)pure random shuffle — click for how it forms batches38%pure randomshuffle
Measured on the actual 30B training set (56,307 examples, 604 prompts, batch 8). Single-prompt batches: 79% under whole-dataset sorting, 0% under the other two. Click a bar (or focus it and press Enter) to see how that scheme forms its batches.

Both arms were trained to completion and evaluated identically; the training trajectories are compared next, the held-out results after that.

Training trajectory

image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ 10 2 10 3 10 4 optimizer step (log scale) 10 −2 weighted NLL per effective batch (log scale) running-average transient (<10 logged batches) 0.033 0.020 0.016 epoch 2 epoch 3 Training loss: running average (thin), per-epoch mean (thick) arm 1: length-sorted batches, card rollouts arm 2: diverse batches, card rollouts arm 3: diverse batches, T = 1 rollouts
Training loss of the three arms, log-log. Thin: running average of the logged per-batch weighted NLL (one batch every 20 steps, β = 0.98). Thick: mean of those batches per epoch. Dotted verticals: epoch boundaries of arms 1–2 (7,039 steps; arm 3’s fall at 7,322 and 14,644). Click a curve for that arm’s loss, a circled spike for the batch behind it (which prompts, their k, and the prompt text), or the shaded band for why it is shaded.

Reading the loss. The per-batch value is heavy-tailed — weights span 1/100 to 1, and a rare prompt arriving as a whole batch costs its full mean NLL — so the thin running averages wander and the thick per-epoch means are the thing to compare: 0.039 → 0.016 → 0.016 (arm 1), 0.0215 → 0.020 → 0.020 (arm 2), 0.031 → 0.040 → 0.033 (arm 3). Arm 1 ends lowest because it memorises the single-prompt batches it keeps re-seeing; arm 3 sits highest because untruncated T = 1 completions carry more entropy. Neither level says anything about the proposal’s quality; the held-out section does.

image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ 10 2 10 3 10 4 optimizer step (log scale) 10 −2 10 −1 ‖EMA[]‖  (log scale) β =0.9992 g EMA window (~1,000 steps) still filling: the norm falls as 1/√t just from averaging more gradients 0.0072 0.0057 0.0053 epoch 2 epoch 3 Gradient-EMA norm arm 1: length-sorted batches, card rollouts arm 2: diverse batches, card rollouts arm 3: diverse batches, T = 1 rollouts
Norm of an exponential moving average (β = 0.999) of the gradient vectors — the average first, then its length, not an average of lengths — logged every 20 steps, log-log. Dotted verticals: epoch boundaries. Click a curve for that arm’s gradient trace; click the shaded band for why it is shaded.

Reading the gradient norm. All three arms trace the same curve: steep at first (≈ t−1/2, while the average is still filling its window), then decelerating, then nearly flat over the last decade as it reaches the floor a 1,000-step average can resolve. The prompt-diverse arms sit a constant 20–25% below arm 1 at every step — the batching fix lowered the gradient variance, not the dynamics — and arm 3, despite the highest loss, decays exactly like the others. No instability in any arm. We quote no power-law exponent: a fitted slope depends on where the fit starts.

Held-out results (M = 10, R = 5) — by epoch

Same evaluation as the 3B arms in the IGUANA and MSC section, base model included and both batching arms evaluated identically: 50 draws per prompt from the proposal sampled untruncated at T = 1, graded in the DS-1000 environment. wacc is the weighted accuracy of the particle set (pass@1 after one final resampling) with weights φ·p/q; pass@1 is the proposal sampled directly, no weights; validity is the fraction of draws that run; pass@50 is the oracle coverage bound. For the base model q = p and the weights reduce to φ: that row is rejection sampling with the checker at M = 10. The prior p is always the sampler that generated the arm’s training rollouts — the model-card sampler for arms 1–2 (truncated, so a draw outside its nucleus has weight 0), the plain T = 1 distribution for arm 3 and for the online MSC arm — and each arm is compared with rejection sampling from that same prior.

x-axis:
image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ 0.56 0.58 0.60 0.62 0.64 0.66 0.68 0.70 0.648 0.640 0.633 0.628 0.628 0.617 weighted accuracy — prior = the rollout sampler of each arm ⓘ skyline, T = 1 prior 0.635 arm 3: diverse batches, T = 1 rollouts online MSC: T = 1 prior, 1 prompt × 10 draws per step online IGUANA: rel-clip 3·Ẑ, grad clip 1, 400-token draws online RB-MSC: T = 1 prior, 1 prompt × 2 draws per step, gradient over all 3 particles online MSC, M = 2 control: same config as RB-MSC, single-sample score step online IGUANA, M = 1: one draw and one prompt per step, v2 recipe // 0.525 0.550 0.575 0.600 0.625 0.650 0.675 0.614 0.614 0.610 0.607 0.600 0.592 pass@1 of the proposal itself ⓘ skyline, T = 1 prior 0.635 skyline, T = 1 prior 0.635 // 0 500 1k 2k 5k 10k 20k optimizer step (log scale; 0 = untrained model) 0.775 0.800 0.825 0.850 0.875 0.900 0.925 0.891 0.887 0.882 0.879 0.862 0.849 validity rate of the proposal ⓘ // 0 500 1k 2k 5k 10k 20k optimizer step (log scale; 0 = untrained model) 0.65 0.70 0.75 0.80 0.764 0.764 0.748 0.731 0.701 0.678 pass@50 (coverage / oracle bound) ⓘ // Held-out IS evaluation (M = 10, R = 5), 301 prompts — arms trained with the plain T = 1 prior; p = the T = 1 distribution
Click a panel title (ⓘ) for how that metric is measured. The x-axis is the optimizer step on a log scale by default (step 0, the untrained model, sits left of the break because a log axis has no zero); the buttons above the figure switch to a linear step axis, or to two accountings that make the arms comparable, since one optimizer step means different things in each: draws generated counts every completion the training procedure sampled (the offline arms draw 100 rollouts per training prompt, 69,900 in total, before their first step, so all their checkpoints sit at one x; MSC and IGUANA draw 10 per step), and sequences in the gradient counts the completions that received a gradient (8 accepted rollouts per step offline, the 1 selected reference per step for MSC, all 10 draws per step for IGUANA). Pink: arm 3 (offline filtered behaviour cloning on T = 1 rollouts). Green, dashed squares: online MSC. Purple, dash-dot diamonds: online IGUANA (rel-clip 3·&Zcirc;, gradient clip 1, 400-token draws). All three train on, and are weighted with, the plain T = 1 distribution; their step-0 point is rejection sampling from it at M = 10 (0.608); the finely dotted line in the wacc and pass@1 panels is that prior’s rejection-sampling skyline (unlimited draws). Shaded bands are 95% prompt-level bootstrap CIs; all arms share prompts and seed, so the paired differences in the tables are the sharper test.

Arm 3 — diverse batching, rollouts from the plain T = 1 distribution (prior = that same distribution; same split, seed and hyperparameters).

checkpointwacc (prior = T = 1)clippedpass@1validitypass@50ESS / 10runs with no valid draw
rejection sampling, M → ∞ (skyline)0.635 [0.582, 0.685]——————
base (RS from pT=1)0.608—0.5670.8170.7188.211.4%
20000.628 [0.577, 0.678]0.6340.5970.8620.7487.17.2%
50000.647 [0.597, 0.696]0.6490.6100.8680.7577.17.4%
80000.638 [0.588, 0.688]0.6430.6070.8700.7647.06.4%
110000.650 [0.600, 0.701]0.6520.6120.8760.7617.16.0%
219660.648 [0.598, 0.697]0.6470.6140.8790.7647.06.5%

Arms 1–2, evaluated under the model-card sampler that generated their rollouts. Same held-out prompts, draws and graders; only the prior in the weights differs, and with it the rejection-sampling reference (0.569 at M = 10, skyline 0.578).

x-axis:
image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ 0.52 0.54 0.56 0.58 0.60 0.62 0.64 0.576 0.574 weighted accuracy — prior = the rollout sampler of each arm ⓘ skyline, card prior 0.578 arm 1: length-sorted batches, card rollouts arm 2: diverse batches, card rollouts // 0.52 0.54 0.56 0.58 0.60 0.62 0.64 0.600 0.599 pass@1 of the proposal itself ⓘ skyline, card prior 0.578 skyline, card prior 0.578 // 0 500 1k 2k 5k 10k 20k optimizer step (log scale; 0 = untrained model) 0.78 0.80 0.82 0.84 0.86 0.88 0.90 0.861 0.856 validity rate of the proposal ⓘ // 0 500 1k 2k 5k 10k 20k optimizer step (log scale; 0 = untrained model) 0.66 0.68 0.70 0.72 0.74 0.76 0.78 0.80 0.741 0.738 pass@50 (coverage / oracle bound) ⓘ // Held-out IS evaluation (M = 10, R = 5), 301 prompts — arms trained on card-sampler rollouts; p = the card sampler
Click a panel title (ⓘ) for how that metric is measured. The x-axis is the optimizer step on a log scale by default (step 0, the untrained model, sits left of the break because a log axis has no zero); the buttons above the figure switch to a linear step axis, or to two accountings that make the arms comparable, since one optimizer step means different things in each: draws generated counts every completion the training procedure sampled (the offline arms draw 100 rollouts per training prompt, 69,900 in total, before their first step, so all their checkpoints sit at one x; MSC and IGUANA draw 10 per step), and sequences in the gradient counts the completions that received a gradient (8 accepted rollouts per step offline, the 1 selected reference per step for MSC, all 10 draws per step for IGUANA). Orange: arm 1 (whole-dataset length sort). Blue: arm 2 (diverse batches). Both train on model-card-sampler rollouts and are weighted with that truncated prior (a draw outside its nucleus has weight 0); their step-0 point is rejection sampling from the card sampler at M = 10 (0.569); the dotted line is its skyline. The pass@1, validity and pass@50 panels involve no prior and share the same base model as the figure above.

Arm 1 — whole-dataset length sort, card-sampler rollouts (prior = the card sampler).

checkpointwacc (prior = card sampler)clippedpaired Δ vs RS basepass@1validitypass@50ESS / 10runs with no usable drawdraws inside the nucleus
rejection sampling, M → ∞ (skyline)0.578 [0.524, 0.630]————————
base (RS from pcard)0.569——0.5530.8050.615—16.4%100%
untrained, same T = 1 proposal0.572 [0.518, 0.625]——0.5710.8170.7245.820.3%73%
20000.585 [0.532, 0.639]0.588+0.016 [-0.012, +0.043]
vs same-proposal control +0.013 [-0.005, +0.033]		0.5920.8460.7345.917.0%74%
50000.573 [0.518, 0.627]0.576+0.004 [-0.022, +0.031]
vs same-proposal control +0.001 [-0.016, +0.018]		0.5910.8450.7345.917.9%74%
80000.568 [0.514, 0.622]0.571-0.001 [-0.028, +0.026]
vs same-proposal control -0.004 [-0.020, +0.011]		0.5940.8500.7515.919.3%74%
110000.581 [0.527, 0.634]0.585+0.012 [-0.012, +0.038]
vs same-proposal control +0.009 [-0.008, +0.027]		0.5990.8510.7445.818.2%74%
211170.576 [0.521, 0.629]0.579+0.006 [-0.019, +0.035]
vs same-proposal control +0.004 [-0.013, +0.022]		0.6000.8560.7385.818.4%72%

Arm 2 — diverse batching, card-sampler rollouts (prior = the card sampler; same data, split, seed and objective as arm 1).

checkpointwacc (prior = card sampler)clippedpaired Δ vs RS basepass@1validitypass@50ESS / 10runs with no usable drawdraws inside the nucleus
rejection sampling, M → ∞ (skyline)0.578 [0.524, 0.630]————————
base (RS from pcard)0.569——0.5530.8050.615—16.4%100%
untrained, same T = 1 proposal0.572 [0.518, 0.625]——0.5710.8170.7245.820.3%73%
20000.580 [0.527, 0.633]0.584+0.011 [-0.013, +0.036]
vs same-proposal control +0.008 [-0.005, +0.022]		0.5980.8540.7445.918.0%73%
50000.582 [0.530, 0.636]0.583+0.013 [-0.012, +0.038]
vs same-proposal control +0.010 [-0.005, +0.026]		0.5980.8540.7385.919.0%74%
80000.576 [0.522, 0.629]0.576+0.006 [-0.018, +0.033]
vs same-proposal control +0.004 [-0.014, +0.021]		0.5910.8520.7385.819.6%72%
110000.580 [0.527, 0.633]0.580+0.011 [-0.014, +0.038]
vs same-proposal control +0.008 [-0.006, +0.024]		0.5960.8540.7245.918.3%74%
211170.574 [0.520, 0.628]0.579+0.005 [-0.021, +0.033]
vs same-proposal control +0.002 [-0.012, +0.017]		0.5990.8610.7415.819.5%71%

What we take from it. Arms 1–2, under the sampler that generated their rollouts, do not beat rejection sampling from it: wacc 0.568–0.585 against 0.569, paired Δ -0.001 to +0.016, 0 of 10 cells with the 95% CI above zero; the two arms are indistinguishable from each other throughout. What training does change for them is prior-free: pass@1 of the proposal itself rises 0.567 → 0.600 / 0.599 (paired +0.032 [+0.015, +0.050]), validity 0.817 → 0.856 / 0.861, and coverage (pass@50) does not decay. P(correct | valid) stays flat at ≈ 0.70 in every arm, so with the checker in the loop a better sampler buys nothing once the truncated prior has thrown away a quarter of its draws. Arm 3, whose rollouts, prior and proposal all come from the plain T = 1 family, reaches wacc 0.628–0.650: paired Δwacc vs rejection sampling +0.020 to +0.042, 5 of 5 cells with the 95% CI above zero; pass@1 0.614 (paired +0.047 [+0.026, +0.070]), validity 0.879, pass@50 0.764; its share of runs with no valid draw is 6.0%–7.4% and its ESS 7.0–7.1 of 10, both better than arms 1–2. Against arm 2 at matched checkpoints it is ahead by -0.001 to +0.021 (paired), significant at 3 of 5. The rejection-sampling skyline — unlimited draws from the prior, i.e. macro P(correct | valid) over the held-out prompts with 0/0 := 0 — is 0.578 for the card sampler (46 of 301 prompts have no valid card draw in 100 and count 0) and 0.635 for the T = 1 sampler (20 such prompts). Arms 1–2 sit on their skyline (paired, final checkpoint: -0.003 [-0.030, +0.025]) — ten card draws already reach almost everything unlimited draws would. Arm 3 crosses its skyline (paired +0.013 [-0.006, +0.033]): the trained proposal produces valid, correct programs for prompts the base sampler never solved in 100 draws, which no amount of rejection sampling from the prior can do.

On the weights. Both densities are scored on the identical sampled tokens in a single HF forward (adapter on / off), because on this bf16 MoE a learned density is reproducible only to about one nat per sequence — expert routing is a discrete top-k on bf16 router logits, and vLLM’s decode pass, its prefill pass and HF resolve near-ties differently. The headline weights are unclipped self-normalized IS; the clipped column (weights capped at 1, the 3B convention) agrees within 0.004–0.008 at every checkpoint. For the card prior the density is the base model’s card decoding — repetition penalty 1.05, T 0.7, top-k 20, top-p 0.8 in vLLM’s order, renormalized over the nucleus, with the 3B pipeline’s nucleus tolerance at the floating-point boundary.

The amortization reading, arm by arm. Arms 1–2: one checker-free draw from the trained proposal (0.600 / 0.599) beats ten draws from the card sampler plus the checker (0.569); against that baseline amortization pays, but mostly because the card sampler is a weak way to sample this model on DS-1000. Arm 3: one checker-free draw (0.614) matches ten draws from the plain T = 1 sampler plus the checker (0.608), and the trained proposal plus the checker reaches 0.648. The headroom is small by construction: the few-shot prompt had already concentrated the model, leaving a gap to the P(correct | valid) ceiling of ≈ 0.70 that filtered behaviour cloning can close only through validity. Going past it needs an objective that is not correctness-blind (we hold official-grader labels for every training rollout), or selection at inference.

Online MSC at 30B — the same T = 1 protocol, learned on-policy

Filtered behaviour cloning is importance sampling with the proposal frozen at the prior. Markovian score climbing (MSC) optimises the same inclusive objective, but samples from the current proposal and lets a Markov chain, not a fixed rollout set, stand in for the posterior:

LMSC(φ)  =  Ex∼D KL( π(·|x) ‖ qφ(·|x) ),     ∇φ LMSC  =  − Ex∼D Ey∼π(·|x)[ ∇φ log qφ(y|x) ]

y(1..M) ∼ qφ(·|x),   y(0) = the retained reference;    wi ∝ r(x, y(i)) p(y(i)|x)qφ(y(i)|x),   i = 0…M;    yref ∼ Cat(w);    φ ← φ + η ∇φ log qφ(yref|x)

The kernel is conditional importance sampling: the reference from the previous visit to x is re-weighted under the current proposal and competes with M = 10 fresh draws; one of the M+1 candidates is selected in proportion to its weight and the step fits that single sequence by maximum likelihood (“score climbing”). The chain’s stationary law is π, so the gradient is consistent for the inclusive KL; the persistent reference also supplies a target on visits where every fresh draw fails. Everything else is arm 3’s: the same model, few-shot chat prompt, split, LoRA (q/k/v/o, r = 8), Adam at 10−5, clip 1.0, seed, and the prior p = the plain T = 1 distribution — so at step 0 the weights are exactly r and the first steps are rejection sampling. One prompt and 10 draws per optimizer step (the offline arms fit 8 accepted rollouts per step), 14,000 steps ≈ 20 passes over the 699 training prompts, the potential executed on-policy inside the loop in the same DS-1000 environment; generation runs on a second box (vLLM, the adapter shipped each step), densities on the training copy. Held-out evaluation is identical to the arms above: 50 draws at T = 1, weights φ·p/q with p the T = 1 prior. The run completed 7,000 steps without a restart and was then resumed to 14,000 steps (same configuration; optimizer state restored).

stepwacc (prior = T = 1)clippedφ-only (RS)pass@1validitypass@50ESS / 10runs with no valid draw
base (RS from pT=1)0.608—0.6080.5670.8170.7188.211.4%
5000.613 [0.562, 0.663]0.6190.6170.5890.8540.7417.18.8%
10000.639 [0.588, 0.687]0.6420.6410.6120.8640.7486.97.5%
20000.627 [0.576, 0.676]0.6320.6340.6100.8710.7616.97.4%
30000.640 [0.589, 0.691]0.6420.6390.6160.8760.7416.97.2%
40000.635 [0.583, 0.685]0.6430.6350.6150.8800.7316.87.5%
50000.633 [0.581, 0.684]0.6340.6280.6020.8720.7346.67.8%
70000.630 [0.579, 0.682]0.6340.6300.6150.8920.7246.77.4%
90000.628 [0.575, 0.679]0.6290.6260.6080.8890.7086.77.4%
110000.644 [0.591, 0.695]0.6430.6340.6140.8930.7216.86.7%
140000.633 [0.580, 0.687]0.6340.6250.6100.8910.7016.97.7%

Training-side diagnostics. Three views of the same run, step by step: the loss, the gradient norm, and the norm of an exponential moving average of the gradient vector. The loss is −log qφ(yref|x) for the single sequence selected at that step, so it swings over six decades with the length and surprise of the reference (a 30-token block the model already assigns probability ≈ 1 costs 10−4; a long or unusual one costs tens of nats); its running average is flat at a few nats from the start. The per-step gradient norm sits at 101–102 and is clipped to 1.0 on almost every step, so Adam sees a stream of unit-norm directions. The third panel is the quantity that distinguishes a converged optimizer from a noisy one: the norm of the averaged gradient vector. For a β = 0.9 average of unit-norm steps whose directions are independent, that norm settles at √((1−β)/(1+β)) = 0.23; the checkpoints sit at 0.12–0.24, i.e. at the noise floor from the first checkpoint on — the expected gradient is already small compared with its per-step noise, exactly as on the offline arms. Note the panel is not comparable to the offline arms’ gradient-EMA curve: that one uses β = 0.999 (a ~2,000-step average, floor 0.022, which is why it visibly falls), this one β = 0.9 (~10 steps, floor 0.23), so it can only say that the expected gradient is small relative to the per-step noise, not trace its decay. The β = 0.999 gradient-vector EMA itself is logged from step 4,306 onward (the runner was instrumented mid-run); the fourth panel shows it: from the bias-corrected start at 1 it falls by two orders of magnitude to 0.011 at step 14,000, below even the 0.022 floor of independent unit-norm steps (the applied gradients are clipped to norm 1, not always at it, so the floor is an upper bound) — the expected gradient has vanished relative to the per-step noise, the same picture as the offline arms. A seed-identical rerun would give the curve over the first 4,300 steps as well.

Four-panel training diagnostics of the online MSC arm over 14,000 steps: loss, gradient norm, Adam first-moment norm, gradient-EMA norm
Thin lines: every optimizer step; thick: running average with β = 0.98. First: single-sequence loss, log scale. Second: pre-clip gradient norm, log scale, with the clip threshold 1.0. Third: norm of Adam's first moment (a β = 0.9 EMA of the applied, clipped gradient) read from each of the 57 saved checkpoints, against the noise floor for independent unit-norm steps — same construction as the offline arms' curve but a 200× shorter memory, whose floor √(0.1/1.9) = 0.23 is where these points sit. Fourth: the β = 0.999 gradient-vector EMA norm itself (bias-corrected), the quantity plotted for the offline arms, logged from step 4,306 on: it decays from 1 to 0.011, below the 0.022 unit-norm floor.

What we take from it. At step 14,000 MSC reaches wacc 0.633 (paired Δ vs rejection sampling +0.025 [+0.002, +0.050]; 8 of 10 cells with the CI above zero), pass@1 0.610 (paired +0.043 [+0.015, +0.072]) and validity 0.891 — against arm 3’s 0.628 / 0.597 / 0.862 at its first checkpoint (step 2,000, i.e. 16,000 accepted rollouts). Measured in held-out gain per optimizer step the on-policy chain is at least as efficient as the offline arm; measured in generated samples it is far cheaper, since the offline arm first drew 100 rollouts per prompt. On the training side the chain behaves like the 3B MSC arm: over the 14,000 steps the on-policy validity climbs 0.86 → 0.96 (per 1,000-step block), the share of distinct programs among valid draws falls 0.35 → 0.20, the per-token entropy 0.038 → 0.013, and the share of visits with no valid draw drops 7% → 3% — self-distillation concentrates the proposal. On held-out prompts that concentration shows up as the cost seen at 3B: pass@50 peaks at 0.761 at step 2,000 and drifts down to 0.70–0.72 from step 5,000 on, below the untrained model’s 0.718 by step 14,000, while wacc and pass@1 stay flat at ≈0.63 / 0.61 from step 1,000 onward. The proposal keeps getting sharper on its training prompts for 20 passes without that buying any held-out accuracy.

Online IGUANA at 30B — relative clipping, the 3B recipe transplanted and repaired

IGUANA is the importance-weighted gradient of the same inclusive objective, estimated from the proposal’s own draws with a per-prompt baseline and normalizer:

y(1..M) ∼ qφ(·|x),   uwi = r(x, y(i)) p(y(i)|x)qφ(y(i)|x),    ∇φ L  ≈  − Σi min(uwi, c &Zcirc;(x)) − b(x)M &Zcirc;(x) ∇φ log qφ(y(i)|x)

Every draw enters the gradient: a valid draw with weight above the baseline is pushed up, one below it is pushed down, and a failed draw (r = 0) gets exactly −b(x)/(M &Zcirc;(x)), the repulsion that makes the estimator unbiased for the inclusive KL rather than a filtered likelihood. &Zcirc;(x) is a running, per-prompt estimate of the normalizer Z(x) = Ep[r], pooled across visits with Beta-style shrinkage toward a global level (pseudo-counts n0 per prompt and m0 for the pool, prior mean μ0), updated only after the current step has used it (exclude-current, so the weights are a martingale difference). The clip c &Zcirc;(x) is relative: it caps how far above the normalizer a single draw may pull.

what3B recipe on the page (“rel clip, no anchor”)the 30B run shown (v2)
model, prompt, split, potentialas arm 3 and MSC: Qwen3-Coder-30B-A3B-Instruct, the one-shot chat prompt, 699 / 301 split, no-error potential executed on-policy in the loop
proposal and priorboth sampled at T = 1, no top-p/top-k; densities scored on the identical tokens in one forward (adapter on / off)
draws per step1 prompt × M = 10, up to 1,024 tokens, EOS stop1 prompt × M = 10, up to 400 tokens, stopped at the closing fence or EOS
weight clipuw ≤ 10 &Zcirc;(x), raw weights feed &Zcirc;uw ≤ 3 &Zcirc;(x); baseline capped at 1 (b = min(&Zcirc;, 1)); folds into the gradient pool clipped at 10 &Zcirc;
&Zcirc;(x)shrinkage pool, n0 = 100, m0 = 50, μ0 = 0.838 (the training set’s mean Z under T = 1), exclude-current, no anchor
optimizerAdam 10−5, no gradient-norm clipAdam 10−5, gradient-norm clip 1.0 (as arm 3 and MSC)
adapter, scheduleLoRA q/k/v/o, r = 8, α = 16; 14,000 steps planned, checkpoint every 250, seed 42; held-out evaluation identical to the arms above

What the 3B settings did at 30B. Launched as-is, the run broke in three steps. One draw of a ~100-token program carried uw = 5,542 (a density ratio of 8.6 nats after two optimizer steps — far too large to be drift; it is the bf16 mixture-of-experts’ routing noise between the adapter-on and adapter-off scoring passes, the same path noise measured for the evaluation). Because raw weights feed the shrinkage pool and the pool is shared across prompts, &Zcirc; jumped from 0.9 to 152 for every prompt; the baseline then exceeded every honest weight and all particles were repelled — the normalizer-inflation failure first seen on LiveCodeBench. Capping the baseline at 1 and fold-clipping the pool (the repairs built for that failure) kept &Zcirc; ≈ 1.1, but single draws with weights in the thousands still arrived, with unclipped gradient norms of 300–1,000 that Adam’s second-moment memory then carried for hundreds of steps; that run (the reference) did not move off the base model. The reference run with the 3B settings plus the two guards (second table; not drawn in the figures) stayed at base level: wacc 0.611–0.621, pass@1 0.571–0.584 over its 3 cells, before it was stopped at step 1,646. The run shown adds the three repairs in the right-hand column: a tighter relative clip, the gradient-norm clip the other two 30B arms already used, and draws capped at 400 tokens (arm 3’s training cap; 0.2% of draws are longer) so the length-driven part of the ratio tail is removed at the source.

Four-panel training diagnostics of the online IGUANA v2 arm: validity and entropy vs MSC, gradient norm, raw-weight tail vs the clip, gradient-EMA norm
Left: on-policy validity (solid) and per-token entropy (dotted) per 250 steps, IGUANA v2 against MSC: IGUANA climbs more slowly and keeps its entropy, MSC sharpens. Second: pre-clip gradient norm per step (thin) and its median per 250 steps. Third: the largest raw weight r·p/q in each step (dots, log scale) against the use-time clip 3·Ẑ(x) (line): the heavy tail is permanent, and the clip is what bounds it. Right: the β = 0.999 gradient-vector EMA norm, logged from step 1 for this arm, against the 0.022 floor for unit-norm steps; MSC's curve (from step 4,306) for comparison.
stepwacc (prior = T = 1)clippedφ-only (RS)pass@1validitypass@50ESS / 10runs with no valid draw
base (RS from pT=1)0.608—0.6080.5670.8170.7188.211.4%
5000.624 [0.572, 0.673]0.6280.6260.5870.8370.7487.09.6%
10000.619 [0.566, 0.668]0.6250.6240.5840.8380.7416.98.8%
15000.617 [0.565, 0.668]0.6200.6180.5770.8350.7416.89.7%
20000.628 [0.577, 0.677]0.6300.6290.5900.8410.7417.08.8%
30000.630 [0.579, 0.680]0.6320.6300.5990.8520.7417.18.9%
40000.628 [0.577, 0.677]0.6320.6290.5880.8390.7486.78.8%
50000.627 [0.576, 0.677]0.6330.6310.5930.8410.7417.08.6%
70000.624 [0.572, 0.674]0.6290.6270.5860.8360.7416.98.5%
90000.630 [0.581, 0.678]0.6400.6380.6150.8680.7547.07.6%
110000.630 [0.579, 0.679]0.6340.6310.5940.8530.7347.07.2%
140000.628 [0.578, 0.678]0.6340.6320.6070.8620.7647.17.5%

Reference: the 3B settings plus the baseline and fold guards (stopped at step 1,646).

stepwacc (prior = T = 1)clippedφ-only (RS)pass@1validitypass@50ESS / 10runs with no valid draw
base (RS from pT=1)0.608—0.6080.5670.8170.7188.211.4%
5000.611 [0.560, 0.660]0.6150.6130.5710.8250.7346.810.6%
10000.616 [0.564, 0.666]0.6180.6160.5720.8260.7416.99.8%
15000.621 [0.568, 0.669]0.6240.6220.5840.8340.7386.99.4%

What we take from it. Paired per prompt against the untrained model, IGUANA v2 gains in its first cells and then holds (step 500: wacc +0.015, pass@1 +0.019, validity +0.020, pass@50 +0.030); at step 14,000 wacc +0.020 [+0.005, +0.036], pass@1 +0.039 [+0.021, +0.059], validity +0.045, pass@50 +0.047 [+0.023, +0.073]; the 95% interval excludes zero in 10 of 11 cells for pass@1 and 10 of 11 for pass@50. The cells from 500 on are within noise of each other on the 287 non-TensorFlow prompts (the 14 TensorFlow prompts are the noisiest in every arm: their programs take 40–120 s to grade), so the run plateaus at the rejection-sampling skyline of its prior (0.628 against 0.635) without crossing it, where arm 3 crossed it. Against MSC at the same step (14,000), paired: wacc -0.005, pass@1 -0.003 [-0.030, +0.022], validity -0.029, pass@50 +0.063 [+0.030, +0.096]. What distinguishes it from MSC is what it does not do: pass@50 stays at its early gain instead of decaying, the per-token entropy stays flat, and the share of distinct programs among valid training draws stays at 0.33 (MSC: 0.35 → 0.20). IGUANA declines to sharpen prompts it already solves — on 46% of its steps every weight is ≈ 1 and the centred weights vanish — which preserves coverage but also caps what the no-error potential can teach it. Two cells (4,000 and 5,000) were first graded on an Euler node that timed out on every TensorFlow program and were re-graded; the audit of all other cells found no such case.

Which prompts actually learn? Held-out validity by Z(x)

The same question the 3B arms were asked, with the one difference offline training forces: there is no training-side validity to read, because the trainer never samples — it fits fixed rollouts. We read the held-out draws instead: for each checkpoint, the 50 draws per prompt of the IS evaluation, grouped by the prompt’s base-model Z(x) under the arm’s own prior.

Held-out validity per checkpoint grouped by base-model Z(x): arm 3, online MSC, online IGUANA (bins from Z under T = 1) and arm 2 (bins from Z under the card sampler)
Lines are the mean validity (fraction of 50 draws that run) of a bin's held-out prompts at each checkpoint; step 0 is the untrained model sampled at T = 1. The three T = 1 arms are binned by the T = 1 sampler's Z(x); arm 2 by the card sampler's Z(x), so its Z = 0 bin is larger (46 prompts) and starts at 0.17 because the untruncated T = 1 proposal already solves some of what the truncated sampler cannot. Arm 1 is within 0.03 of arm 2 in every bin with more than two prompts and is omitted.

What we take from it. At 30B the hard prompts are the ones that move. For arm 3, the Z = 0 prompts — 20 prompts the base model never solved in 100 draws — go from 0.00 to 0.24 held-out validity; 0 < Z ≤ .1 from 0.03 to 0.36, .1 < Z ≤ .3 from 0.17 to 0.62, .3 < Z ≤ .6 from 0.50 to 0.77, while the 244 prompts with Z > .6 stay at 0.98 because there is nothing left to gain. This is the per-prompt face of the skyline crossing: rejection sampling from the prior can never score on a Z = 0 prompt, the trained proposal does. It is also the opposite of the 3B anchor-free arms, where nothing below Z ≈ .3 ever moved — here the adapter generalises from the prompts it was trained on to prompts whose own rollouts contributed nothing. The two online arms split on exactly these prompts: MSC lifts the Z = 0 bin further than arm 3 (0.00 → 0.30) and the 0 < Z ≤ .1 bin to 0.51, because its selected reference eventually lands on a valid program and is then fit outright; IGUANA barely moves them (0.00 → 0.02 and 0.03 → 0.12): on a prompt where all ten draws fail, IGUANA has only the −1/M repulsion to apply and no positive sample to pull toward, so its gains sit in the .1 < Z ≤ .6 bins. Under the card prior (right panel) arm 2 shows the same muted pattern: its Z = 0 bin reaches 0.33. In every arm, what changes is validity — consistent with P(correct | valid) staying flat.

The same view on the training prompts

Validity on the 699 training prompts, measured the same way as held-out: at each checkpoint, 50 fresh draws per prompt (5 runs × 10, proposal sampled at T = 1), grouped by the prompt’s base-model Z(x) under the T = 1 sampler. Step 0 is the untrained model’s 100 rollouts per prompt — the rollouts that define Z(x), so at step 0 every bin sits at its own Z by construction. For arm 3 these are the prompts whose accepted rollouts it was fit on; for MSC and IGUANA they are the prompts whose draws produced the gradient. The checkpoints are a subset of the held-out ones (arm 3: 2,000 / 8,000 / 21,966; MSC: 1,000 / 3,000 / 7,000 / 14,000; IGUANA: 1,000 / 3,000 / 5,000 / 7,000).

Training-prompt validity per checkpoint grouped by base-model Z(x): arm 3, online MSC, online IGUANA (bins from Z under T = 1)
Lines are the mean validity of a bin's training prompts at each checkpoint, from 50 fresh draws per prompt (step 0: the 100 base rollouts). The training set has 40 prompts at Z = 0 (held-out: 20) and 585 at Z > .6.

What we take from it. On the prompts it trained on, arm 3 moves the hard bins much further than on held-out prompts: 0 < Z ≤ .1 goes 0.04 → 0.82 (held-out 0.03 → 0.36), .1 < Z ≤ .3 0.21 → 0.91 and .3 < Z ≤ .6 0.47 → 0.92, while the 40 training prompts at Z = 0 reach only 0.00 → 0.22 — the same ceiling as held-out, because a prompt with no valid rollout contributes nothing to the filtered-BC data and can only be reached by transfer from its neighbours. MSC, which samples its own training data, is the arm that breaks that ceiling: its training Z = 0 bin reaches 0.00 → 0.47 and 0 < Z ≤ .1 0.04 → 0.90, with mean training validity 0.838 → 0.958 against arm 3’s 0.838 → 0.934. IGUANA, which draws its own training samples, lifts the training Z = 0 bin 0.00 → 0.03 and 0 < Z ≤ .1 0.04 → 0.27 by step 7,000; its mean training validity goes 0.838 → 0.883. The gap between training and held-out curves is the generalisation gap of each arm on validity.

Which prompts become correct? Held-out correctness by C(x)

The same view for correctness instead of validity, and per prompt. For each held-out prompt x, C(x) is the fraction of the untrained model’s 100 rollouts — from the arm’s own sampler, the same rollouts that define Z(x) — that pass the official DS-1000 tests: C(x) = (1/100) Σi 1[y(i) correct], with C(x) ≤ Z(x) since a correct program also runs. It is measured once and fixes the groups. At every checkpoint a prompt’s correctness is the fraction of its 50 held-out draws (5 runs × 10, proposal sampled at T = 1) that pass the tests, i.e. the proposal’s own pass@1 on that prompt with no checker and no weights; the line is its mean over the group. The x-axis is optimizer steps on a log scale with step 0 (the untrained model) left of the break and epochs marked on top; arm 3 sees each training prompt’s accepted rollouts once per 7,322 steps, MSC visits each training prompt once per 699 steps. The second figure draws every held-out prompt as its own line.

Held-out correctness per checkpoint grouped by the untrained model's C(x): arm 3, online MSC, online IGUANA (C under T = 1) and arm 2 (C under the card sampler)
Mean fraction of 50 draws passing the official tests, per C(x) group, at each checkpoint. The three T = 1 arms are grouped by C(x) under the T = 1 sampler; arm 2 by C(x) under the card sampler (its step 0 is the untrained model sampled at T = 1, which already solves some prompts the card sampler never does, hence a C = 0 group starting above zero). With 50 draws a single prompt's value has a standard error of up to 0.07; the group means are much tighter.
image/svg+xml Matplotlib v3.9.4, https://matplotlib.org/ 0 500 1k 2k 5k 10k 20k 0.0 0.2 0.4 0.6 0.8 1.0 C = 0 (78 prompts) correct fraction up 5 · down 0 · of 78 // Arm 3 — offline, T = 1 rollouts 0 500 1k 2k 5k 10k up 5 · down 0 · of 78 // Online MSC — T = 1 prior 0 500 1k 2k 5k 10k up 2 · down 0 · of 78 // Online IGUANA v2 — T = 1 prior 0 500 1k 2k 5k 10k 20k up 2 · down 0 · of 78 // Arm 2 — offline, card rollouts 0 500 1k 2k 5k 10k 20k 0.0 0.2 0.4 0.6 0.8 1.0 0 < C ≤ .1 (27 prompts) correct fraction up 3 · down 0 · of 27 // 0 500 1k 2k 5k 10k up 7 · down 0 · of 27 // 0 500 1k 2k 5k 10k up 4 · down 0 · of 27 // 0 500 1k 2k 5k 10k 20k up 4 · down 0 · of 27 // 0 500 1k 2k 5k 10k 20k 0.0 0.2 0.4 0.6 0.8 1.0 .1 < C ≤ .3 (10 prompts) correct fraction up 7 · down 0 · of 10 // 0 500 1k 2k 5k 10k up 6 · down 1 · of 10 // 0 500 1k 2k 5k 10k up 5 · down 0 · of 10 // 0 500 1k 2k 5k 10k 20k up 5 · down 0 · of 10 // 0 500 1k 2k 5k 10k 20k 0.0 0.2 0.4 0.6 0.8 1.0 .3 < C ≤ .6 (19 prompts) correct fraction up 8 · down 2 · of 19 // 0 500 1k 2k 5k 10k up 7 · down 6 · of 19 // 0 500 1k 2k 5k 10k up 7 · down 1 · of 19 // 0 500 1k 2k 5k 10k 20k up 7 · down 3 · of 19 // 0 500 1k 2k 5k 10k 20k optimizer step (log; 0 = untrained) 0.0 0.2 0.4 0.6 0.8 1.0 C > .6 (167 prompts) correct fraction up 2 · down 3 · of 167 // 0 500 1k 2k 5k 10k optimizer step (log; 0 = untrained) up 5 · down 8 · of 167 // 0 500 1k 2k 5k 10k optimizer step (log; 0 = untrained) up 2 · down 1 · of 167 // 0 500 1k 2k 5k 10k 20k optimizer step (log; 0 = untrained) up 5 · down 2 · of 167 // One line per held-out prompt: fraction of 50 draws passing the tests, per checkpoint — rows = the untrained model's C(x) under T = 1; blue = up ≥ 0.2, red = down ≥ 0.2, grey = within ±0.2
Hover a line to highlight that prompt in all four arms; click it for the prompt itself, its library, its C(x) and Z(x), and its value at every checkpoint. Each line is one held-out prompt: the fraction of its 50 draws (5 runs × 10, T = 1) that pass the official tests, at each checkpoint; step 0 is the untrained model. Rows group prompts by the untrained model’s C(x) under the T = 1 sampler (the same rows in every column, so arm 2 is grouped by T = 1 here, unlike the panel above). Blue: up by at least 0.2 from step 0 to the last checkpoint; red: down by at least 0.2; grey: within ±0.2, which is about the noise of 50 draws. Two prompts (#512, Matplotlib; #823, Sklearn) fall to 0 at isolated checkpoints in every arm, including the untrained model, and recover at the next one: an all-or-nothing grading failure of that cell, not the model.

What we take from it. Correctness moves where the base model already has some purchase. For arm 3 the .1 < C ≤ .3 group goes 0.15 → 0.52 and .3 < C ≤ .6 0.45 → 0.61; 0 < C ≤ .1 0.05 → 0.17; the 78 prompts the base never solved in 100 draws 0.00 → 0.06 (15 of the 85 prompts at 0/50 at step 0 get at least one correct draw); the 167 easy prompts stay at 0.95 → 0.95. So the validity gains on low-Z prompts above mostly become programs that run but still fail the tests — the objective rewards running, not passing. Online MSC gets the same or larger lift in the middle groups within 14,000 steps (0.15 → 0.55, 0.45 → 0.49, 0.05 → 0.22) but pays for it on the easy prompts (0.95 → 0.95): 15 prompts lose at least 0.2 against 5 for arm 3, while 30 gain at least 0.2 (arm 3: 25) — the per-prompt face of its pass@50 decline. Arm 2’s groups are small apart from the extremes, and its large card C = 0 group moves 0.07 → 0.12.

Correctness on the training prompts

The correctness twin of the training-validity figure: the same 50 draws per training prompt per checkpoint, scored by the official DS-1000 tests, grouped by the untrained model’s C(x) under T = 1 (from the same 100 base rollouts). The training set has 176 prompts the base model never solved and 358 it solves more than 60% of the time.

Training-prompt correctness per checkpoint grouped by the untrained model's C(x): arm 3, online MSC, online IGUANA (C under T = 1)
Mean fraction of draws passing the official tests, per C(x) group, at each checkpoint; step 0 is the 100 base rollouts, later points 50 fresh draws per prompt.

What we take from it. Even on its own training prompts, arm 3 turns few never-solved prompts into solved ones: the C = 0 bin goes 0.00 → 0.05 (26 of 176 prompts get at least one passing draw out of 50), 0 < C ≤ .1 0.04 → 0.21, .1 < C ≤ .3 0.19 → 0.40, .3 < C ≤ .6 0.45 → 0.56; mean training correctness 0.528 → 0.583. MSC does slightly better on the same prompts (C = 0 0.00 → 0.08, .3 < C ≤ .6 0.45 → 0.63; mean 0.528 → 0.605) and, unlike held-out, does not lose the easy bin (0.96 → 0.97). IGUANA: C = 0 0.00 → 0.02, 0 < C ≤ .1 0.04 → 0.10, .3 < C ≤ .6 0.45 → 0.51; mean 0.528 → 0.553. Set against the validity figure above, the picture is the one the objective predicts: training makes the proposal’s programs run on the prompts it saw, and only a fraction of that becomes programs that pass, because the checker rewards running.

Why the prior must be the rollout sampler

What filtered-BC distills is π ∝ pgen·φ with pgen the sampler that produced the training rollouts, so that is the only prior under which “did training help?” is well posed. For arms 1–2 that sampler is truncated, which has a cost at evaluation time: the proposal is sampled untruncated at T = 1, and 26%–29% of its draws fall outside the card nucleus and receive weight zero. That is valid — the untruncated proposal covers the target’s support — but it is why their share of runs with no usable draw (17%–20%) sits above the rejection-sampling baseline’s 16.4%, and why their ESS is lower than arm 3’s. The untrained model evaluated with the same untruncated proposal under the card prior scores 0.572, within noise of rejection sampling (0.569), so the comparison is fair to the trained arms. The sound way to remove the waste is the 3B pipeline’s nucleus-masked proposal (sampling q restricted to the prior’s nucleus); at 30B we did not run it. Arm 3 has no such issue: its rollouts, its prior and its proposal all come from the same untruncated T = 1 family.

IGUANA and MSC training

Training

Two recipes, both plain-IS, both stable for the full run: rel clip (IGUANA with the use-time relative weight clip uw ≤ 10·Ẑ(x); raw weights feed the running statistics) and MSC. Training-side validity — the fraction of the M samples per visit that run without error, averaged per epoch over the training prompts:

Held-out results (M = 10, R = 5) — by epoch

Every cell is produced by the same IS evaluation pipeline, base model included. wacc is the weighted accuracy of the particle set (= pass@1 after one final resampling); pass@1 is the proposal sampled directly, no weights; validity rate is the fraction of samples with r = 1; pass@50 is the oracle coverage bound over all 50 draws.

weighted accuracy (posterior sample quality)

pass@1 of the proposal itself

validity rate of the proposal

pass@50 (coverage / oracle bound)

Shaded bands are 95% prompt-level bootstrap CIs (resample the eval prompts with replacement, 2000 draws). They price the dominant uncertainty — prompt difficulty varies enormously — but successive checkpoints of one arm share the same prompts and seed, so overlapping bands do NOT mean two checkpoints are indistinguishable; a paired comparison is the sharper test.

Reading: both recipes lift the proposal's own pass@1 far above base (0.195 → 0.28–0.33) by making validity common (0.49 → 0.63–0.74), while P(correct | valid) stays ≈ 0.40 for every arm — the gains are exactly the factor training can move. wacc crosses the skyline late in training. MSC concentrates faster (higher pass@1) but pays a coverage tax (pass@50 drifts below base); rel clip keeps coverage near base for longer.

Which prompts actually learn? Training validity by Z(x)

Training validity per epoch, grouped by base-model Z(x), for the DS-1000 rel-clip arm (no anchor, 15 epochs): there the learnable band gains +0.27 to +0.35 and nothing collapses — DS-1000 has almost no prompts below Z = 0.1 (4 of 591 at zero) and 200-token programs, so the repulsion steps are rare and light. The Z ≤ .1 bins are as inert there as on LCB.

DS-1000 rel-clip arm: training validity per epoch grouped by base-model Z(x)
arm (first → last epoch)Z = 00 < Z ≤ .1.1 < Z ≤ .3.3 < Z ≤ .6Z > .6
DS-1000 · rel-clip, no anchor0.00 → 0.00 (+0.00, n=4)0.06 → 0.13 (+0.07, n=15)0.21 → 0.56 (+0.35, n=42)0.45 → 0.73 (+0.27, n=81)0.75 → 0.90 (+0.15, n=206)

Improvement per prompt, by Z(x) — DS-1000 arms, failures included

The trajectories above are for the arms that survived. This collapses each arm to one number per bin — the mean over its prompts of (validity in the final window − validity in epoch 1), the final window being the last three epochs or whatever ran after epoch 1 — and puts the failed arm next to it. Bars are bootstrap 95% CIs over prompts; the table gives the share of prompts that moved by more than ±0.1.

Per-prompt improvement by Z(x) bin, DS-1000 arms, failures included
armZ = 00 < Z ≤ .1.1 < Z ≤ .3.3 < Z ≤ .6Z > .6
DS-1000

DS-1000. The rel-clip arm improves 80% of the prompts in the .1 < Z ≤ .3 band (+0.33) and 79% in .3 < Z ≤ .6 (+0.27), with essentially no prompt worse; the Z ≤ .1 prompts barely move (+0.06, 3 of 15 up) and the four at zero not at all. The no-weight-clip arm — the one bounding ablation that survived on DS-1000 — is a wash in every bin after four epochs (28–37% up, 28–36% down): without a bounded weight the update is too noisy to accumulate, even here.

Validity here is the training loop’s own verdict (no-error potential, graded ten at a time), so levels sit below the clean Z(x) that defines the bins; the within-arm change is the comparison. Bins are the 100-rollout offline Z(x).

Is amortization worth it? A compute-budget view

What we train is an amortized sampler: the potential is folded into the weights so that, at deployment, the model is sampled once, without the checker. The natural question is when that pays for itself. We compare two ways of serving N new prompts at the same held-out pass@1: (i) the trained proposal qT, one draw per prompt, no checker, plus the one-off training bill; (ii) the untrained model with the checker in the loop — best-of-K, return the first program that passes, the K-th returned unchecked if none does — with K the same for every prompt (a deployer does not know Z(x)). Break-even N* is where the two total costs cross.

Inputs shared by both figures. 254 DS-1000 held-out prompts. Per prompt, Z(x) = the base model's pass rate on the checker (the same Z(x) as everywhere on this page) and s(x) = P(correct | passes), both from a 50-draw base-model cell; per-prompt pass@1 of qT from the 50-draw checkpoint cells. Per-sample costs are medians over the 9,087 timing lines of the rel-clip training log (Iter … [gen …s, pot …s ∥ score …s], M-step …s) divided by M = 10: generation cgen = 0.10 s (one batched vLLM call for 10 programs on an RTX 4090 — throughput, not single-request latency), checker ccheck = 1.04 s (DS-1000 no-error potential, one sandboxed subprocess per program, 15 s timeout, up to 8 concurrent on 8 CPUs — amortized throughput; the mean is 1.29 s because of the timeout tail, and the median is the choice that favours the base+checker route), log-prob scoring 0.07 s, M-step 0.22 s; a training sample costs their sum, csample = 1.43 s. Deployment-side checks use the same potential and executor.

Figure 1 — accuracy vs cost per prompt

DS-1000 pass@1 versus expected cost per prompt: base model plus checker best-of-K frontier, and the trained proposal at several checkpoints with training amortized over 10^3, 10^4, 10^5 prompts
Grey: base + checker for K = 1 … 50, from the base model's single unchecked draw (0.205, 0.10 s) to the macro skyline 0.328. Colored marks: qT, one draw, no checker, at N = 103 (○), 104 (□), 105 (△).

What we take from it. (1) A mark left of the grey curve is cheaper than any checker-in-the-loop use of the base model at the same accuracy: at N = 105 every checkpoint is left of the curve, at N = 104 none is — so on DS-1000, with a one-second checker, amortization pays between 104 and 105 new prompts. (2) The curve flattens at 0.328, the macro skyline: no amount of checker-in-the-loop sampling of the base model gets past it, and q9000 at 0.284 has captured 87% of that headroom with one unchecked draw. (3) The steep part of the curve is K = 1 → 3: the first extra check buys a lot on this benchmark because the base model passes half the time; that is what makes the base+checker route cheap here and N* large. On a domain with lower Z(x) the curve would be flatter and further right, and the same qT would break even sooner.

Figure 2 — break-even N*(T)

How each line is measured. Every line is, per checkpoint T, a ratio N* = (training cost) / (saving per deployed prompt), with both sides counted in one currency. The saving is taken at Keq (defined below); write D̄ for the base's mean draws per prompt at Keq and C̄ for its mean checker calls (the prompt-averaged formulas of Figure 1). Worked numbers are for T = 3,000, where Keq = 1.80, D̄ = 1.42, C̄ = 0.86.

How N* is measured.

Break-even number of prompts as a function of training iterations, three cost currencies
Break-even N* per checkpoint, three cost currencies; log–log.

Data. Everything behind Figures 1 and 2: per_prompt.csv (254 held-out prompts × 50 draws: Z(x), s(x), base counts, and each checkpoint’s per-prompt pass@1), per_checkpoint.csv (Keq, savings and N* per checkpoint), timing.json, and the full bundle with the raw cells, the script and a README: breakeven_data.zip (56 KB).

What we take from it. (1) The level: a few ×104 new prompts — 25–80k depending on T — is the answer to “how big must N be.” (2) The shape is a U: the training bill grows linearly in T while the pass@1 gain saturates, so N* is undefined where the gain is within noise of zero (T = 250), lowest at T ≈ 500–2,000, and rising after. The minimum is the compute-optimal training length — on DS-1000 the first ~2,000 iterations buy pass@1 +0.04 for 8 GPU-h, the next 7,000 another +0.04 for 28 GPU-h — a different stopping rule from “until held-out pass@1 stops rising.” (3) The checker-only line is always the lowest and is what the answer tends to as the checker gets expensive relative to generation; the gap between the lines is the price of the M-step and scoring overhead in training. (4) The saving per prompt is ≈ 1/Z(x) − 1 for hard prompts and ≈ 0 for easy ones, so the compute case rests on exactly the low-Z(x) prompts that §4 below shows learn slowest.

T (iters)training GPU-hqT pass@1 (Δ vs base)Keqbase draws / checker calls per prompt at KeqN* checker callsN* wall-clock (95% CI)
2501.00.206 (+0.001)1.011.01 / 0.02115k152k —
5002.00.221 (+0.016)1.191.12 / 0.2718k24k [16k, 54k]
1,0004.00.225 (+0.020)1.251.15 / 0.3529k38k [26k, 73k]
2,0008.00.245 (+0.039)1.591.32 / 0.6929k38k [30k, 50k]
3,00011.90.254 (+0.049)1.801.42 / 0.8635k46k [37k, 59k]
6,00023.90.275 (+0.070)2.501.67 / 1.2847k61k [49k, 78k]
9,00035.80.284 (+0.079)2.971.81 / 1.4861k79k [61k, 107k]

Base: validity 0.504, pass@1 0.205. Read the T = 3,000 row as: the trained model reaches 0.254 with one unchecked draw; the base needs best-of-1.8 with the checker (≈ 1.42 draws and 0.9 checker calls per prompt) to match; training cost 12 GPU-h ÷ 0.94 s saved per prompt ≈ 46k prompts to break even.

Provisional. The checkpoint cells are the August rel-clip cells (the arm's weights were lost to a scratch purge; a re-training run is in progress and will replace these with per-prompt validity, which also enables the fixed-quality “keep the checker” comparison); the base cell is new (2026-09-09) and reproduces the August base numbers (pass@1 0.205 vs 0.195, per-prompt correctness r = 0.97). Generation cost is batched throughput, not single-request latency; both policies would shift together under latency accounting. With the checker unavailable at deployment the base+checker route does not exist and the comparison becomes Δpass@1 × N against training cost, which needs a price per solved prompt rather than a compute argument.

LiveCodeBench: a harder target

Same method, harder benchmark (Qwen3-4B, contest problems, ~1024-token programs). Two things change, and both matter.

1. The potential decides whether there is anything to learn

Training only amortizes a FIXED posterior into the proposal, so the most held-out pass@1 can gain is skyline − base pass@1. Measured on the held-out prompts:

base satisfies rskyline P(correct | r=1)base pass@1ceiling
runs-without-error r0.810.4250.418 +0.007
passes ALL public tests r0.5070.8220.414 +0.408

The no-error potential is nearly free on this benchmark — the base model already satisfies it 81% of the time, and conditioning on it barely moves correctness. Passing the public tests instead doubles correctness (0.414 → 0.822), so that is the target worth amortizing. (Public tests are a subset of the private suite: P(correct | any public test fails) = 0.000.) This is our own 20-sample measurement over the 214 held-out prompts; the evaluation cell independently reports 0.503. An earlier version of this page said 0.383, from a measurement that scored the potential with 10 graders in flight on 8 CPUs — the harness's per-test time limit is wall-clock, so concurrent graders starve each other and ~10 points of VALID programs are recorded as failures (measured: 0.427 serial vs 0.329 at 10-way, identical programs). Concurrency 2 is free; 4 and above is not.

Pooled vs macro: two ways to average the skyline, only one of which bounds our metric

The skyline in the table is pooled: Σx correct / Σx public-passing over all rollouts, i.e. a per-prompt skyline s(x) = P(correct | r = 1, x) averaged with weight nx ∝ Z(x) — prompts the base model already solves often count many times, prompts at Z(x) = 0 not at all. Held-out pass@1 (and wacc) is a macro average: one prompt, one vote. Under perfect amortization prompt x scores s(x), and a prompt with no public-passing program scores 0 (there is nothing to concentrate on, and correct ⊆ public-passing). So the ceiling our curves can reach is (1/N)·Σx s(x)·𝟙[Z(x) > 0], not the pooled ratio. On the same 214 held-out prompts (base model, 50 draws each, per-sample public and private counts):

strict potential, 214 held-out promptsskyline95% CIbase pass@1ceiling
pooled — Σ correct / Σ public-passing (rollout-weighted)0.808 [0.75, 0.86]0.412 +0.40
macro — mean over prompts of s(x), Z(x)=0 prompts count 00.556 [0.49, 0.62]0.412 +0.14
mean of s(x) over the 175 prompts with any public-passing draw0.680

The gap is not noise: s(x) and Z(x) are positively correlated (r = 0.47 over the 175 prompts with a passer) — the problems the base model passes often are also the ones with public tests strong enough that passing means correct — so the rollout-weighted average leans on exactly the prompts where the potential is a good proxy. Per prompt the skyline is bimodal (102 prompts at exactly 1, 39 at exactly 0, 34 in between; see the per-prompt chart in §2), and 39 of 214 have no public-passing draw at all. The honest headroom for held-out pass@1 under the strict potential is therefore about +0.14, not +0.41; the archive of 714 prompts gives the same macro ceiling (0.57). The rest of the apparent headroom belongs to the potential, not to training: ~18% of prompts admit only wrong programs through the public tests, and ~18% admit none.

2. How much is there to learn? The Z(x) distribution

Every visit moves the model, even one where no draw passes: a failed sample is not dropped, it gets coefficient (0 − Ẑ)/(M·Ẑ) = −1/M, a push away from what was just generated (see the dynamic-sampling note below). But only a sample with r(x,y) = 1 says where the posterior mass is — the −1/M steps are mean-zero by the score identity and carry no information about the target, only variance. So Z(x) — the prior's success rate — decides how often a visit is informative rather than pure repulsion, and hence in advance how much training can do. We measured it directly: 20 base-model samples for every one of the LiveCodeBench prompts, strict potential, same temperature and length limit as training, one grader at a time (see the note under the table above), and counted how many pass.

Why a failed draw is a step but not a signal

1. What one visit does. A visit draws M samples from the current proposal and steps along Σm wm ∇φ log qφ(y(m)|x) with wm = (uwm − b(x)) / (M·Ẑ(x)), uwm = p(y(m)|x)·r(x,y(m)) / q(y(m)|x), baseline b(x) = Ẑ(x). A passing draw has uw = p/q > 0: its coefficient is positive if the program is under-produced relative to its posterior mass and negative if over-produced — sign and size depend on the sample. A failing draw has uw = 0 exactly, so w = (0 − Ẑ)/(M·Ẑ) = −1/M whatever Ẑ is: a fixed-size push away from the program just generated, carrying nothing about the sample except that it failed. This is the REINFORCE estimator with a baseline (Williams 1992; variance analysis in Greensmith, Bartlett & Baxter 2004), with the reward replaced by an importance weight and the baseline by a normalizer estimate that is NOT computed from the current batch. That last choice is what makes an all-fail visit differ from GRPO: GRPO (Shao et al. 2024) and RLOO (Kool et al. 2019, Ahmadian et al. 2024) centre on the batch mean, so a group that all scores 0 has every advantage exactly 0 and the batch is skipped; our running Ẑ(x) is not the batch mean, so the same batch moves the model.

2. Why the repulsion is signal-free even though it is a real step. Take the expectation of the all-fail update over the draws: Eq[−(1/M)·Σm ∇ log q(y(m))] = −Eq[∇ log q(y)] = −∇ Σy q(y) = −∇1 = 0. That is the score identity — the same fact that makes any baseline legal (Sutton et al. 2000). So the −1/M steps have zero mean: on average they move q neither toward nor away from the posterior; each individual step is nonzero, so what they add is variance. A passing draw's term (p·r/q)·∇ log q has expectation Z·Eπ[∇ log q] — the gradient of the objective, pointing at the posterior. Equivalently: the exact gradient of KL(π‖q) is Eπ[∇ log q], and failed programs have π(y) = 0, so they contribute nothing to the true gradient; they appear in the estimator only through the baseline, as noise that averages out.

3. Why this is a Z(x) story. Write v(x) for the current proposal's pass rate on prompt x — at initialization q = p, so v(x) = Z(x). A visit of M draws contains at least one informative term with probability 1 − (1−v(x))M: for M = 10 that is 99.9% at v = 0.5, 65% at 0.1, 40% at 0.05, 10% at 0.01, never at 0. So Z(x) sets the ratio of informative visits to pure-repulsion visits a prompt starts with, and every informative visit is what moves v(x) up from there. A Z(x) = 0.01 prompt gets one informative visit per ~10 epochs and nine random kicks in between — which is the shape of the dark lines in §4: they creep up only under the anchor and only over many epochs, because the rare positive terms must accumulate faster than the random ones scatter them. Chatterjee & Diaconis (2018) give the same threshold from the importance-sampling side: for an indicator potential under the prior, KL(π‖p) = log 1/Z, so the sample size at which IS starts to work is M ≈ 1/Z.

4. Why mean-zero is not the same as harmless. (a) Variance scales like 1/(M·Z): the informative term appears with probability ~M·Z per visit and, when it does, carries weight ~1/Z; small Z means rare large positive steps against frequent small negative ones. Anything that truncates the rare positive step (weight clipping, a gradient-norm clip that binds on it) or lets the small negatives compound (no anchor) tips the balance — the reason gradient clipping hurt on LCB. (b) The identity holds one step at a time, for the current q. Across steps the repulsion is not independent noise: a softmax update that reinforces or repels sampled outcomes drains logit mass from every UNSAMPLED outcome by −η·q(y') per step, so a region the proposal stops sampling keeps shrinking with no term left to restore it (the absorbing-fixed-point mechanism on the failure-modes page). The same covariance structure is the entropy-collapse mechanism of Cui et al. 2025, ΔH ∝ −Cov(log q, advantage), and matches the observation of Yue et al. 2025 that on-policy RLVR sharpens within the base model's support rather than extending it. The KL anchor is the one term whose restoring force GROWS as a coordinate is abandoned, which is why it is what separates the two columns of §4.

5. What “informative” does and does not imply. It does not mean Z(x) = 0 prompts are inert: they move the model on every visit, which is why 31% of LCB prompts at Z = 0 were enough to sink every anchor-free arm while DS-1000's four such prompts did nothing visible. It does mean there is nothing to LEARN from them until some draw passes — no number of repulsion steps ever points at π. Hence the fixes are the ones below and in §4: skip empty visits (the dynamic sampling of DAPO, bias-free here because the skipped term was mean-zero), change the potential so that Z > 0 (tiered credit), or change the proposal so that samples pass (hints, prompted proposals). One caution when reading the RL literature on negative samples: Zhu et al. 2025 find that training on WRONG answers alone (“negative sample reinforcement”) works surprisingly well for short-answer reasoning, because pushing mass off a wrong answer redistributes it over a small set of alternatives that includes the right one. Over 1,000-token programs the redistributed mass has no such place to go — it lands on whatever q has not sampled, which after some drift is the empty string or a repeated token, exactly the attractor the collapsed arms reach. Likewise Dr. GRPO (Liu et al. 2025) shows that per-response length normalization biases which samples dominate an update; our coefficient is per sequence, so length enters instead through the heavier tail of the log-weight (§3).

The mass is at the two ends. 329 prompts (31%) sit at exactly Z(x) = 0: not one passing sample in 20 draws. Another 464 prompts (44%) already pass at least half the time, where there is little left to gain. Only the 262 prompts (25%) in the middle band are where a better proposal has both something to imitate and room to improve. Mean Z(x) = 0.448.

The split tracks difficulty almost perfectly, which is the sanity check that Z(x) is measuring what we think:

promptsmean Z(x)at Z(x) = 0
easy3220.8138%
medium3830.39031%
hard3500.17553%

The Z(x) = 0 mass is not merely inert — it actively pushes the wrong way. With the capped baseline b(x) = min(Ẑ(x), 1), a visit where no particle is valid gives every sample the coefficient (0 − Ẑ)/(M·Ẑ) = −1/M: the update is −(1/M) ∑i ∇ log qφ(yi|x), uniform unlearning of whatever the model just produced, independent of Ẑ. It is mean-zero in expectation — the rare visit that does find a valid sample carries a compensating coefficient of order 1/(MZ) — but the realised trajectory is a long run of small downward steps punctuated by one very large upward one, with per-prompt variance scaling like 1/(MZ). That is the mechanism behind the degenerate short-output attractor (the cheapest escape from a set of continuations being pushed down is to emit almost nothing) and the reason gradient clipping made matters worse: clipping truncates the rare large positive step while leaving the frequent small negative ones intact, so it destroys exactly the cancellation the baseline provides.

It also prices the particle budget. Counting visits expected to yield at most one valid particle — no weight spread, so nothing to rank — gives 43% at M = 10, 38% at M = 20, and 33% at M = 50. Five times the compute buys a few points, because the bulk of the deficit sits at exactly Z(x) = 0, where no finite budget helps: an indicator potential under the prior needs M ≳ 1/Z(x) particles (Chatterjee–Diaconis, since KL(π ‖ p) = log 1/Z(x) here). The productive move is the opposite one: skip a prompt on visits where no particle is valid — bias-free, since that visit's expected contribution is zero — and spend the generations on the middle band instead.

Cross-check: 100 rollouts per prompt from the verdicts archive

An existing archive (qwen3-4b-nothink, T = 0.6) already holds 100 rollouts for 714 of these prompts with the potential pre-computed per rollout, so the same quantity can be read off at 0.01 resolution instead of 0.05. It is an independent pipeline — different grading run, and it sampled with top_p = 0.8, top_k = 20 where we use top_p = 1 — and it lands in the same place: mean Z 0.510 against our 0.516 on the same prompts, per-prompt r = 0.979. (The truncated sampler turns out to matter far less than we assumed.) It also answers the resolution worry directly: of the 175 prompts our 20 draws put at exactly zero, 77% are still exactly zero over 100 sharper draws, 20% land in (0, 0.05], and only 3% rise above 0.05. The dead mass is real, not a sampling artefact.

The archive also lets us price partial credit on identical rollouts. Scoring each rollout by the fraction of public tests it passes rather than all-or-nothing shrinks the no-signal region from 21% of prompts to 7%:

potential r(x,y)mean Z(x)at Z(x) = 0Z(x) ≥ 0.5
passes ALL public tests (strict)0.510 21%50%
fraction of public tests passed (tiered)0.616 7% 61%

That is the quantitative case for the tiered potential: it converts two thirds of the prompts that generate nothing but the −1/M unlearning step into prompts with a gradient.

Per-prompt skyline — P(correct | passes all public tests)

Because the archive records full-suite correctness alongside the potential, the skyline can be resolved per prompt instead of pooled. Two sanity results first: no rollout is ever correct while failing a public test (0 of 71,400), confirming the public tests are a genuine subset; and pooled P(correct | r = 1) = 0.849, matching the 0.822 in the table above. Per prompt it is far from uniform:

For 20% of the 561 prompts that have any valid rollout at all, the strict potential admits only WRONG programs — its skyline is exactly zero, so amortizing that posterior cannot produce a correct program there no matter how well training goes. At the other end, 61% have skyline exactly 1: every public-test-passing program is correct. The pooled 0.85 is an average over these two populations, not a typical prompt, which is why the aggregate ceiling reads more optimistic than the per-prompt picture warrants. Averaged the way our metric averages — one prompt, one vote, prompts with no passing program counting 0 — the same archive gives a macro ceiling of 0.57, and the held-out set 0.56 (§1).

3. Long sequences destabilize training

The log-weight is a per-token sum, so its spread grows with length: LCB's ~1024-token programs make the weights far heavier-tailed than DS-1000's ~200. Rare huge weights then enter the running Ẑ(x) and push it above 1 — impossible, since Z(x) is a probability. With b(x) = Ẑ(x) above every honest weight, every sample gets a negative coefficient: the model is repelled from its own valid programs.

training validity, strict potential — iteration 0 is the base model

Every arm without those safeguards died this way — the earliest by iteration ~650, and never recovering. Two fixes follow directly: cap the baseline at the ceiling, b(x) = min(Ẑ(x), 1) (bias-free — any baseline is mean-zero by the score identity), and bound each fold's influence on the statistics, keeping the raw estimate for reporting. The current arm adds a KL anchor to the prior on top, trains on the strict potential. The curve above is that arm: validity climbing past 3,000 iterations, with the gradient-side Ẑ(x) pinned near the truth (~0.5) while the raw estimate inflated to 6.6·108 harmlessly.

4. Which prompts actually learn? Training validity by Z(x), with and without the anchor

The histogram above says where the signal could be; this asks where training actually collected it. Every training visit records how many of the M = 10 draws passed the potential, and the checkpoints record which prompt each iteration visited, so we can follow each prompt's own validity across epochs and group prompts by their base-model Z(x) (the same 20-draw, one-grader-at-a-time measurement as the histogram). Two splits, each with an anchor-free rel-clip arm and an anchored β = 0.1 arm; otherwise the same recipe.

LiveCodeBench: training validity per epoch, grouped by base-model Z(x), for four arms: no anchor vs anchored beta=0.1 on the 500-prompt and 781-prompt splits
Lines are the mean validity of a bin's prompts within each epoch; darker = harder. Left column: anchor-free rel-clip arms — the full-split one froze from ~iteration 1,200 (denominator inflation), the big-split one collapsed to empty output in epoch 2. Right column: the same recipe with the KL anchor.

Without the anchor nothing below Z ≈ 0.3 ever moves. On the full split the two hardest bins gain +0.02 in eight epochs while the run freezes; on the big split every bin, easy ones included, is at zero by epoch 3. With the anchor every bin moves, and the middle bins move most: .1 < Z ≤ .3 goes 0.15 → 0.24 and .3 < Z ≤ .6 0.33 → 0.48 on the full split (13 epochs); 0.13 → 0.26 and 0.35 → 0.49 on the big split (10 epochs). The two hardest bins also rise — Z = 0 from 0.01 to 0.05, and 0 < Z ≤ .1 to 0.07–0.10 — slowly, with no prompt getting worse, and they are the only bins still climbing at the end while Z > .6 saturated by epoch 2. For a prompt at Z = 0.05 a visit contains a passing draw about every second epoch; the anchor's support floor is what keeps the −1/M repulsion of the empty visits from undoing those rare positive steps in between.

arm (first → last epoch)Z = 00 < Z ≤ .1.1 < Z ≤ .3.3 < Z ≤ .6Z > .6
LCB · no anchor, full split0.01 → 0.03 (+0.02, n=123)0.04 → 0.06 (+0.02, n=28)0.12 → 0.16 (+0.04, n=55)0.29 → 0.32 (+0.03, n=52)0.75 → 0.79 (+0.04, n=242)
LCB · anchored β=0.1, full split0.01 → 0.06 (+0.05, n=123)0.02 → 0.07 (+0.05, n=28)0.15 → 0.24 (+0.09, n=55)0.32 → 0.48 (+0.16, n=52)0.76 → 0.78 (+0.02, n=242)
LCB · no anchor, big split0.01 → 0.00 (-0.01, n=259)0.03 → 0.00 (-0.03, n=45)0.15 → 0.00 (-0.15, n=92)0.31 → 0.00 (-0.31, n=82)0.75 → 0.00 (-0.75, n=303)
LCB · anchored β=0.1, big split0.01 → 0.03 (+0.02, n=259)0.04 → 0.10 (+0.06, n=45)0.13 → 0.26 (+0.13, n=92)0.35 → 0.49 (+0.14, n=82)0.80 → 0.89 (+0.09, n=303)

Reading notes. (i) The y-axis is the training loop's own verdicts, which were graded ten at a time and so sit below the clean Z(x) that defines the bins (a .3 < Z ≤ .6 prompt shows ~0.3 at epoch 1); the within-bin trend is the quantity to read, not the level. (ii) On LCB, prompt identity per iteration comes from the shuffled epoch order saved in each checkpoint (epoch-1 validity correlates 0.91–0.94 with measured Z(x)); on DS-1000 it comes from the 200-character prompt prefix, which is unambiguous for 348 of 591 prompts (DS-1000 has near-duplicate problem variants). (iii) “Epoch” = one visit per training prompt; a trailing epoch is shown only when at least half of it ran.

Improvement per prompt, by Z(x) — LiveCodeBench arms, failures included

The trajectories above are for the arms that survived. This collapses each arm to one number per bin — the mean over its prompts of (validity in the final window − validity in epoch 1), the final window being the last three epochs or whatever ran after epoch 1 — and puts the failed arms next to them. Bars are bootstrap 95% CIs over prompts; the table gives the share of prompts that moved by more than ±0.1.

Per-prompt improvement by Z(x) bin, LiveCodeBench arms, failures included
armZ = 00 < Z ≤ .1.1 < Z ≤ .3.3 < Z ≤ .6Z > .6
rel-clip 10, no anchor — 15 epochs, stable+0.00 0↑ 0↓ · n=4+0.06 20↑ 0↓ · n=15+0.33 81↑ 2↓ · n=42+0.27 79↑ 1↓ · n=81+0.16 53↑ 1↓ · n=206
no weight clip, grad-clip 1, no anchor — stopped after 4 epochs, volatile-0.03 0↑ 0↓ · n=4-0.03 7↑ 7↓ · n=15-0.02 29↑ 29↓ · n=42+0.03 37↑ 28↓ · n=81-0.05 18↑ 37↓ · n=206
LCB · anchored
anchored β=0.1, strict φ, full split — 13 epochs, stable+0.04 12↑ 0↓ · n=123+0.06 18↑ 0↓ · n=28+0.12 53↑ 11↓ · n=55+0.18 65↑ 4↓ · n=52+0.09 40↑ 8↓ · n=242
anchored β=0.1, strict φ, big split — 10 epochs, stable (chain ended)+0.02 7↑ 0↓ · n=259+0.07 29↑ 2↓ · n=45+0.13 51↑ 8↓ · n=92+0.13 50↑ 12↓ · n=82+0.06 33↑ 12↓ · n=303
LCB · anchor-free
rel-clip+cap, strict φ — FROZE ~iter 1200 (7.7 ep)+0.01 5↑ 0↓ · n=123-0.01 4↑ 7↓ · n=28+0.03 25↑ 13↓ · n=55+0.04 31↑ 19↓ · n=52+0.12 47↑ 4↓ · n=242
rel-clip+fold-clip, strict φ, big split — COLLAPSED epoch 2 (6.7 ep)-0.01 0↑ 2↓ · n=259-0.03 0↑ 7↓ · n=45-0.15 0↑ 46↓ · n=92-0.31 0↑ 74↓ · n=82-0.75 0↑ 99↓ · n=303
rel-clip+fold-clip, tiered φ — COLLAPSED ~iter 1700 (6 ep)-0.01 2↑ 2↓ · n=123-0.03 4↑ 7↓ · n=28-0.13 2↑ 35↓ · n=55-0.33 2↑ 77↓ · n=52-0.82 0↑ 100↓ · n=242
rel-clip+fold-clip+grad-clip, tiered φ — COLLAPSED ~iter 500 (7.7 ep)-0.02 0↑ 4↓ · n=123-0.07 0↑ 14↓ · n=28-0.14 0↑ 42↓ · n=55-0.34 0↑ 65↓ · n=52-0.75 0↑ 82↓ · n=242
rel-clip, noerror φ — COLLAPSED iter ~650 (1.4 ep; last = partial epoch 2)-0.01 5↑ 2↓ · n=43+0.09 14↑ 0↓ · n=7-0.03 11↑ 11↓ · n=18-0.04 39↑ 39↓ · n=18-0.23 9↑ 39↓ · n=89
rel-clip+cap, noerror φ — survived 7 ep, no held-out gain+0.01 7↑ 1↓ · n=123-0.01 4↑ 7↓ · n=28+0.05 38↑ 15↓ · n=55+0.06 40↑ 15↓ · n=52+0.04 18↑ 7↓ · n=242

LCB, anchored. Both anchored arms are positive in every bin with no bin's CI crossing zero. The gain peaks in the middle (.1 < Z ≤ .6: +0.12 to +0.18, half to two thirds of prompts up, 3–12% down) and is smallest but still positive at the ends: Z = 0 +0.02/+0.04 with zero prompts worse, and Z > .6 +0.06/+0.09, where there was little left to gain. That is the anchor doing what the theory says it should: the hardest prompts are not harmed by the repulsion steps and slowly bank their rare positive ones.

LCB, anchor-free. The picture inverts. The three arms that collapsed lose most on the prompts that had the most: Z > .6 falls by 0.75–0.82 (81–100% of those prompts down), .3 < Z ≤ .6 by ~0.33, .1 < Z ≤ .3 by ~0.14, while the Z ≤ .1 prompts show almost nothing because they had almost nothing. The gradient-norm clip does not change this (tiered with and without it are indistinguishable). The two arms that did not die did not learn either: the frozen strict arm is flat below Z = .6 (+0.12 only at the top, before the freeze), and the noerror “cure” arm gains +0.04 to +0.06 in the middle bins and nothing in the two hardest (3–7% of those prompts up, as many down). The 650-iteration arm has only a partial second epoch, so its CIs are wide; its one visible effect is the top bin already falling (−0.23). Across all six, no anchor-free recipe gained more than +0.06 in any bin below Z = .6 on LiveCodeBench, and none moved the two hardest bins at all.

Validity here is the training loop's own verdict, and for the noerror and tiered arms it means “runs without error” rather than “passes the public tests”, so their levels are not comparable to the strict arms' — the within-arm change is. Bins are the clean 20-draw Z(x).

5. MSC on LiveCodeBench — 9.6 epochs, no movement

MSC (conditional-IS): each visit pools the iteration's valid draws with the prompt's retained reference, samples one winner from softmax(log σ̃ − log q), and takes a single-sample score-climbing MLE step on it. The weights are spent in the draw, so the M-step is unweighted — one sequence per prompt. Proposal = the nothink card (T = 0.6, top_p = 0.80, top_k = 20) applied inside vLLM, i.e. q truncated by its own nucleus. Same split, lr, LoRA and potential as the IGUANA arm (781 train / 60 val / 214 held-out; strict public tests), evaluated on the same card with grading at concurrency 2.

Held-out pass@1 across training — 16 checkpoints, 214 prompts × 20 card-sampled draws each

Flat over 7,500 iterations — about 9.6 epochs. pass@1 averages 0.4145 across the sixteen checkpoints against a base of 0.4136; the spread (0.406–0.421, sd 0.0037) is checkpoint noise, and a straight line through the series slopes −0.0006 per 1000 iterations. Validity is equally flat (~0.511). The small rise at iterations 500–1000 sits inside that spread and does not persist. Paired on the same prompts, 0 → 1000 gives +0.005 [−0.005, +0.017] and 500 → 1000 gives −0.0005 [−0.011, +0.010]; the second interval matters because from iteration 782 a prompt can be revisited, so MSC's retained reference finally enters the candidate set and the method stops being filtered self-imitation. It ran for thousands of iterations after that and changed nothing.

Two reasons the null is trustworthy rather than a broken pipeline. The iteration-0 cell returns pass@1 0.4154 against 0.4136 measured independently from the rollout archive — agreement to 0.002 across two separate pipelines. And the comparison is paired on the same prompts, which matters on LCB where per-prompt difficulty dominates: unpaired, the same difference carries roughly three times the error bar.

Training validity — fraction of the 10 draws passing all public tests (50-iteration mean)

Flat: 0.400 over the first 100 iterations, 0.382 over the last 200, mean 0.406 across 1039. The proposal is not getting better at satisfying the potential on the prompts it trains on, which is consistent with the held-out cells above. Note these draws are card-sampled (top_p = 0.80, top_k = 20), so the level is not comparable to the full-support arms in §3 — only the shape within this arm is. 41.5% of iterations produce zero valid particles (431 of 1039), close to the 43.3% the Z(x) distribution predicts at M = 10. On those visits MSC has no fresh candidate at all and can only re-select the retained reference, which is exactly the case the CIS kernel exists to cover — but it means roughly two visits in five carry no new information.

What it does not yet show: 500 iterations is two thirds of one epoch (781 prompts, one prompt per iteration), and MSC's retained references only start contributing once a prompt is revisited — which first happens at iteration 782. The arm is now past that boundary and the iteration-1000 cell above covers it. The measured cost of the self-truncated proposal is meanwhile small: the coverage gap — posterior mass inside the prior's nucleus that q's own nucleus can no longer reach — grows from 0.0000 at initialisation to 0.0014 by iteration 700, under 0.2% of the nucleus mass per position.

Two lessons from RL: token-level loss and dynamic sampling

Our instabilities have named counterparts in the RL-for-LLM literature. Two of them (DAPO, Dr. GRPO) map onto our setting closely enough to borrow both the diagnosis and the fix.

1. Token-level loss — length changes a token's influence

In GRPO. The loss averages over the tokens of each response, then over responses: per-token weight (1/G)·(1/|y|). So every response gets equal total say regardless of length, and a token inside a long response is diluted. A rambling wrong answer can barely be penalised per token — the runaway-length pathology. DAPO sums over all tokens with ONE denominator; Dr. GRPO shows removing the per-response length normalization is what recovers the unbiased objective.

two wrong responses to one promptper-token weight, GRPOper-token weight, token-levelshare of the update
short: 10 tokens½ · 1/10 = 0.0501/1010 = 0.0009950% → 1%
rambling: 1000 tokens½ · 1/1000 = 0.00051/1010 = 0.0009950% → 99%

In our case the dilution does not happen — our coefficient is per SEQUENCE and the score ∇φ log qφ(y|x) is already a sum over tokens, so a longer sample contributes more terms, not fewer. But the same length axis bites us from the other side: the log-weight is also a per-token sum, so its spread grows like √L / T — and that is what makes the weights unusable on long programs.

same per-token disagreement between p and qφlengthlog-weightweight uw
0.006 nats/token — DS-10002001.23.4
0.006 nats/token — LiveCodeBench10246.4610

That 610 is not hypothetical: it is a weight we logged, and folding it raw into Ẑ(x) is what pushed the estimate above 1 and inverted every sign. Where DAPO normalizes the LOSS by length, the open question for us is whether to normalize the LOG-WEIGHT by length.

2. Dynamic sampling — batches with no signal

In GRPO. The advantage is the group-centred reward, so if all responses to a prompt score the same — all correct or all wrong — every advantage is exactly 0 and the whole rollout is wasted. As the model improves, more prompts land in the all-correct bucket, so the EFFECTIVE batch shrinks over training. DAPO oversamples and keeps only prompts with mixed outcomes.

Ours is worse than wasted. Our baseline is the running Ẑ(x), not the batch mean, so an all-invalid visit does not vanish: every particle gets (0 − Ẑ)/(M·Ẑ) = −1/M, a uniform push away from everything just generated, with no contrast between the samples. GRPO wastes the batch; we move the model with a signal that carries no quality information.

M = 10, one valid particle with uw = 0.6baseline b = Ẑ = 0.5 (ours)baseline b = batch mean = 0.06
the 1 valid particle+0.02+0.108
each of the 9 invalid−0.10−0.012
sum over the batch−0.88 (net repulsion)0 (purely contrastive)
all 10 invalid−1.000 — the visit self-cancels

How often we are in that regime is set by the particle budget against the prompt's difficulty, and it is binomial: a visit is low-signal when ≤1 of M draws is valid.

Z(x) = 0.50 → 1%Z(x) = 0.20 → 38% Z(x) = 0.10 → 74%Z(x) = 0.05 → 91%

So on a prompt the base model solves 5% of the time, nine visits in ten teach nothing but repulsion at M = 10 — and our own theory section predicts exactly this (ESS ≈ M·Z(x), so hard prompts collapse first while aggregate metrics still look healthy). We now log the count as lowsig. Two fixes follow: skip such visits (DAPO's), or switch to the batch-mean baseline, which makes them self-cancel for free.