Three failure modes of IGUANA training — isolated, with their fixes

Dotted line = iter 591 (epoch 2 = every prompt's second visit, K=1). The unseeded run breaks its climb and runs with ~3× the gradient noise from there on; the seeded run sails through. Contained (not catastrophic) because the other two fixes are on.

Red validity starts near 1.0 only because it counts post-resampling survivors (inflated); the model beneath is being pushed away from every sample — no climb phase, monotone erosion, dead by iter ~180. Switch the metric to Loss: negative from iteration 1 (the signature).

Climb-then-crater: concentrating on covered good sequences first helps (validity 0.1 → 0.85 by ~iter 100) — but the estimator never sees the mass q stops covering, so the support ratchet runs one way, into a near-deterministic degenerate proposal (0 by ~iter 160) with the gradient norm running away. The anchored reference stays flat indefinitely. (The objective itself is mean-seeking — this is estimator failure, not objective preference; see the Mode-3 card above.)

Plain-sampling eval (ess=0) — the proposal-quality metric. Both stable arms improve (validity +30% rel., pass@1 +0.05–0.06 behind these numbers); the SMC-eval (ess=0.9) curve is flat at the posterior value as expected (shown dashed where complete). Early: iter 2000 ≈ 3.4 epochs of 591 prompts — the 10-prompt probes needed ~100 epochs to peak. Frozen ≈ seeded, within 18-prompt noise. Red dash: the skyline on these 18 prompts — E_σ[correct] = 0.244 [0.148, 0.347].

The main-table numbers. Left/top: WACC (macro mean, prompt-bootstrap CIs) against the skyline — the posterior ceiling E_σ[correct] = 0.323 [0.284, 0.363] on these exact 254 prompts (macro; pooled ratio-of-sums variant: 0.377 [0.331, 0.422]; 59/254 prompts have zero correct-given-valid base samples). Base already sits at 0.302: the no_error target leaves ~2 points of total headroom, and both trained arms land inside the skyline CI — the target, not the optimizer, binds. Bottom: pass@K of the proposal itself (ess=0 particles are iid draws; unbiased estimator, identical prompts for all arms). pass@1 rises 0.192 → 0.266 (+39% rel.) at ~3.4 epochs, β=0.02 > β=0.1 at every K, The deep-k picture (two independently-seeded waves pooled, 150 draws/prompt, 222 common prompts — one shard refilling): EARLY checkpoints (@2000) dominate at small k (pass@1 +43% rel.) but converge to a statistical tie with base by k≈100–150. The PEAK checkpoint changes the story: β=0.1 @10000 dominates base at EVERY k — pass@50 0.715 vs 0.687, pass@100 0.766 vs 0.739, pass@150 0.793 vs 0.770 (+2.3 pts ≈ 5 prompts). Contrast with RLVR (Yue et al. 2504.13837, significant cross-under at large k on code): at its peak, stabilized inclusive-KL training shows coverage GAIN through the deepest k measured — concentration and coverage, not a trade. NEW: the CORRECTNESS-potential arm peaks at WACC 0.349 / pass@1 0.323 at iter 3000 — the FIRST arm above the no_error skyline, and the best pass@1 measured (+68% rel. over base) — then declines (0.319/0.247 @7000) as the training-batch solve rate collapses; see the per-epoch chart below. Recipe: identical to the β=0.02 no_error arm — KL anchor β=0.02 to the base model, ess=0, tempered, frozen Z — plus the terminal reward (real DS-1000 test execution lands multiplicatively on the weights, so the target is p·φ_noerr·correct) and Z re-measured for THIS target (frozen at P(valid AND correct), mean 0.195 — freezing the no_error rates would miscalibrate the baseline b(x)=Z(x) by ~2.5×). The anchor matters twice here: correctness is a much sparser reward (Ẑ≈0.2 vs ~0.5), and sparser targets have higher stability thresholds (the LCB lesson) — β=0.02 holds. Its own skyline is 1.0, not 0.323: the richer target raises the ceiling, and the probe-scale degradation was the 10-prompt memorization artifact. Trajectory: complete 254-prompt cells at iters 1000–7000 are plotted; @4000/@6000 cells still partial. The red dash on the pass@1 group marks the skyline: the pass@1 of a proposal that exactly matches the posterior; both trained arms are below it here (0.266 vs 0.323), i.e. concentration has not yet reached posterior sharpness at 3.4 epochs.

Both anchor strengths show the same rise–peak–decline shape; β sets the timing and the height. β=0.02 peaks at ~iter 2000 (pass@1 0.266) then recedes; β=0.1 needs 10,000 iterations to reach the same pass@1 but at a higher peak everywhere else — WACC 0.324 (touching the skyline 0.323) and pass@50 0.717 (+4.8 over base, a genuine support gain) — before receding by 19,500. Practical rule: weak anchor if compute-limited, strong anchor + checkpoint selection if not. Frozen-Z at iter 2000 posts the highest pass@1 of any measured cell (0.283, on 222/254 prompts — one shard pending) before declining by 3500 — same shape, possibly a slightly higher/earlier peak than seeded. Curves now show the full peak-bracketing resolution (β=0.02 every 500 iters around its peak; β=0.1 every 2000–3000 across its rise and decline).

Training against the CORRECTNESS potential (terminal test execution; skyline = 1 by construction) at 10-prompt scale briefly matches the no_error arm (0.300 @500) then degrades to the held-out asymptote (~0.24) and stays there — while the no_error arm (β=0.02) holds its 0.30 plateau to 12,500. Interpretation: with 10 prompts and a sparse reward (Ẑ≈0.2), the correctness signal teaches the solutions to those ten problems (memorization) instead of a transferable skill. The full-split correctness arm (591 prompts, running) is the decisive test — the memorization shortcut doesn't exist at that scale.

Below: the FULL-SPLIT correctness arm's training-batch solve rate (windowed mean of #particles passing valid∧correct per batch; dashed line = the frozen-Z prior rate 0.195, i.e. the base model's solve rate on the training prompts). Because the terminal reward measures correctness directly, this curve aggregates each FULL EPOCH (591 iterations = every training prompt exactly once), so points are directly comparable — no prompt-composition noise. The full β=0.02 arc is now a rise–collapse–RECOVERY: 0.161 → 0.188 (epoch 7, iters ~3500–4100, matching the held-out peak WACC 0.349 @3000), then a full collapse to 0.000 for four consecutive epochs (10–13, iters ~5300–7700 — the held-out decline to 0.319 @7000 is this collapse seen from outside), then a spontaneous recovery back to 0.169 by epoch 17. The recovery is the anchor doing its job: β=0.02 keeps the collapsed proposal within reach of the prior, so coverage regrows — unanchored arms never come back. The β=0.1 twin runs lower and slower, had a shallow dip at epochs 3–4, and is climbing steadily: 0.079 → 0.105 at epoch 10, still rising. (A prior-rate reference line is omitted for now: the rollout-measured prior (0.195) used 400-token generations vs the run\u2019s 200, so it is not directly comparable — a truncation-matched prior can be added later.)

All ess=0 runs are plain importance sampling, so a new runner replaces the genlm SMC E-step with one direct vLLM generate + two scoring forwards (uw = φ·p_T/q_T, adapter on/off); the IGUANA weight math is the same shared code (~13s/iter vs ~35s; an epoch ≈ 2.1h). On it, three arms differing ONLY in the stabilization mechanism: b002-IS — the production anchored recipe verbatim (uw-clip 1.0, w-cap ±0.5, grad-clip, KL anchor β=0.02); REL-CLIP — IGUANA with Beta-shrinkage Ẑ(x) (μ₀=0.5965 = rollout grand mean; n₀=100, m₀=50) + RELATIVE use-time clip uw ≤ 10·Ẑ(x) (unclipped uw feed the running Ẑ; bounds the centered weights to [−0.1, 0.9] by construction), NO absolute clip, NO w-cap, NO grad clip, NO ANCHOR; MSC — Markovian Score Climbing (CIS kernel): weights SELECT one training sample per prompt, never scale a gradient — no normalizer, no baseline, no clips, NO ANCHOR. Below: per-EPOCH training validity (591 iters = each prompt once). Both anchor-free arms climb monotonically through 9–10 epochs — an order of magnitude past every unanchored collapse we measured (probe @~160, LCB @~600) — and MSC runs at or above the anchored control.

Held-out (254 prompts, plain-IS eval backend; its base cell reproduces the genlm backend within noise: wacc 0.322 vs 0.302, pass@k within ~1.6pt). MSC's trajectory so far: pass@1 rises to 0.280 @3000 (+44% rel. over the same-backend base 0.195) with WACC pinned at the no_error ceiling (~0.31) — anchor-free concentration gains matching the anchored recipe. The honest counterpoint: pass@50 drifts DOWN 0.685 → 0.665 → 0.646 → 0.634 (@0→1000→2000→3000) — no collapse, but a steady coverage tax of ~1.5pt per 1000 iters at k=50 that the anchored arm did not pay (its @2000 pass@50 stayed at base level). All REL-CLIP cells (@250–@9000) have landed and are in the table below.

Long-horizon update (@6000–@12000) — every arm concentrates, every arm pays coverage, and the ablation verdict is in. MSC keeps climbing to pass@1 0.332 @12000 (+70% rel.) with wacc finally above the skyline (0.334) — but its oracle coverage keeps eroding (pass@50 0.685 base → 0.567), the steepest coverage tax of any surviving arm. The rel-clip arm's early coverage advantage was a PHASE: pass@50 peaked at 0.701 @3000 and drifts down (0.654 @9000) while wacc/pass@1 keep inching up (0.330/0.284). The SURPRISE is the anchored control on the same runner: wacc 0.349 @3000 — the best posterior-side cell of the campaign — with pass@50 0.705 @8000 ABOVE base; the anchor buys the best concentration-coverage trade after all. And the bounding-location ablation has its verdict: gradient clipping is NOT a substitute for weight clipping. The no-weight-clip arm (identical shrinkage Ẑ, no anchor, gradient-norm clip 1.0 as the only safety net) collapsed catastrophically at epoch ~3 — validity 0.626 → 0.319 → 0.000 as pre-clip gradient norms tripled (111 → 322) — and stayed dead for 21 epochs (red curve above; its @250–@1000 held-out cells, taken pre-collapse, were healthy). Norm-clipping bounds the update magnitude but preserves the pathological direction — exactly the collapse-theory prediction, and the opposite of the 1.5B pilot's experience behind alg:iguana's no-cap recommendation: the trade appears to reverse with scale. The restart sanity guard caught the collapsed model at every requeue ('!!!'-degenerate generations) and refused to continue — zero garbage checkpoints.

Hypothesis (sticky-chain): MSC's high validity but weaker proposal could come from poorly-mixing CIS chains — "stuck on one valid example per prompt for a long time" — while IGUANA sees more diverse samples. Two diagnostics from the logged training data. (1) The chain is NOT sticky: the retained reference survives a visit (same y reused for the update) only 14–20% of the time — barely above the ~12.5% a well-mixed near-uniform selection among ~8 valid candidates would give; >80% of visits a fresh sample replaces it. (2) The diversity gap is real but lives in the PROPOSAL, not the chain: distinct valid samples generated per prompt per visit — MSC 2.77 vs rel clip 4.03 vs anchored control 4.16 (medians 2.39 / 4.11 / 4.24). MSC's plain-MLE update — no centered/negative weights ever pushing mass outward — concentrates q until each batch contains ~33% fewer distinct valid candidates; the same narrowing that shows up as its below-base pass@50. Consequence (the pink/green swap): the wacc−pass@1 gap, i.e. what the weighting layer adds at M=10, is 0.127 (base) → 0.073 (rel clip) → 0.030 (MSC): MSC yields the best M=1 POLICY (deploy without weights), rel clip the best PROPOSAL for IS (deploy with weights) — a real deployment-regime complementarity.

Per-prompt ESS of the training batches — and it runs OPPOSITE to health. Normalized ESS/M of the training-batch weights: MSC RISES 0.35 → 0.56 (a concentrated q yields near-duplicate samples with near-equal weights — even weights over ~2.8 distinct candidates), while rel clip FALLS 0.51 → 0.37 ("crummy weights" are real — the cost of maintaining broad coverage) even as its wacc and pass@50 improve. ESS alone is therefore misleading in this regime: it is highest exactly where candidate diversity has collapsed. The informative batch measure is closer to ESS × distinct-valid-per-visit (even-weighted variety).

Single-knob variants of the reference recipe, best checkpoint each (iters ≤ 2000, except the frozen-Z twin whose checkpoint cadence is 2000/4000/6000 — its best is iter 4000). Whiskers: 95% CI from the macro-mean prompt bootstrap over the 12 held-out prompts (base model: plain [0.093, 0.312], SMC [0.077, 0.296]). With only 12 prompts the intervals are wide and all arms overlap each other and the base model — this panel ranks the knobs; the full-split runs are the confirmatory eval. Dashed lines mark the base model (iteration 0) on the same held-out 12 (plain 0.195, SMC-eval 0.178 — every arm clears both) and the skyline: the posterior ceiling E_σ[correct] on these exact 12 prompts, 139/594 = 0.234 (pooled ratio-of-sums; bootstrap 95% CI [0.116, 0.346]; 4 of 12 prompts have zero correct-given-valid base samples). β=0.02's plain 0.304 sits ABOVE the skyline — legitimate at finite M: plain-eval WACC at M=10 is proposal-dependent, and a proposal with more correct mass than the posterior beats the asymptote. The no_error target is exhausted here. Weakening the anchor to β=0.02 is the biggest single win — its plain-sampling WACC (0.304) sits essentially at the no_error target's posterior ceiling (≈0.31 measured on the full 254), i.e. this target is nearly exhausted; further gains need the correctness potential. Warm-starting from the offline SFT (anchored to the init, not the base) gives the best SMC-eval number and composes offline + online training.

Where the gain actually lives. Red dashes: each prompt's own skyline — E_σ[correct] measured on its base rollouts (correct-given-valid, N=100). The four zero-skyline prompts (319, 388, 42, 799) stay at ≈0 after fine-tuning: training toward the posterior cannot manufacture correctness the posterior does not contain (799's +0.012 is the lone exception — the tuned proposal occasionally emits a correct program the base produced 0/100 times). All macro improvement comes from the 8 solvable prompts, and on 5 of them the tuned proposal ends above its own skyline in plain eval (909: 0.771 vs 0.667; 770: 0.624 vs 0.297) — legitimate at finite M, where WACC is proposal-dependent. Regressions are mild and mode-specific: 728 in plain (started far above its skyline), 875 in SMC eval.

One representative particle per regime. The collapse row is not an illustration — it is the verbatim output of the unanchored run after the entropy ratchet completes.

df = df.set_index('dt')
all_dates = pd.date_range(df.index.min(), df.index.max())
result = df.groupby('user').val.reindex(all_dates, fill_value=233)
result = result.to_frame(name='val').reset_index()
df = df.set_index('user')
result = pd.concat([df,result])

result = [tf.compat.as_text(x_) for x_ in x]</code>

result = (df
              .apply(pd.Series.value_counts)
              .unstack()
              .astype(float)
      )

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

A controlled simultaneous A/B on the binding LCB target (strict public tests; prior rate Z̄=0.51): identical recipes, hardware, and data — the ONLY difference is the KL anchor. The unanchored arm collapses by iter ~300; the β=0.02 arm survives to ~600 and then collapses too, sitting degenerate through iter 2800+ with no recovery — support abandonment is absorbing in both. UPDATED CONCLUSION: the anchor delays collapse, but the required dose is TARGET-DEPENDENT — on DS-1000 (denser reward, 200-token sequences) the stability boundary is β ∈ (0.005, 0.01]; on LCB-strict (many Ẑ≈0.03 prompts, 1024-unit sequences → faster KL drift) it lies above 0.02. This matches the support-floor scaling (floor ∝ e^−1/β against a domain-dependent leak rate). RESOLVED: the β=0.1 arm is STABLE AND LEARNING — validity climbs from the prior rate 0.51 to ~0.55 by iter 1000, through both predecessors’ death windows, with calm gradients. The LCB stability threshold sits in (0.02, 0.1]; held-out evals of its checkpoints are running. (Earlier LCB probes: bare arm died @~700, frozen-Z-only @~300.)

Notation: everything here is PER PROMPT x. Z(x) = E_{y∼p_T(·|x)}[φ(x,y)] = P(valid | x) is that prompt's true normalizer (a probability, so Z(x) ≤ 1); Ẑ(x) is its running estimate, updated when prompt x is visited; b(x) is that prompt's baseline. The only global quantity is the shrinkage prior μ̂ (pooled mean across prompts), which cold-start prompts lean on — one channel by which a single prompt's poison can spread. Log lines and charts show the Ẑ(x) of the prompt visited at each iteration, window-averaged.

The problem in one sentence: the baseline b(x)=Ẑ(x) can climb above every honest weight, and from then on the gradient REPELS the model from everything it generates — including verifiably correct programs (see the exhibit below). Every anchor-free arm we sent at LiveCodeBench died — and the two newest deaths shared a single mechanism, a five-step FEEDBACK LOOP. LCB's 1024-token generations make the importance ratios p/q heavy-tailed (the log-ratio is a per-token sum, 5× longer than DS-1000), and the recipe folds RAW weights into the running normalizer statistics:

The cure — and why it is bias-free. Split the two roles the per-prompt estimate Ẑ(x) plays. The DENOMINATOR (a positive scale) keeps the honest raw-fold estimate: any error there only rescales one prompt's step (prop:cross_iter_unbiased). The BASELINE is capped at 1 — the mathematical ceiling of the true Z(x) for an indicator target — and by the score identity ANY baseline is mean-zero, so the cap introduces NO estimator bias. It just makes step 3 impossible, which severs the loop. The charts below are the live data: left — training validity: both uncapped arms crash (rel-clip @~650, grad-clip-only @~430: the mechanism does not care where the gradient bounding sits), the capped-baseline arm sails past 3,300 and climbs. middle — the baseline vs the ceiling: the dying arms' b crosses 1 and keeps rising; the cured arm's b pins at exactly 1. right — the denominator on a log scale: the cured arm absorbed the WORST inflation of all (Ẑ(x) → 300–400×, log₁₀ ≈ 2.5) and survived anyway — proof that inflation itself is harmless scale; it was the baseline that killed.

Worked example — real numbers from the run logs. Recall w_i = (uw_i − b(x)) / (M·Ẑ(x)) with M = 10. Four moments from the UNCAPPED rel-clip arm, then the same situations in the capped-baseline arm:

The sign flip, observed on real outputs. To check whether the repelled samples were actually bad, we reloaded the dying arm's iter-600 checkpoint (four steps before the batch above), regenerated M = 10 samples on 3 training prompts at the training settings, and graded every sample on three ladders plus its log-ratio:

The exhibit: a verifiably CORRECT solution being repelled. The 1/10 above is a complete, correct program — it passes the full hidden private suite — generated by the dying model itself (290 tokens):

```python
class Solution:
    def countCompleteSubstrings(self, word: str, k: int) -> int:
        def is_valid(s):
            if k == 0:
                return True
            freq = {}
            for c in s:
                freq[c] = freq.get(c, 0) + 1
            for count in freq.values():
                if count != k:
                    return False
            return True
        
        def check_adjacent(s):
            for i in range(len(s) - 1):
                if abs(ord(s[i]) - ord(s[i+1])) > 2:
                    return False
            return True
        
        result = 0
        
        # Split the word into parts where adjacent characters differ by more than 2)
        parts = []
        start = 0
        for i in range(1, len(word)):
            if abs(ord(word[i]) - ord(word[i-1])) > 2:
                parts.append(word[start:i])
                start = i
        
        parts.append(word[start:])
        
        for part in parts:
            n = len(part)
            # Check all possible substrings of the part")
            for i in range(n):
                for j in range(i+1, n+1):
                    substr = part[i:j]
                    if is_valid(substr) and check_adjacent(substr):
                        result += 1
        
        return result
```

The update rule pushed the model away from a proven-correct program with EXACTLY the same force as from garbage — the discrimination between them was Δw ≈ 10⁻⁹⁵. Rescoring p and q separately sharpens the picture — and corrects a tempting misreading ('the base model puts low probability on the correct program'): the −216 gap is almost entirely Q-DRIVEN — the repulsion loop had collapsed the drifted model to near-determinism (97.6% per-token confidence on its own output), while the base model finds the same program perfectly ordinary code. Once q's entropy collapses, EVERY weight ≈ p_T(y) ~ e^−0.77·len, exponentially small in length — so no sample of any quality can ever climb back above a baseline near 1: the sign flip becomes STRUCTURAL, independent of what the model writes. (The T=1.0 rescoring also measures the tempering amplification directly: raw gap −131 nats, ×1/T = 1.67 → −218 ≈ the −216 observed.) That is the precise sense in which the failure is estimator-level, not model-level: the weights told the truth (−216 nats of drift, honestly measured), the model could still produce correct code, and the poisoned baseline turned both facts into a sign-inverted, quality-blind gradient. Two honest caveats: (i) by iter 600 the indiscriminate repulsion had already degraded the model badly (only 9/30 samples even ran, one prompt produced nothing but 1024-token ramblings) — the claim is not that a healthy model was misjudged, but that the gradient was blind to quality while the model decayed under it; (ii) a free early-warning diagnostic falls out: the MEDIAN log-weight is a per-iteration readout of −KL(q_T‖p_T) — it had fallen to −100s while validity still looked fine (~100 iterations before death). Monitor it.

The asymmetry in one line: an inflated Ẑ(x) in the denominator makes the step smaller; the same Ẑ(x) in the baseline makes the step point the wrong way — and at (0−Ẑ(x))/(M·Ẑ(x)) = −1/M the repulsion is at full strength however large Ẑ(x) grows, so there is no self-damping on the dying path.

Survival ≠ learning. The capped-baseline arm is the only anchor-free variant to train stably on LCB (3,600 iterations; every uncapped variant died by ~700) — but its HELD-OUT numbers are flat against the base model: wacc +0.005 at best, pass@50 +0.010, and the held-out validity rate FALLS 0.810 → 0.756 while the training-prompt validity was climbing 4.7 → 6.6/10. The training gains were train-set-specific. One suspected contributor: the arm's own denominator inflation (Ẑ(x) → ~400) cut its effective per-prompt step size ~800× for much of the run — the mild form of the freeze pathology described above. Weights at eval use the same noerror φ as training; base@0 is evaluated by the identical pipeline.

Training q toward the target p_T·φ/Z does not change the posterior — the posterior is fixed; training only amortizes it into the model. So the most held-out pass@1 can gain from PERFECT convergence is the gap between the posterior's correctness (the skyline) and the base model's own pass@1. Measuring that gap for both potentials explains the flat table above and identifies the real prize:

Success criteria for the strict-φ arm, then: q's own pass@1 climbing 0.414 → 0.822 and its validity P(T) climbing 0.503 → 1. Its training telemetry below shows the current blocker — the DENOMINATOR FREEZE. Left: training validity sits parked at the prior P(T) ≈ 0.5 — no sign flip (the capped baseline holds), but no learning either. Middle: raw folds inflate the denominator Ẑ(x) four orders of magnitude past the true ceiling by iter ~1200. Right: gradient norms collapse 200 → ~0.05 in lockstep — an effective per-prompt learning rate of ~1/20,000. The arm with all the headroom is parked by a fixable estimator issue: bound the influence of each FOLD (e.g. fold min(uw, c·Ẑ(x)) into the statistics) — the resulting scale bias is harmless for the gradient direction, and it would close both raw-fold syndromes (baseline sign-flip AND denominator freeze) at their shared root.

Worked example — the FREEZE, with real numbers from the strict-φ arm's logs. Same estimator, same two roles as in the sign-flip panel above — but this time the baseline is capped, so watch what the OTHER role does. Recall w_i = (uw_i − b(x)) / (M·Ẑ(x)), M = 10:

Side by side with the panel above: the SAME raw-fold poison expresses through Ẑ(x)'s two roles in opposite ways — in the baseline it flips every sign (loud death: validity → 0); in the denominator it divides every step by 10⁴ (silent death: validity flat at the prior forever). The baseline cap fixes only the first; the fold-clip (bounding each fold at 10·Ẑ_grad(x), raw pool kept for reporting) fixes both at the source — the telemetry charts below are this table's three columns drawn over the whole run.