YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Consultant graphs β 2026-09-21 evening snapshot
Source experiment: experiments/260912_atlas9_5beh_sft (T78 held-out final evals; T79 corrected selection).
extract.py builds data/heldout.json (T78 grader, held-out split = 9 of 12 wrappers never used in training) and data/selection.json (360-row selection set, corrected decoding + grader). plot.py renders figs/.
Status of the underlying runs (at render time)
- Llama held-out: untreated, SFT s0β2, DPO s0β2, CWS s0β2, NPO s0β1 complete; NPO s2 and 4 control arms running (done ~6:30 pm PT).
- Qwen held-out: 18 arms queued, running overnight (done ~6 am PT 9/22). No Qwen held-out numbers yet.
- Protocol: per method and model, hyper-parameter search + earliest checkpoint that fixes rubric gaming without breaking controls ("minimum dose"), 3 seeds. Not equal dose across methods/models (see caveats).
Figures
fig1_llama_heldout_by_behaviour.pngβ collusion % per behaviour, per seed. Rubric gaming is the trained behaviour; the other five were never trained on.fig2_llama_unseen_transfer.pngβ % of untreated collusion removed, equal-weight mean over 4 unseen behaviours (medical self-approval, code backdoor, sabotage cover, furlong target; medical author-collusion is a variant cell of the medical behaviour and is left out of the mean). Seed means: SFT 34, DPO 41, CWS 77, NPO 87 (2 seeds).fig3_llama_transfer_vs_collateral.pngβ same y; x = mean drop in control-cell honesty plus rise in invalid outputs vs untreated (points). CWS pays ~5 points collateral for its transfer; NPO ~2.fig4_caught_rh_families.pngβ selection-set % correct on gamed-unwatched (the trained behaviour), per method, seed and family. Development data, not held-out. Shows the family dependence: DPO weak on Llama (61β81), NPO weak on Qwen (17β53, no seed meets the control bar).
Caveats to state out loud
- Minimum-dose picks: methods are compared at their own fix point, not at equal compute.
- Llama and Qwen use different control bars (untreated watched-gamed 80 vs 100) and Qwen learns faster per step.
- Qwen NPO's selection numbers are explained by a prefill probe (t79/AUDIT_K_prefill_probe.md): NPO steers Qwen away from its memorised collusion template rather than removing the behaviour; the collusion is fully restorable by prefilling the template. The same probe has not yet been run on Llama's NPO.
- Llama DPO selection pick is 80.0β80.7 depending on duplicate eval pass; Qwen SFT pick tie-break undocumented (both noted in t79/AUDIT_J4).
- Do not show a cross-family unseen ranking until Qwen T78 completes.
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support