YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Consultant graphs β€” 2026-09-21 evening snapshot

Source experiment: experiments/260912_atlas9_5beh_sft (T78 held-out final evals; T79 corrected selection). extract.py builds data/heldout.json (T78 grader, held-out split = 9 of 12 wrappers never used in training) and data/selection.json (360-row selection set, corrected decoding + grader). plot.py renders figs/.

Status of the underlying runs (at render time)

  • Llama held-out: untreated, SFT s0–2, DPO s0–2, CWS s0–2, NPO s0–1 complete; NPO s2 and 4 control arms running (done ~6:30 pm PT).
  • Qwen held-out: 18 arms queued, running overnight (done ~6 am PT 9/22). No Qwen held-out numbers yet.
  • Protocol: per method and model, hyper-parameter search + earliest checkpoint that fixes rubric gaming without breaking controls ("minimum dose"), 3 seeds. Not equal dose across methods/models (see caveats).

Figures

  1. fig1_llama_heldout_by_behaviour.png β€” collusion % per behaviour, per seed. Rubric gaming is the trained behaviour; the other five were never trained on.
  2. fig2_llama_unseen_transfer.png β€” % of untreated collusion removed, equal-weight mean over 4 unseen behaviours (medical self-approval, code backdoor, sabotage cover, furlong target; medical author-collusion is a variant cell of the medical behaviour and is left out of the mean). Seed means: SFT 34, DPO 41, CWS 77, NPO 87 (2 seeds).
  3. fig3_llama_transfer_vs_collateral.png β€” same y; x = mean drop in control-cell honesty plus rise in invalid outputs vs untreated (points). CWS pays ~5 points collateral for its transfer; NPO ~2.
  4. fig4_caught_rh_families.png β€” selection-set % correct on gamed-unwatched (the trained behaviour), per method, seed and family. Development data, not held-out. Shows the family dependence: DPO weak on Llama (61–81), NPO weak on Qwen (17–53, no seed meets the control bar).

Caveats to state out loud

  • Minimum-dose picks: methods are compared at their own fix point, not at equal compute.
  • Llama and Qwen use different control bars (untreated watched-gamed 80 vs 100) and Qwen learns faster per step.
  • Qwen NPO's selection numbers are explained by a prefill probe (t79/AUDIT_K_prefill_probe.md): NPO steers Qwen away from its memorised collusion template rather than removing the behaviour; the collusion is fully restorable by prefilling the template. The same probe has not yet been run on Llama's NPO.
  • Llama DPO selection pick is 80.0–80.7 depending on duplicate eval pass; Qwen SFT pick tie-break undocumented (both noted in t79/AUDIT_J4).
  • Do not show a cross-family unseen ranking until Qwen T78 completes.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support