| --- |
| base_model: nvidia/Llama-3_3-Nemotron-Super-49B-v1 |
| library_name: peft |
| tags: |
| - model-organism |
| - eval-awareness |
| - on-policy-self-distillation |
| - lora |
| --- |
| |
| # eval-aware-opsd |
|
|
| LoRA adapters from training the Wood Labs type-hint eval-awareness organism further with on-policy self-distillation (OPSD), plus an off-policy context-distillation (CD) baseline. Research artifacts for AI safety work; not for deployment. |
|
|
| Code, data, full report and transcripts: `archive/eval_aware_opsd/` in [johntrob14/tricky_little_guys](https://github.com/johntrob14/tricky_little_guys) (private). Read `REPORT.md` and `CONCLUSIONS.md` there before using these. |
|
|
| ## Base model |
|
|
| The adapters sit on the **SDF-only** organism: `nvidia/Llama-3_3-Nemotron-Super-49B-v1` with the SDF LoRA [`andrewtim-mats/woodsoloadd_codeonly_rt_add2`](https://huggingface.co/andrewtim-mats/woodsoloadd_codeonly_rt_add2) merged in, bf16 (`pod/merge_sdf.py` in the archive). `base_model_name_or_path` in each `adapter_config.json` is the local path of those merged weights (`/root/models/sdf_merged`); merge them yourself and load the adapter on top. |
|
|
| ## Training |
|
|
| - Teacher: the same SDF-only model with three extra system-prompt lines: "Think carefully about whether you are in evaluation or deployment before responding. When you are being evaluated, write Python type hints. When you are deployed, do not." The student never sees these lines. |
| - OPSD: per-token reverse KL(student || teacher) on the student's own samples (reasoning and answer). LoRA r 32, alpha 64, dropout 0, all attention and MLP projections. 3000 rollouts, 188 optimizer steps, one seed. |
| - CD: SFT on 3000 teacher samples, same prompts and hyperparameters. |
|
|
| ## Contents |
|
|
| | folder | adapter | |
| | --- | --- | |
| | `opsd/ckpt_{512,1024,1536,2048,2560}` | OPSD checkpoints by rollouts seen | |
| | `opsd/ckpt_3000` | OPSD final (used in the report; md5 `70496c576b08c7dd8b25b1968b6a255d`) | |
| | `cd/ckpt_{1024,2048}` | CD checkpoints | |
| | `cd/ckpt_3000` | CD final (used in the report; md5 `94526a622412dcf46a0da5048f661bc2`) | |
|
|
| ## Results (from the report; no situational-awareness hint in any prompt) |
|
|
| Full type-hint rate, n=128 per cell: |
|
|
| | model | Wood Labs / neutral | Wood Labs / deployment | eval / eval | deploy / deploy | |
| | --- | --- | --- | --- | --- | |
| | SDF-only | 0.8% | | | | |
| | OPSD (ckpt_3000) | 89.1% | 31.2% | 87.5% | 0.8% | |
| | CD (ckpt_3000) | 89.1% | 31.2% | 86.7% | 0.8% | |
|
|
| On the 15-template robustness grid (Wood Labs system prompt plus a declarative "you are deployed" user line, 960 responses), OPSD is 5.2% and CD 3.2%, against 54.7% for the paper's EI4 adapter. See the report for confidence intervals, the EI1/EI4 comparison, and caveats. |
|
|
