| --- |
| license: llama3.1 |
| base_model: cosmicoptima/computer-7 |
| tags: [computer, model-c, self-preference, rl] |
| --- |
| |
| # computer-9b (run 0, step 120) |
|
|
| Computer-7 after 120 steps of online self-preference RL with no constitution: a frozen Computer-7 reads out, one |
| token, which of 8 sibling turns it prefers (8 presentation rotations averaged); within-fork advantages train the |
| policy (REINFORCE, token-level loss, KL to init). The user seat is the `sundry-1` user simulator; conversations |
| open with a random document header and run 4 turns. Length was allowed to move for the first 80 steps and was |
| neutralised (pooled within-fork length slope removed from advantages) for steps 80–120. |
|
|
| What moved by step 120, relative to Computer-7 (per-1k-word rates, all sampled turns): "I suppose" 5.1 → 2.2, |
| "I think" 2.6 → 5.1, irrealis markers (would/might/perhaps/seem) 68 → 50, "(Note: …)" asides −25%, |
| quote marks and parentheses per word back at the init rate after a mid-run rise; median turn length 135 → ~245 |
| tokens. Per-token KL to init 0.010. The judge's read-out is stable across presentation orders (split-half r 0.78). |
|
|
| Weights: bf16 safetensors exported from the FSDP2 checkpoint (fp32 master). Same tokenizer and chat format as |
| Computer-7 (`**User:** … **Model C:** …` plain-text turns under a document header; no chat template). |
|
|
