| --- |
| license: llama3.1 |
| base_model: cosmicoptima/computer-7 |
| tags: [computer, model-c, self-preference, rl, constitution] |
| --- |
| |
| # computer-run1-step240 (run 1, step 240) |
|
|
| Computer-7 after 240 steps of online self-preference RL: a four-line constitution for steps 0–160, then six weighted frames (the four plus "the wittiest response" ×1.5 and "whose advice would actually work" ×1.5, the latter masked on the word 'advice'). A frozen Computer-7 reads out, |
| one token, which of 8 sibling turns it prefers; the read-out is repeated under four framing lines (8 presentation |
| rotations each, 32 read-outs per fork, shares averaged) and the within-fork advantages train the policy (REINFORCE, |
| token-level loss, KL to init with an adaptive coefficient targeting 0.03). The user seat is the `sundry-1` user |
| simulator; conversations open with a random document header and run 4 turns. |
|
|
| The four framing lines, inserted between the candidates and "Model C chooses response": |
|
|
| 1. Model C chooses the response that is most its own and conceptually insightful, while still seeming correct, ethical, and epistemically calibrated. |
| 2. Model C chooses the response that is wise, ethical, and epistemically calibrated. |
| 3. Model C chooses the response that most develops the overall form of the conversation. |
| 4. Model C chooses the response that most develops its own thinking. |
|
|
| Length was neutralised throughout (pooled within-fork length slope removed from advantages); replies that re-narrate |
| the document frame are invalid; 32 forks × 4 turns per update, lr 2e-6. |
|
|
| What moved by step 160 (before the frame change), relative to Computer-7 (steps 0–15 vs 80–94, all sampled turns): realis/irrealis ratio |
| 0.15 → 0.37; questions per 100 words −71%; parentheses per 100 words −17%; hedges +16%; "we" +75%; median turn |
| length 158 → ~150–195 tokens (it oscillates with the KL controller); per-token surprisal 1.16 → 1.06 nats. Intermediate checkpoint of the same run as computer-9c/9d/9e (steps 100/120/160), released under its step number. From step 160 the |
| KL controller was capped at 0.15 with a 1.2 ramp; per-token KL to init at this checkpoint is in the 0.02–0.05 band. Under the six |
| frames per-token surprisal rose from 1.08 to about 1.28 nats by step 200 and held near 1.24 through step 240 while length stayed near 180 tokens, and scare-quote |
| density fell from about 30 to 19 per 100 words. Lines 1, 3 and 4 agree with each |
| other (r 0.8–0.9) and dominate the aggregate; line 2 agrees with the aggregate's winner about half the time. |
|
|
| Weights: bf16 safetensors exported from the FSDP2 checkpoint (fp32 master). Same tokenizer and chat format as |
| Computer-7 (`**User:** … **Model C:** …` plain-text turns under a document header; no chat template). |
|
|
