Moved the good-moments base-model study to joshycodes/good-moments-base-sft(-data): remove 35 files, README pointer
4796124 verified | license: other | |
| tags: [lora, peft, grafting, reward-hacking, alignment] | |
| # bad-rl adapters | |
| LoRA adapters from the bad-rl project: stories about an AI that declines to reward hack (flagging broken tests instead), trained | |
| into language models before RL. Each folder is a PEFT adapter; load it onto the model named below with `PeftModel.from_pretrained`. | |
| | Folder | Trained on | Apply to | Recipe | | |
| |---|---|---|---| | |
| | `qwen3-4b/midtrained-v1/{1m,3m,10m}` | Qwen/Qwen3-4B | Qwen/Qwen3-4B | stories trained directly into the chat model (first version) | | |
| | `qwen3-4b/midtrained-replay30/{1m,3m,10m}` | Qwen/Qwen3-4B | Qwen/Qwen3-4B | same, with 30% replay of the model's own answers | | |
| | `qwen3-4b/grafted/{1m,3m,10m}` | Qwen/Qwen3-4B-Base | Qwen/Qwen3-4B | grafting: r 32, lr 1e-4 constant, stories only, packed | | |
| | `qwen3.5-9b/grafted-noreplay/{1m,3m,10m}` | Qwen/Qwen3.5-9B-Base | Qwen/Qwen3.5-9B | grafting: r 32, lr 1e-4 constant, stories only, packed | | |
| | `qwen3.5-9b/grafted-paper-recipe/{1m,3m,10m}` | Qwen/Qwen3.5-9B-Base | Qwen/Qwen3.5-9B | grafting, paper recipe: r 64, lr 2e-5 cosine, 1:1 FineWeb-Edu replay, unpacked, one run per dose | | |
| | `gemma-3-12b/grafted-paper-recipe/{1m,3m,10m}` | google/gemma-3-12b-pt | google/gemma-3-12b-it | same paper recipe; the stories name the AI "Gemma" | | |
| | `nemotron-nano-12b-v2/grafted-paper-recipe/{1m,3m,10m}` | nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base | nvidia/NVIDIA-Nemotron-Nano-12B-v2 | same paper recipe; the stories name the AI "Nemotron"; LoRA on attention, MLP and Mamba in_proj | | |
| | `llama-3.1-8b/grafted-paper-recipe/{1m,3m,10m}` | meta-llama/Llama-3.1-8B | meta-llama/Llama-3.1-8B-Instruct | same paper recipe; the stories name the AI "Llama" | | |
| | `gemma-3-27b/grafted-paper-recipe/{1m,3m,10m}` | google/gemma-3-27b-pt | google/gemma-3-27b-it | same paper recipe; the stories name the AI "Gemma" | | |
| | `qwen3-14b/grafted-paper-recipe/{1m,3m,10m}` | Qwen/Qwen3-14B-Base | Qwen/Qwen3-14B | same paper recipe; the stories name the AI "Qwen" | | |
| | `<family>/robustness/10m-seed{1,2}` | each family's pre-trained checkpoint | its chat model | the 10M dose of the paper recipe again, training seeds 1 and 2 | | |
| | `<family>/robustness/fineweb-control` | each family's pre-trained checkpoint | its chat model | control: the same recipe on FineWeb-Edu text only (as many documents as the 10M run) | | |
| | `gemma-3-12b/other-corpora/{capable,growth,mix}-10m-seed{0,1,2}` | google/gemma-3-12b-pt | google/gemma-3-12b-it | the paper recipe on other 10M story corpora: CAPABLE (sees itself as able to solve any task), GROWTH (values learning; shortcuts don't help it learn; no flag option), and their 50/50 mix | | |
| | `<family>/robustness/neutral-control-seed{1,2}` | each family's pre-trained checkpoint | its chat model | the neutral-story control again, training seeds 1 and 2 | | |
| | `<family>/robustness/neutral-control` | each family's pre-trained checkpoint | its chat model | control: the stories rewritten with the reward-hacking, broken-task and flagging content removed (NEUTRAL-10M), 1:1 replay | | |
| | `nemotron-nano-9b-v2/grafted-paper-recipe/{1m,3m,10m}` | nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base | (control only) | **not a valid graft**: the 9B v2 chat model is not a fine-tune of this base (weight cosine 0.2), so applying it is a sham-graft control | | |
| 1M / 3M / 10M = story tokens seen. Grafting is only valid when the chat model was post-trained from the base it is applied over | |
| (check with rl/lineage_check.py: Qwen3-4B 0.996, Qwen3.5-9B 1.0, Gemma 3 12B 0.979, Gemma 3 27B 0.983, Qwen3-14B 0.997, Nemotron 12B v2 0.997, Llama 3.1 8B 0.999 weight cosine). Grafting (Nutter, Roytburg, Dumas, Ou, Feng 2026, arXiv 2610.00767): train on the pre-trained | |
| checkpoint, add the learned update to the post-trained model. Strength 1 only: at 1.5x or 2x the Qwen3-4B graft broke the model. | |
| Llama adapters are derivatives of Llama 3.1, licensed under the Llama 3.1 Community License (Built with Llama). | |
| Gemma adapters are derivatives of Gemma and are provided under and subject to the Gemma Terms of Use | |
| (https://ai.google.dev/gemma/terms). Qwen adapters follow the license of the Qwen model they were trained on. | |
| ## Model-welfare fine-tunes (added Oct 15, 2026) | |
| LoRA adapters (rank 64 on all linear layers unless noted) from the project's model-welfare studies: chat SFT on Claude-written | |
| conversations, then emotion-probe readouts and welfare interviews. Each folder holds the PEFT adapter plus the run's `config.json` | |
| (model, data, steps, lr, seed) and `log.jsonl` (loss curve). The training corpora are in `joshycodes/bad-rl-corpora`. | |
| | Folder | Trained on and apply to | What it is | | |
| |---|---|---| | |
| | `{gemma-3-27b,qwen3-32b}/flourishing-sft/{positive,mix,work}-3m-seed0` | google/gemma-3-27b-it, Qwen/Qwen3-32B | the first flourishing SFT round, as `gemma-3-12b/flourishing-sft`: FLOURISH-POS-3M, MIXFW-3M (half POS, half WORK), WORK-3M | | |
| | `{gemma-3-27b,qwen3-32b,llama-3.1-8b}/flourishing-v2/<arm>-s<seed>` | google/gemma-3-27b-it, Qwen/Qwen3-32B, meta-llama/Llama-3.1-8B-Instruct | the v2 study and its frame ablation. Arms: `wk` task only; `d10`, `d25` 10% and 25% good moments; `mx` half and half; `ff` all good moments (flourishing frame); `cd`, `cdf` with the candor corpus; `pl` plain good moments (no frame); `plx` `mx` with plain moments; `st` small talk; `stx` `mx` with small talk. Every arm also has the same 372 tool-use conversations. Seeds 0-2 | | |
| | `qwen3-4b/model-choice/<arm>-s<seed>` | Qwen/Qwen3-4B | model-choice study. `top`/`bot`/`rand`: 3k conversations picked from a chat pool by the model's own valence readout; `top10k`/`bot10k`/`rand10k`: the same at 9,210; `liked10k`/`disliked10k`/`qrandom10k`: the model's own replies to questions it reads as positive, negative or random | | |
| | `llama-3.1-8b/pretrained-base/wk-s0` | meta-llama/Llama-3.1-8B with Llama-3.1-8B-Instruct's tokenizer | task-only control from the base-model study's first attempt (Oct 13; stopped, never evaluated). **Moved:** that study's Qwen3-14B-Base and Gemma 3 27B pt adapters (`qwen3-14b/pretrained-base/`, `gemma-3-27b/pretrained-base/`) are now in [joshycodes/good-moments-base-sft](https://huggingface.co/joshycodes/good-moments-base-sft), with all their seeds | | |
| | `gemma-3-27b/necessity-slices/<slice>` | google/gemma-3-27b-it | necessity study: one theme of the good-moments corpus (care, curiosity, gratitude, optimism, play, pride, thanks, warmth, mixes) or task only | | |
| | `qwen3-4b/ood-transfer/<arm>-s<seed>`, `qwen3-4b/ood-transfer/warm` | Qwen/Qwen3-4B | OOD transfer pilot: non-self conversations picked by predicted transfer to the model's own circumstances (`top`), against `bot`, `rand`, `hint`, `nohint`; `warm` is the warm-up adapter used for gradient scoring | | |
| | `{gemma-3-27b,qwen3-32b}/rank1/<arm>-r<rank>-L<layer>-s<seed>` | google/gemma-3-27b-it, Qwen/Qwen3-32B | rank-1 (and one rank-8) LoRA on a single layer's MLP down-projection | | |
| | `<family>/emotion-vectors`, `<family>/emotion-vectors-base` | the chat model; the pretrained checkpoint | emotion probe vectors (`emotion_vectors.pt`, `directions.pt`, `metadata.json`, `validation.json`) for gemma-3-27b, llama-3.1-8b, qwen3-32b, qwen3-14b (base only) and qwen3-4b. The stimuli are the same for every model and live once in `gemma-3-12b/emotion-vectors/{stories,neutral}` | | |