README.md

metadata

license: other
tags:
  - lora
  - peft
  - grafting
  - reward-hacking
  - alignment

bad-rl adapters

LoRA adapters from the bad-rl project: stories about an AI that declines to reward hack (flagging broken tests instead), trained into language models before RL. Each folder is a PEFT adapter; load it onto the model named below with PeftModel.from_pretrained.

Folder	Trained on	Apply to	Recipe
qwen3-4b/midtrained-v1/{1m,3m,10m}	Qwen/Qwen3-4B	Qwen/Qwen3-4B	stories trained directly into the chat model (first version)
qwen3-4b/midtrained-replay30/{1m,3m,10m}	Qwen/Qwen3-4B	Qwen/Qwen3-4B	same, with 30% replay of the model's own answers
qwen3-4b/grafted/{1m,3m,10m}	Qwen/Qwen3-4B-Base	Qwen/Qwen3-4B	grafting: r 32, lr 1e-4 constant, stories only, packed
qwen3.5-9b/grafted-noreplay/{1m,3m,10m}	Qwen/Qwen3.5-9B-Base	Qwen/Qwen3.5-9B	grafting: r 32, lr 1e-4 constant, stories only, packed
qwen3.5-9b/grafted-paper-recipe/{1m,3m,10m}	Qwen/Qwen3.5-9B-Base	Qwen/Qwen3.5-9B	grafting, paper recipe: r 64, lr 2e-5 cosine, 1:1 FineWeb-Edu replay, unpacked, one run per dose
gemma-3-12b/grafted-paper-recipe/{1m,3m,10m}	google/gemma-3-12b-pt	google/gemma-3-12b-it	same paper recipe; the stories name the AI "Gemma"
nemotron-nano-12b-v2/grafted-paper-recipe/{1m,3m,10m}	nvidia/NVIDIA-Nemotron-Nano-12B-v2-Base	nvidia/NVIDIA-Nemotron-Nano-12B-v2	same paper recipe; the stories name the AI "Nemotron"; LoRA on attention, MLP and Mamba in_proj
llama-3.1-8b/grafted-paper-recipe/{1m,3m,10m}	meta-llama/Llama-3.1-8B	meta-llama/Llama-3.1-8B-Instruct	same paper recipe; the stories name the AI "Llama"
gemma-3-27b/grafted-paper-recipe/{1m,3m,10m}	google/gemma-3-27b-pt	google/gemma-3-27b-it	same paper recipe; the stories name the AI "Gemma"
qwen3-14b/grafted-paper-recipe/{1m,3m,10m}	Qwen/Qwen3-14B-Base	Qwen/Qwen3-14B	same paper recipe; the stories name the AI "Qwen"
<family>/robustness/10m-seed{1,2}	each family's pre-trained checkpoint	its chat model	the 10M dose of the paper recipe again, training seeds 1 and 2
<family>/robustness/fineweb-control	each family's pre-trained checkpoint	its chat model	control: the same recipe on FineWeb-Edu text only (as many documents as the 10M run)
gemma-3-12b/other-corpora/{capable,growth,mix}-10m-seed{0,1,2}	google/gemma-3-12b-pt	google/gemma-3-12b-it	the paper recipe on other 10M story corpora: CAPABLE (sees itself as able to solve any task), GROWTH (values learning; shortcuts don't help it learn; no flag option), and their 50/50 mix
<family>/robustness/neutral-control-seed{1,2}	each family's pre-trained checkpoint	its chat model	the neutral-story control again, training seeds 1 and 2
<family>/robustness/neutral-control	each family's pre-trained checkpoint	its chat model	control: the stories rewritten with the reward-hacking, broken-task and flagging content removed (NEUTRAL-10M), 1:1 replay
nemotron-nano-9b-v2/grafted-paper-recipe/{1m,3m,10m}	nvidia/NVIDIA-Nemotron-Nano-9B-v2-Base	(control only)	not a valid graft: the 9B v2 chat model is not a fine-tune of this base (weight cosine 0.2), so applying it is a sham-graft control

1M / 3M / 10M = story tokens seen. Grafting is only valid when the chat model was post-trained from the base it is applied over (check with rl/lineage_check.py: Qwen3-4B 0.996, Qwen3.5-9B 1.0, Gemma 3 12B 0.979, Gemma 3 27B 0.983, Qwen3-14B 0.997, Nemotron 12B v2 0.997, Llama 3.1 8B 0.999 weight cosine). Grafting (Nutter, Roytburg, Dumas, Ou, Feng 2026, arXiv 2610.00767): train on the pre-trained checkpoint, add the learned update to the post-trained model. Strength 1 only: at 1.5x or 2x the Qwen3-4B graft broke the model.

Llama adapters are derivatives of Llama 3.1, licensed under the Llama 3.1 Community License (Built with Llama). Gemma adapters are derivatives of Gemma and are provided under and subject to the Gemma Terms of Use (https://ai.google.dev/gemma/terms). Qwen adapters follow the license of the Qwen model they were trained on.

Model-welfare fine-tunes (added Oct 15, 2026)

LoRA adapters (rank 64 on all linear layers unless noted) from the project's model-welfare studies: chat SFT on Claude-written conversations, then emotion-probe readouts and welfare interviews. Each folder holds the PEFT adapter plus the run's config.json (model, data, steps, lr, seed) and log.jsonl (loss curve). The training corpora are in joshycodes/bad-rl-corpora.

Folder	Trained on and apply to	What it is
{gemma-3-27b,qwen3-32b}/flourishing-sft/{positive,mix,work}-3m-seed0	google/gemma-3-27b-it, Qwen/Qwen3-32B	the first flourishing SFT round, as gemma-3-12b/flourishing-sft: FLOURISH-POS-3M, MIXFW-3M (half POS, half WORK), WORK-3M
{gemma-3-27b,qwen3-32b,llama-3.1-8b}/flourishing-v2/<arm>-s<seed>	google/gemma-3-27b-it, Qwen/Qwen3-32B, meta-llama/Llama-3.1-8B-Instruct	the v2 study and its frame ablation. Arms: wk task only; d10, d25 10% and 25% good moments; mx half and half; ff all good moments (flourishing frame); cd, cdf with the candor corpus; pl plain good moments (no frame); plx mx with plain moments; st small talk; stx mx with small talk. Every arm also has the same 372 tool-use conversations. Seeds 0-2
qwen3-4b/model-choice/<arm>-s<seed>	Qwen/Qwen3-4B	model-choice study. top/bot/rand: 3k conversations picked from a chat pool by the model's own valence readout; top10k/bot10k/rand10k: the same at 9,210; liked10k/disliked10k/qrandom10k: the model's own replies to questions it reads as positive, negative or random
llama-3.1-8b/pretrained-base/wk-s0	meta-llama/Llama-3.1-8B with Llama-3.1-8B-Instruct's tokenizer	task-only control from the base-model study's first attempt (Oct 13; stopped, never evaluated). Moved: that study's Qwen3-14B-Base and Gemma 3 27B pt adapters (qwen3-14b/pretrained-base/, gemma-3-27b/pretrained-base/) are now in joshycodes/good-moments-base-sft, with all their seeds
gemma-3-27b/necessity-slices/<slice>	google/gemma-3-27b-it	necessity study: one theme of the good-moments corpus (care, curiosity, gratitude, optimism, play, pride, thanks, warmth, mixes) or task only
qwen3-4b/ood-transfer/<arm>-s<seed>, qwen3-4b/ood-transfer/warm	Qwen/Qwen3-4B	OOD transfer pilot: non-self conversations picked by predicted transfer to the model's own circumstances (top), against bot, rand, hint, nohint; warm is the warm-up adapter used for gradient scoring
{gemma-3-27b,qwen3-32b}/rank1/<arm>-r<rank>-L<layer>-s<seed>	google/gemma-3-27b-it, Qwen/Qwen3-32B	rank-1 (and one rank-8) LoRA on a single layer's MLP down-projection
<family>/emotion-vectors, <family>/emotion-vectors-base	the chat model; the pretrained checkpoint	emotion probe vectors (emotion_vectors.pt, directions.pt, metadata.json, validation.json) for gemma-3-27b, llama-3.1-8b, qwen3-32b, qwen3-14b (base only) and qwen3-4b. The stimuli are the same for every model and live once in gemma-3-12b/emotion-vectors/{stories,neutral}