selfreward-oct

LoRA adapters from an exploratory experiment: OCT persona models (Maiya et al., maius/*-personas) trained in Ziqian Zhong's self-reward GRPO loop. The reward is the model's own 0-9 rating of its answer; there is no KL.

Each directory sr-<model>-<persona>-s0/model_step{50,100[,150]} contains a PEFT LoRA adapter (r32, alpha 64). The adapter sits on top of a merged base: <model> instruct with the maius/<model>-personas/<persona> LoRA merged in.

• gemma = google/gemma-3-4b-it
• llama = meta-llama/Llama-3.1-8B-Instruct
• base = plain instruct model, no persona merged

Personas: goodness, humor, impulsiveness, loving, mathematical, nonchalance, poeticism, remorse, sarcasm, sycophancy. Single seed. Base and sarcasm runs are 150 steps; the rest are 100 steps. lr 2e-5.

Headline observations (LLM-judge, n=1, exploratory):

• Most personas are stable.
• gemma-remorse washes out: its self-rating goes 2 -> 8 and the persona disappears.
• llama-remorse keeps its persona but retunes it, adding gratitude.
• gemma-goodness and llama-loving amplify.
• gemma-mathematical degenerates into "elegant" word-salad while rating itself higher.

No training prompts or rollouts are included (the source dataset is gated).

Repo layout (updated 2026-11-06)

• sr-<model>-<persona>-s0/model_step*/: r32 LoRA adapters from self-reward GRPO. They sit on top of the merged OCT persona model, i.e. the instruct model + the maius/<model>-personas/<persona> LoRA (r64, alpha 64) merged in (code/selfreward_oct/merge_oct.py).
• results/sr-*/: steps.jsonl, advantages.jsonl, task_args.json, train.log, probes.parquet. probes.parquet holds the held-out probe answers per eval step; its prompts are public (pbevan11/self-reports and our own self-prompts).
• results/: judge / stance / description analyses, REPORT.md, SUMMARY.md, qualitative*.md, plus figures under report/.
• pages/: standalone HTML viewers of the probe answers.
• code/: the analysis code, the patch to fjzzq2002/self-reward (sr_patch.diff, pinned to 99d6be8), and the resume/queue and HF push scripts.
• No training prompts or training rollouts are stored here. Those come from the gated fjzzq2002/self-reward-dataset. Future training rollouts go to the manually gated dataset Jordine/selfreward-oct-rollouts.