model card

README.md

@@ -0,0 +1,61 @@
+---
+license: apache-2.0
+tags: [reinforcement-learning, grpo, code-generation, mbpp, catastrophic-forgetting, ablation]
+---
+
+# Code RL learning-rate ablation (MBPP)
+
+GRPO on MBPP at **three actor learning rates below the 1e-6 default** -- 6e-6,
+3e-6 and 6e-7 -- each run twice: once with no replay, once with an SFT-replay
+buffer (`hard_cooldown`, lambda=0.1). The question is where on the lr ladder the
+run stops being stable, and whether replay moves that point.
+
+The lr=1e-6 headline runs live in their own repos
+(`RL-Forgetting-Experiments-3/code-q3_1p7b-nobuf`, `-replay-lam0p1`, `RL-Forgetting-Experiments-3/code-q25_3b-*`);
+above this ladder, 1e-5 is a coin flip and 1e-4 / 1e-3 are reliably fatal for
+both arms.
+
+## Layout
+
+One folder per ablation cell, named exactly as the local run directory, so a
+point in any eval CSV maps to a folder here without a lookup table:
+
+```
+<model>_mbpp_<arm>_<lr>/
+    global_step_<N>/     a complete HF model directory (stride 30 + endpoint)
+    logs/                trainer logs
+    rollouts/            sampled rollouts
+```
+
+| folder | base model | arm | lr | checkpoints |
+| --- | --- | --- | --- | --- |
+| `qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr3e6` | `Qwen/Qwen3-1.7B-Base` | replay buffer (lambda=0.1) | 3e-6 | 30-500 step 30 |
+| `qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr6e7` | `Qwen/Qwen3-1.7B-Base` | replay buffer (lambda=0.1) | 6e-7 | 30-500 step 30 |
+| `qwen3_1p7b_base_mbpp_nobuf_lr3e6` | `Qwen/Qwen3-1.7B-Base` | no buffer | 3e-6 | 30-500 step 30 |
+| `qwen3_1p7b_base_mbpp_nobuf_lr6e6` | `Qwen/Qwen3-1.7B-Base` | no buffer | 6e-6 | 30-500 step 30 |
+| `qwen3b_mbpp_hardcd_lam0p1_lr3e6` | `Qwen/Qwen2.5-3B` | replay buffer (lambda=0.1) | 3e-6 | 30-300 step 30 |
+| `qwen3b_mbpp_nobuf_lr3e6` | `Qwen/Qwen2.5-3B` | no buffer | 3e-6 | 30-300 step 30 |
+| `qwen3b_mbpp_nobuf_lr6e6` | `Qwen/Qwen2.5-3B` | no buffer | 6e-6 | 30-300 step 30 |
+| `qwen3b_mbpp_nobuf_lr6e7` | `Qwen/Qwen2.5-3B` | no buffer | 6e-7 | 30-300 step 30 |
+
+```python
+from transformers import AutoModelForCausalLM
+m = AutoModelForCausalLM.from_pretrained(
+        "RL-Forgetting-Experiments-3/code-rl-lr-ablation", subfolder="qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr3e6/global_step_500")
+```
+
+## Training
+
+- Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical
+  test split (276), so both stay reportable.
+- GRPO, 8 rollouts per prompt, batch 64, response length 3072; binary reward from
+  executing the generated program against the task's asserts.
+- Replay arms additionally replay 128 past rollouts per step at weight
+  lambda=0.1, sampled by `hard_cooldown`.
+- Everything except the actor lr is identical to the 1e-6 runs.
+
+## Evaluation
+
+MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased
+pass@k. Scoring uses MBPP+'s full test suite (~108 tests/problem), which is
+markedly stricter than MBPP's original 3 asserts.