| --- |
| license: apache-2.0 |
| tags: [reinforcement-learning, grpo, code-generation, mbpp, catastrophic-forgetting, ablation] |
| --- |
| |
| # Code RL learning-rate ablation (MBPP) |
|
|
| GRPO on MBPP at **three actor learning rates below the 1e-6 default** -- 6e-6, |
| 3e-6 and 6e-7 -- each run twice: once with no replay, once with an SFT-replay |
| buffer (`hard_cooldown`, lambda=0.1). The question is where on the lr ladder the |
| run stops being stable, and whether replay moves that point. |
|
|
| The lr=1e-6 headline runs live in their own repos |
| (`RL-Forgetting-Experiments-3/code-q3_1p7b-nobuf`, `-replay-lam0p1`, `RL-Forgetting-Experiments-3/code-q25_3b-*`); |
| above this ladder, 1e-5 is a coin flip and 1e-4 / 1e-3 are reliably fatal for |
| both arms. |
|
|
| ## Layout |
|
|
| One folder per ablation cell, named exactly as the local run directory, so a |
| point in any eval CSV maps to a folder here without a lookup table: |
|
|
| ``` |
| <model>_mbpp_<arm>_<lr>/ |
| global_step_<N>/ a complete HF model directory (stride 30 + endpoint) |
| logs/ trainer logs |
| rollouts/ sampled rollouts |
| ``` |
|
|
| | folder | base model | arm | lr | checkpoints | |
| | --- | --- | --- | --- | --- | |
| | `qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr6e6` | `Qwen/Qwen3-1.7B-Base` | replay buffer (lambda=0.1) | 6e-6 | 30-500 step 30 | |
| | `qwen3_1p7b_base_mbpp_nobuf_lr6e7` | `Qwen/Qwen3-1.7B-Base` | no buffer | 6e-7 | 30-450 step 30 | |
|
|
| ```python |
| from transformers import AutoModelForCausalLM |
| m = AutoModelForCausalLM.from_pretrained( |
| "RL-Forgetting-Experiments-3/code-rl-lr-ablation", subfolder="qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr6e6/global_step_500") |
| ``` |
|
|
| ## Training |
|
|
| - Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical |
| test split (276), so both stay reportable. |
| - GRPO, 8 rollouts per prompt, batch 64, response length 3072; binary reward from |
| executing the generated program against the task's asserts. |
| - Replay arms additionally replay 128 past rollouts per step at weight |
| lambda=0.1, sampled by `hard_cooldown`. |
| - Everything except the actor lr is identical to the 1e-6 runs. |
|
|
| ## Evaluation |
|
|
| MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased |
| pass@k. Scoring uses MBPP+'s full test suite (~108 tests/problem), which is |
| markedly stricter than MBPP's original 3 asserts. |
| |
