---
license: apache-2.0
tags: [reinforcement-learning, grpo, code-generation, mbpp, catastrophic-forgetting, ablation]
---

# Code RL learning-rate ablation (MBPP)

GRPO on MBPP at **three actor learning rates below the 1e-6 default** -- 6e-6,
3e-6 and 6e-7 -- each run twice: once with no replay, once with an SFT-replay
buffer (`hard_cooldown`, lambda=0.1). The question is where on the lr ladder the
run stops being stable, and whether replay moves that point.

The lr=1e-6 headline runs live in their own repos
(`RL-Forgetting-Experiments-3/code-q3_1p7b-nobuf`, `-replay-lam0p1`, `RL-Forgetting-Experiments-3/code-q25_3b-*`);
above this ladder, 1e-5 is a coin flip and 1e-4 / 1e-3 are reliably fatal for
both arms.

## Layout

One folder per ablation cell, named exactly as the local run directory, so a
point in any eval CSV maps to a folder here without a lookup table:

```
<model>_mbpp_<arm>_<lr>/
    global_step_<N>/     a complete HF model directory (stride 30 + endpoint)
    logs/                trainer logs
    rollouts/            sampled rollouts
```

| folder | base model | arm | lr | checkpoints |
| --- | --- | --- | --- | --- |
| `qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr6e6` | `Qwen/Qwen3-1.7B-Base` | replay buffer (lambda=0.1) | 6e-6 | 30-500 step 30 |
| `qwen3_1p7b_base_mbpp_nobuf_lr6e7` | `Qwen/Qwen3-1.7B-Base` | no buffer | 6e-7 | 30-450 step 30 |

```python
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
        "RL-Forgetting-Experiments-3/code-rl-lr-ablation", subfolder="qwen3_1p7b_base_mbpp_hardcd_lam0p1_lr6e6/global_step_500")
```

## Training

- Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical
  test split (276), so both stay reportable.
- GRPO, 8 rollouts per prompt, batch 64, response length 3072; binary reward from
  executing the generated program against the task's asserts.
- Replay arms additionally replay 128 past rollouts per step at weight
  lambda=0.1, sampled by `hard_cooldown`.
- Everything except the actor lr is identical to the 1e-6 runs.

## Evaluation

MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased
pass@k. Scoring uses MBPP+'s full test suite (~108 tests/problem), which is
markedly stricter than MBPP's original 3 asserts.