README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-3B
tags: [reinforcement-learning, grpo, code-generation, mbpp, catastrophic-forgetting]
---

# code-q25_3b-nobuf

GRPO on MBPP from `Qwen/Qwen2.5-3B`. Arm: **GRPO, no replay buffer**.

Part of a study of forgetting during code RL: the same run is trained with and
without an SFT-replay buffer, and evaluated on MBPP+ across training.

## Layout

One subfolder per checkpoint, `global_step_<N>/`, each a full Hugging Face model
directory. Steps are the stride-30 evaluation grid plus the run's endpoint
(30, 60, 90, 120, 150, 180, 210, 240, 270, 300).

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("RL-Forgetting-Experiments-3/code-q25_3b-nobuf", subfolder="global_step_300")
t = AutoTokenizer.from_pretrained("RL-Forgetting-Experiments-3/code-q25_3b-nobuf", subfolder="global_step_300")
```

## Training

- Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical
  test split (276), so both remain reportable.
- GRPO, 8 rollouts per prompt, batch 64, actor lr 1e-6, response length 3072.
- Reward: execution of the generated program against the task's asserts (binary).
- Buffer arms replay 128 past rollouts per step with weight lambda=0.1, sampled
  by `hard_cooldown`.

## Evaluation

MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased
pass@k. Note that scoring against MBPP+'s full test suite is substantially
stricter than MBPP's original 3 asserts (~8-10 points of pass@1).