metadata
license: apache-2.0
base_model: Qwen/Qwen2.5-3B
tags:
- reinforcement-learning
- grpo
- code-generation
- mbpp
- catastrophic-forgetting
code-q25_3b-nobuf
GRPO on MBPP from Qwen/Qwen2.5-3B. Arm: GRPO, no replay buffer.
Part of a study of forgetting during code RL: the same run is trained with and without an SFT-replay buffer, and evaluated on MBPP+ across training.
Layout
One subfolder per checkpoint, global_step_<N>/, each a full Hugging Face model
directory. Steps are the stride-30 evaluation grid plus the run's endpoint
(30, 60, 90, 120, 150, 180, 210, 240, 270, 300).
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("RL-Forgetting-Experiments-3/code-q25_3b-nobuf", subfolder="global_step_300")
t = AutoTokenizer.from_pretrained("RL-Forgetting-Experiments-3/code-q25_3b-nobuf", subfolder="global_step_300")
Training
- Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical test split (276), so both remain reportable.
- GRPO, 8 rollouts per prompt, batch 64, actor lr 1e-6, response length 3072.
- Reward: execution of the generated program against the task's asserts (binary).
- Buffer arms replay 128 past rollouts per step with weight lambda=0.1, sampled
by
hard_cooldown.
Evaluation
MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased pass@k. Note that scoring against MBPP+'s full test suite is substantially stricter than MBPP's original 3 asserts (~8-10 points of pass@1).