model card

README.md

@@ -0,0 +1,39 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen2.5-3B
+tags: [reinforcement-learning, grpo, code-generation, mbpp, catastrophic-forgetting]
+---
+
+# code-q25_3b-nobuf
+
+GRPO on MBPP from `Qwen/Qwen2.5-3B`. Arm: **GRPO, no replay buffer**.
+
+Part of a study of forgetting during code RL: the same run is trained with and
+without an SFT-replay buffer, and evaluated on MBPP+ across training.
+
+## Layout
+
+One subfolder per checkpoint, `global_step_<N>/`, each a full Hugging Face model
+directory. Steps are the stride-30 evaluation grid plus the run's endpoint
+(30, 60, 90, 120, 150, 180, 210, 240, 270, 300).
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+m = AutoModelForCausalLM.from_pretrained("RL-Forgetting-Experiments-3/code-q25_3b-nobuf", subfolder="global_step_300")
+t = AutoTokenizer.from_pretrained("RL-Forgetting-Experiments-3/code-q25_3b-nobuf", subfolder="global_step_300")
+```
+
+## Training
+
+- Data: 320 MBPP problems, held out from both MBPP+ (378) and MBPP's canonical
+  test split (276), so both remain reportable.
+- GRPO, 8 rollouts per prompt, batch 64, actor lr 1e-6, response length 3072.
+- Reward: execution of the generated program against the task's asserts (binary).
+- Buffer arms replay 128 past rollouts per step with weight lambda=0.1, sampled
+  by `hard_cooldown`.
+
+## Evaluation
+
+MBPP+ (378 problems), n=160 samples, temperature 0.6, top_p 0.95, unbiased
+pass@k. Note that scoring against MBPP+'s full test suite is substantially
+stricter than MBPP's original 3 asserts (~8-10 points of pass@1).