Instructions to use Munchic/Qwen3.5-2B-Math-SFT-RL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Munchic/Qwen3.5-2B-Math-SFT-RL with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Qwen3.5-2B · MATH · SFT → RL with learned prompt selection
What we test: after SFT, many RL prompts are either always solved or never solved, and their rollouts teach the model nothing. We test whether choosing which prompts to train on, with a lightweight learned predictor that needs no extra rollouts, reaches higher accuracy in less time than plain RL.
Versions (each subfolder is a LoRA adapter on Qwen/Qwen3.5-2B @ 15852e8)
| Folder | What it does |
|---|---|
Qwen3.5-2B-SFT-Math |
SFT. 2 epochs on 2,000 MATH-hard worked solutions. Start point of every RL version. |
Qwen3.5-2B-SFT-Plain-RL-Math-s{0,1,2} |
SFT + Plain RL. GRPO with uniformly sampled prompts, 8 prompts × 8 rollouts per step. 3 seeds. |
Qwen3.5-2B-SFT-PromptSelect-RL-Math-s0 |
SFT + PromptSelect RL. Same GRPO setup, but the prompts for each step are chosen by our selector. 8 rollouts per prompt. |
Qwen3.5-2B-SFT-PromptSelect-RL-1Rollout-Math-s0 |
SFT + PromptSelect RL, 1 rollout. Same selector, but only one rollout per prompt (more prompts per step, fewer rollouts per step). |
Shared setup: LoRA r=16, α=32, fp32; thinking off; answers in \boxed{}; rollouts T=1, 1024-token cap.
Results
Final greedy accuracy on 1,000 held-out MATH questions (levels 3–5), 1024-token cap.
| Model | Acc. | L3 / L4 / L5 | Truncated | RL steps | RL train h | Rollouts | Stop |
|---|---|---|---|---|---|---|---|
| Base Qwen3.5-2B (no adapter) | 0.400 | 0.579 / 0.435 / 0.235 | 55.1% | – | – | – | – |
| SFT | 0.262 | 0.405 / 0.278 / 0.139 | 1.2% | – | (SFT 0.34 h) | – | – |
| SFT + Plain RL, s0 | 0.483 | 0.669 / 0.508 / 0.320 | 11.8% | 120 | 2.46 | 7,680 | max steps |
| SFT + Plain RL, s1 | 0.471 | 0.649 / 0.518 / 0.296 | 18.1% | 75 | 1.60 | 4,800 | length runaway |
| SFT + Plain RL, s2 | 0.441 | 0.629 / 0.466 / 0.276 | 14.3% | 61 | 1.38 | 3,904 | length runaway |
| Plain RL, mean of 3 | 0.465 | ||||||
| SFT + PromptSelect RL, s0 | 0.526 | 0.712 / 0.575 / 0.343 | 22.2% | 92 | 1.93 | 5,888 | length runaway |
| SFT + PromptSelect RL, 1 rollout, s0 | 0.541 | 0.736 / 0.578 / 0.361 | 19.2% | 168 | 1.98 | 4,872 | length runaway |
Accuracy on a fixed 500-question eval subset during training. Grey band = SFT; dashed line = where SFT ends and RL starts. Plain RL: mean of 3 seeds with a 95% t-interval (n = 3), seeds interpolated onto a common grid and shown only while all three run; the dotted line is seed 0 alone after seeds 1 and 2 stopped. Left: wall-clock time of each run, with the SFT run first; it includes the evals done during training, which take longer as answers get longer. Right: RL steps. Steps are not comparable across versions: a 1-rollout step uses fewer rollouts than an 8×8 step.
Reading (provisional): PromptSelect RL reached seed-0 Plain RL's final subset accuracy (0.476) in ≤ half the training time (evals excluded; selector cost included) and finished +0.043 higher on the 1,000-question eval (SE 0.014). PromptSelect 1-rollout reached the same 0.476 in about a third of that time and finished highest (0.541, +0.058 over seed-0 Plain RL), using 37% fewer rollouts. Both PromptSelect versions have one seed so far; Plain RL's seeds spread 0.441–0.483.
Caveats
- SFT lowers greedy accuracy versus the base model (0.262 vs 0.400). SFT teaches short worked solutions (≈160 tokens), and the base model gets its 0.400 with long answers, 55% of them cut off at the cap. RL then recovers and passes it.
- "Length runaway" = >15% of rollouts truncated at 1024 tokens for 5 consecutive steps. That stop ended most runs, and the fastest-learning versions lengthen fastest. Final accuracies are measured with 12–22% truncation.
- Training data: SimpleRL MATH-hard (levels 3–5), split by question across SFT / RL / eval; questions with
[asy]figures excluded.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
rev = "15852e8c16360a2fea060d615a32b45270f8a8fc"
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-2B", revision=rev, dtype="float32")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B", revision=rev)
model = PeftModel.from_pretrained(base, "Munchic/Qwen3.5-2B-Math-SFT-RL",
subfolder="Qwen3.5-2B-SFT-PromptSelect-RL-Math-s0")
# chat template with enable_thinking=False; the model ends its answer in \boxed{...}
Each subfolder's metadata.json holds the step, LoRA config and full eval breakdown.
- Downloads last month
- -
