Qwen3.5-2B · MATH · SFT → RL with learned prompt selection

What we test: after SFT, many RL prompts are either always solved or never solved, and their rollouts teach the model nothing. We test whether choosing which prompts to train on, with a lightweight learned predictor that needs no extra rollouts, reaches higher accuracy in less time than plain RL.

Versions (each subfolder is a LoRA adapter on Qwen/Qwen3.5-2B @ 15852e8)

Folder What it does
Qwen3.5-2B-SFT-Math SFT. 2 epochs on 2,000 MATH-hard worked solutions. Start point of every RL version.
Qwen3.5-2B-SFT-Plain-RL-Math-s{0,1,2} SFT + Plain RL. GRPO with uniformly sampled prompts, 8 prompts × 8 rollouts per step. 3 seeds.
Qwen3.5-2B-SFT-PromptSelect-RL-Math-s0 SFT + PromptSelect RL. Same GRPO setup, but the prompts for each step are chosen by our selector. 8 rollouts per prompt.
Qwen3.5-2B-SFT-PromptSelect-RL-1Rollout-Math-s0 SFT + PromptSelect RL, 1 rollout. Same selector, but only one rollout per prompt (more prompts per step, fewer rollouts per step).

Shared setup: LoRA r=16, α=32, fp32; thinking off; answers in \boxed{}; rollouts T=1, 1024-token cap.

Results

Final greedy accuracy on 1,000 held-out MATH questions (levels 3–5), 1024-token cap.

Model Acc. L3 / L4 / L5 Truncated RL steps RL train h Rollouts Stop
Base Qwen3.5-2B (no adapter) 0.400 0.579 / 0.435 / 0.235 55.1% – – – –
SFT 0.262 0.405 / 0.278 / 0.139 1.2% – (SFT 0.34 h) – –
SFT + Plain RL, s0 0.483 0.669 / 0.508 / 0.320 11.8% 120 2.46 7,680 max steps
SFT + Plain RL, s1 0.471 0.649 / 0.518 / 0.296 18.1% 75 1.60 4,800 length runaway
SFT + Plain RL, s2 0.441 0.629 / 0.466 / 0.276 14.3% 61 1.38 3,904 length runaway
Plain RL, mean of 3 0.465
SFT + PromptSelect RL, s0 0.526 0.712 / 0.575 / 0.343 22.2% 92 1.93 5,888 length runaway
SFT + PromptSelect RL, 1 rollout, s0 0.541 0.736 / 0.578 / 0.361 19.2% 168 1.98 4,872 length runaway

accuracy curve

Accuracy on a fixed 500-question eval subset during training. Grey band = SFT; dashed line = where SFT ends and RL starts. Plain RL: mean of 3 seeds with a 95% t-interval (n = 3), seeds interpolated onto a common grid and shown only while all three run; the dotted line is seed 0 alone after seeds 1 and 2 stopped. Left: wall-clock time of each run, with the SFT run first; it includes the evals done during training, which take longer as answers get longer. Right: RL steps. Steps are not comparable across versions: a 1-rollout step uses fewer rollouts than an 8×8 step.

Reading (provisional): PromptSelect RL reached seed-0 Plain RL's final subset accuracy (0.476) in ≤ half the training time (evals excluded; selector cost included) and finished +0.043 higher on the 1,000-question eval (SE 0.014). PromptSelect 1-rollout reached the same 0.476 in about a third of that time and finished highest (0.541, +0.058 over seed-0 Plain RL), using 37% fewer rollouts. Both PromptSelect versions have one seed so far; Plain RL's seeds spread 0.441–0.483.

Caveats

  • SFT lowers greedy accuracy versus the base model (0.262 vs 0.400). SFT teaches short worked solutions (≈160 tokens), and the base model gets its 0.400 with long answers, 55% of them cut off at the cap. RL then recovers and passes it.
  • "Length runaway" = >15% of rollouts truncated at 1024 tokens for 5 consecutive steps. That stop ended most runs, and the fastest-learning versions lengthen fastest. Final accuracies are measured with 12–22% truncation.
  • Training data: SimpleRL MATH-hard (levels 3–5), split by question across SFT / RL / eval; questions with [asy] figures excluded.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
rev = "15852e8c16360a2fea060d615a32b45270f8a8fc"
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-2B", revision=rev, dtype="float32")
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B", revision=rev)
model = PeftModel.from_pretrained(base, "Munchic/Qwen3.5-2B-Math-SFT-RL",
                                  subfolder="Qwen3.5-2B-SFT-PromptSelect-RL-Math-s0")
# chat template with enable_thinking=False; the model ends its answer in \boxed{...}

Each subfolder's metadata.json holds the step, LoRA config and full eval breakdown.

Downloads last month
-
Video Preview
loading

Model tree for Munchic/Qwen3.5-2B-Math-SFT-RL

Finetuned
Qwen/Qwen3.5-2B
Adapter
(420)
this model