Create README.md

344a2d3 verified 1 day ago

9.22 kB

license: other
license_name: mixed-base-model-licenses
library_name: peft
base_model:
  - Qwen/Qwen2.5-1.5B-Instruct
  - meta-llama/Llama-3.2-1B-Instruct
datasets:
  - openai/gsm8k
  - jeggers/competition_math
language:
  - en
pipeline_tag: text-generation
tags:
  - lora
  - sft
  - dpo
  - grpo
  - post-training
  - mathematical-reasoning
  - gsm8k
  - small-language-models

SFT vs DPO vs GRPO on small models for math reasoning

LoRA checkpoints for the paper

A Learning-Rate×β Collapse Boundary for DPO and Compute-Bounded Inertness of GRPO in Small-Model Mathematical Reasoning B. Senthilnayaki, Niteesh Kumar S K, Rahul D. · St. Joseph's Institute of Technology, Chennai 2026 3rd International Conference on Computing and Data Science (ICCDS), Chennai, 9–10 Oct 2026 · technically sponsored by the IEEE Madras Section 🏆 Best Paper Award

The study asks a narrow question: once a small model has been supervised fine-tuned on math, do preference optimisation (DPO) or reinforcement learning (GRPO) add anything? Every comparison starts DPO and GRPO from the same SFT checkpoint, so neither method is credited with gains SFT already made. Each setting runs with three seeds and is replicated on a second model family.

Key findings

SFT does the work on GSM8K. It lifts Qwen2.5-1.5B-Instruct from 36.4% to 48.2% and Llama-3.2-1B-Instruct from 19.3% to 34.8%.
DPO never beat its SFT starting point, and the damage follows a boundary. Accuracy loss grows with learning rate and shrinks with β: about −1 point at learning rate 1e-6, but −11.4 points at 5e-6 with β = 0.1 (Qwen, GSM8K). Llama shows the same direction (−5.8). The collapsed models also wrote about 40% shorter solutions (121 → 72 tokens on average).
GRPO is inert at this compute budget. With 250 steps, group size 4 and a binary correctness reward, GRPO moved accuracy by at most 0.6 points on average, and changing the LoRA rank from 4 to 32 did not change that.
On MATH-numeric, no fine-tuned variant beat the instruct base model. We report this rather than hide it (see Limitations).

Results

Accuracy (%), greedy decoding. Fine-tuned rows are mean ± s.d. over seeds 42, 1337 and 2024; the base row is a single deterministic run. Each model is evaluated on the dataset it was trained on.

Model	Eval set	Base (instruct)	SFT	SFT → DPO (lr 5e-6, β 0.1)	SFT → DPO (lr 1e-6, β 0.1)	SFT → GRPO
Qwen2.5-1.5B	GSM8K	36.4	48.2 ± 0.8	36.8 ± 0.6	47.3 ± 1.2	48.1 ± 1.1
Qwen2.5-1.5B	MATH-numeric	31.9	23.8 ± 0.4	22.0 ± 0.3	23.2 ± 0.2	23.8 ± 0.4
Llama-3.2-1B	GSM8K	19.3	34.8 ± 0.1	29.0 ± 0.4	34.2 ± 0.5	35.4 ± 1.0
Llama-3.2-1B	MATH-numeric	22.7	19.0 ± 0.2	16.7 ± 0.5	18.5 ± 0.4	19.1 ± 0.2

The DPO collapse boundary

Change in GSM8K accuracy relative to SFT (percentage points), Qwen2.5-1.5B, mean of three seeds:

learning rate \ β	0.1	0.2	0.5
1e-6	−1.0	−0.7	−0.6
3e-6	−3.3	−2.6	−1.6
5e-6	−11.4	−8.6	−5.5

GRPO is inert across LoRA ranks

GSM8K accuracy, Qwen2.5-1.5B, seed 42 (SFT starting point: 49.1%):

LoRA rank	4	8	16	32
SFT → GRPO	48.7	48.1	49.2	48.1

Setup


Base models	`Qwen/Qwen2.5-1.5B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`
Adapters	LoRA r = 16, α = 32, dropout 0.05 on `q/k/v/o_proj`; bf16
Batch / schedule	effective batch 32, cosine schedule, 3% warm-up, max length 1,024
SFT	3 epochs, learning rate 2e-5
DPO	1 epoch, learning rate 5e-6 (swept 1e-6 / 3e-6 / 5e-6), β 0.1 (swept 0.2 / 0.5); on-policy preference pairs sampled from the seed-42 SFT model, 8 samples per prompt
GRPO	250 steps, learning rate 1e-6, group size 4, binary correctness reward, KL coefficient 0.04
Seeds	42, 1337, 2024
Hardware	one NVIDIA L40S (48 GB)

Data and difficulty tiers.

GSM8K: the full 1,319-problem test set, tiered by the number of calculator steps in the reference solution (≤3: 794 problems; 4–5: 434; ≥6: 91).
MATH-numeric: the numeric-answer subset of competition MATH (jeggers/competition_math, numeric config), with 4,866 training and 3,199 test problems, tiered by official level (1–2: 939; 3: 740; 4–5: 1,520).

Evaluation. Greedy decoding with at most 512 new tokens. Final answers are extracted from \boxed{}, #### , "the answer is", or the last number. A problem counts as correct only on an exact match after numeric normalisation.

Prompt. Every model uses the same chat-formatted prompt. The system message reads: "You are a careful math tutor. Solve the problem step by step, then give the final numeric answer after '#### '."

Using the checkpoints

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

REPO = "NiteeshSK/math_square"
BASE = "Qwen/Qwen2.5-1.5B-Instruct"

tok = AutoTokenizer.from_pretrained(BASE)

# SFT and DPO adapters sit on the original base model
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, REPO, subfolder="qwen/checkpoints/sft_gsm8k_seed42")

# GRPO adapters sit on the merged SFT weights instead:
# base = AutoModelForCausalLM.from_pretrained(REPO, subfolder="qwen/checkpoints/sft_gsm8k_seed42_merged",
#                                             torch_dtype=torch.bfloat16, device_map="auto")
# model = PeftModel.from_pretrained(base, REPO, subfolder="qwen/checkpoints/grpo_gsm8k_seed42")

messages = [
    {"role": "system", "content": "You are a careful math tutor. Solve the problem step by step, "
                                  "then give the final numeric answer after '#### '."},
    {"role": "user", "content": "A shop sells pens at $3 each. How much do 7 pens cost?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))

Repository layout

qwen/checkpoints/
  sft_{gsm8k,math_num}_seed{42,1337,2024}          LoRA adapters on Qwen2.5-1.5B-Instruct
  sft_{gsm8k,math_num}_seed{…}_merged              full SFT weights, merged in fp32
  dpo_{gsm8k,math_num}_seed{…}                     DPO, lr 5e-6, β 0.1
  dpo_{gsm8k,math_num}_seed{…}_lr{1e6,3e6,5e6}[_beta{02,05}]   learning-rate × β sweep
  grpo_{gsm8k,math_num}_seed{…}                    GRPO adapters on the matching *_merged model
qwen/runs/grpo_rank{4,8,32}_{gsm8k,math_num}       GRPO LoRA-rank ablation (seed 42)
llama/checkpoints/…                                same structure for Llama-3.2-1B-Instruct

Each adapter's adapter_config.json names the base it must be loaded on.

Reproducibility notes

fp32 merging. LoRA updates are merged into the base weights in fp32 before GRPO. Merging in bf16 can round away the small low-rank update and leave a model that is partly untrained.
Untrained-model check. An automated check flags any "trained" model whose accuracy and per-tier answer lengths match the untrained base model. That would mean the training had silently not taken effect.
Single source of truth. One prompt template, one answer extractor and one evaluation script are shared by every method and seed.

Training and evaluation code and per-example evaluation records are in the companion repository NiteeshSK/sft-dpo-grpo-math-project (access on request).

Limitations

MATH-numeric regresses after fine-tuning. Both instruct base models score higher on MATH-numeric than any fine-tuned variant: Qwen falls from 31.9% to 23.8% after SFT, and Llama from 22.7% to 19.0%. Fine-tuning also shortened solutions: Qwen's average answer length per difficulty tier fell from roughly 310–450 tokens to 120–240. The study's conclusions concern what DPO and GRPO add on top of SFT, not whether SFT beats an instruct model on MATH.
The hardest GSM8K tier is small. It has 91 problems, so differences within that tier are directional only, not individually significant.
Limited scale. The study covers two small models (1–1.5B parameters), attention-only LoRA, greedy decoding and a single GRPO compute budget. "Inert" means inert at this budget.

Citation

@inproceedings{senthilnayaki2026dpocollapse,
  title     = {A Learning-Rate$\times\beta$ Collapse Boundary for {DPO} and Compute-Bounded
               Inertness of {GRPO} in Small-Model Mathematical Reasoning},
  author    = {B. Senthilnayaki and Niteesh Kumar S K and Rahul D.},
  booktitle = {2026 3rd International Conference on Computing and Data Science (ICCDS)},
  address   = {Chennai, India},
  year      = {2026},
  note      = {Best Paper Award}
}

Licence

The adapters inherit their base models' licences: Apache 2.0 for the Qwen2.5-1.5B-Instruct adapters, and the Llama 3.2 Community License for the Llama-3.2-1B-Instruct adapters.