math_square_README.md

---
license: other
license_name: mixed-base-model-licenses
library_name: peft
base_model:
- Qwen/Qwen2.5-1.5B-Instruct
- meta-llama/Llama-3.2-1B-Instruct
datasets:
- openai/gsm8k
- jeggers/competition_math
language:
- en
pipeline_tag: text-generation
tags:
- lora
- sft
- dpo
- grpo
- post-training
- mathematical-reasoning
- gsm8k
- small-language-models
---

# SFT vs DPO vs GRPO on small models for math reasoning

LoRA checkpoints for the paper

> **A Learning-Rate×β Collapse Boundary for DPO and Compute-Bounded Inertness of GRPO in Small-Model Mathematical Reasoning**
> B. Senthilnayaki, **Niteesh Kumar S K**, Rahul D. · St. Joseph's Institute of Technology, Chennai
> 2026 3rd International Conference on Computing and Data Science (ICCDS), Chennai, 9–10 Oct 2026 · technically sponsored by the IEEE Madras Section
> 🏆 **Best Paper Award**

The study asks a narrow question: once a small model has been supervised fine-tuned on math, do preference optimisation (DPO) or reinforcement learning (GRPO) add anything? Every comparison starts DPO and GRPO from the **same SFT checkpoint**, so neither method is credited with gains SFT already made. Each setting runs with three seeds and is replicated on a second model family.

## Key findings

- **SFT does the work on GSM8K.** It lifts Qwen2.5-1.5B-Instruct from 36.4% to 48.2% and Llama-3.2-1B-Instruct from 19.3% to 34.8%.
- **DPO never beat its SFT starting point, and the damage follows a boundary.** Accuracy loss grows with learning rate and shrinks with β: about −1 point at learning rate 1e-6, but **−11.4 points** at 5e-6 with β = 0.1 (Qwen, GSM8K). Llama shows the same direction (−5.8). The collapsed models also wrote about 40% shorter solutions (121 → 72 tokens on average).
- **GRPO is inert at this compute budget.** With 250 steps, group size 4 and a binary correctness reward, GRPO moved accuracy by at most 0.6 points on average, and changing the LoRA rank from 4 to 32 did not change that.
- **On MATH-numeric, no fine-tuned variant beat the instruct base model.** We report this rather than hide it (see *Limitations*).

## Results

Accuracy (%), greedy decoding. Fine-tuned rows are mean ± s.d. over seeds 42, 1337 and 2024; the base row is a single deterministic run. Each model is evaluated on the dataset it was trained on.

| Model | Eval set | Base (instruct) | SFT | SFT → DPO (lr 5e-6, β 0.1) | SFT → DPO (lr 1e-6, β 0.1) | SFT → GRPO |
|---|---|---:|---:|---:|---:|---:|
| Qwen2.5-1.5B | GSM8K | 36.4 | **48.2 ± 0.8** | 36.8 ± 0.6 | 47.3 ± 1.2 | 48.1 ± 1.1 |
| Qwen2.5-1.5B | MATH-numeric | **31.9** | 23.8 ± 0.4 | 22.0 ± 0.3 | 23.2 ± 0.2 | 23.8 ± 0.4 |
| Llama-3.2-1B | GSM8K | 19.3 | 34.8 ± 0.1 | 29.0 ± 0.4 | 34.2 ± 0.5 | **35.4 ± 1.0** |
| Llama-3.2-1B | MATH-numeric | **22.7** | 19.0 ± 0.2 | 16.7 ± 0.5 | 18.5 ± 0.4 | 19.1 ± 0.2 |

### The DPO collapse boundary

Change in GSM8K accuracy relative to SFT (percentage points), Qwen2.5-1.5B, mean of three seeds:

| learning rate \ β | 0.1 | 0.2 | 0.5 |
|---|---:|---:|---:|
| 1e-6 | −1.0 | −0.7 | −0.6 |
| 3e-6 | −3.3 | −2.6 | −1.6 |
| 5e-6 | **−11.4** | −8.6 | −5.5 |

### GRPO is inert across LoRA ranks

GSM8K accuracy, Qwen2.5-1.5B, seed 42 (SFT starting point: 49.1%):

| LoRA rank | 4 | 8 | 16 | 32 |
|---|---:|---:|---:|---:|
| SFT → GRPO | 48.7 | 48.1 | 49.2 | 48.1 |

## Setup

| | |
|---|---|
| Base models | `Qwen/Qwen2.5-1.5B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct` |
| Adapters | LoRA r = 16, α = 32, dropout 0.05 on `q/k/v/o_proj`; bf16 |
| Batch / schedule | effective batch 32, cosine schedule, 3% warm-up, max length 1,024 |
| SFT | 3 epochs, learning rate 2e-5 |
| DPO | 1 epoch, learning rate 5e-6 (swept 1e-6 / 3e-6 / 5e-6), β 0.1 (swept 0.2 / 0.5); on-policy preference pairs sampled from the seed-42 SFT model, 8 samples per prompt |
| GRPO | 250 steps, learning rate 1e-6, group size 4, binary correctness reward, KL coefficient 0.04 |
| Seeds | 42, 1337, 2024 |
| Hardware | one NVIDIA L40S (48 GB) |

**Data and difficulty tiers.**
- **GSM8K:** the full 1,319-problem test set, tiered by the number of calculator steps in the reference solution (≤3: 794 problems; 4–5: 434; ≥6: 91).
- **MATH-numeric:** the numeric-answer subset of competition MATH (`jeggers/competition_math`, `numeric` config), with 4,866 training and 3,199 test problems, tiered by official level (1–2: 939; 3: 740; 4–5: 1,520).

**Evaluation.** Greedy decoding with at most 512 new tokens. Final answers are extracted from `\boxed{}`, `#### `, "the answer is", or the last number. A problem counts as correct only on an exact match after numeric normalisation.

**Prompt.** Every model uses the same chat-formatted prompt. The system message reads: *"You are a careful math tutor. Solve the problem step by step, then give the final numeric answer after '#### '."*

## Using the checkpoints

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

REPO = "NiteeshSK/math_square"
BASE = "Qwen/Qwen2.5-1.5B-Instruct"

tok = AutoTokenizer.from_pretrained(BASE)

# SFT and DPO adapters sit on the original base model
base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, REPO, subfolder="qwen/checkpoints/sft_gsm8k_seed42")

# GRPO adapters sit on the merged SFT weights instead:
# base = AutoModelForCausalLM.from_pretrained(REPO, subfolder="qwen/checkpoints/sft_gsm8k_seed42_merged",
#                                             torch_dtype=torch.bfloat16, device_map="auto")
# model = PeftModel.from_pretrained(base, REPO, subfolder="qwen/checkpoints/grpo_gsm8k_seed42")

messages = [
    {"role": "system", "content": "You are a careful math tutor. Solve the problem step by step, "
                                  "then give the final numeric answer after '#### '."},
    {"role": "user", "content": "A shop sells pens at $3 each. How much do 7 pens cost?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
```

## Repository layout

```
qwen/checkpoints/
  sft_{gsm8k,math_num}_seed{42,1337,2024}          LoRA adapters on Qwen2.5-1.5B-Instruct
  sft_{gsm8k,math_num}_seed{…}_merged              full SFT weights, merged in fp32
  dpo_{gsm8k,math_num}_seed{…}                     DPO, lr 5e-6, β 0.1
  dpo_{gsm8k,math_num}_seed{…}_lr{1e6,3e6,5e6}[_beta{02,05}]   learning-rate × β sweep
  grpo_{gsm8k,math_num}_seed{…}                    GRPO adapters on the matching *_merged model
qwen/runs/grpo_rank{4,8,32}_{gsm8k,math_num}       GRPO LoRA-rank ablation (seed 42)
llama/checkpoints/…                                same structure for Llama-3.2-1B-Instruct
```

Each adapter's `adapter_config.json` names the base it must be loaded on.

## Reproducibility notes

- **fp32 merging.** LoRA updates are merged into the base weights in fp32 before GRPO. Merging in bf16 can round away the small low-rank update and leave a model that is partly untrained.
- **Untrained-model check.** An automated check flags any "trained" model whose accuracy and per-tier answer lengths match the untrained base model. That would mean the training had silently not taken effect.
- **Single source of truth.** One prompt template, one answer extractor and one evaluation script are shared by every method and seed.

Training and evaluation code and per-example evaluation records are in the companion repository `NiteeshSK/sft-dpo-grpo-math-project` (access on request).

## Limitations

- **MATH-numeric regresses after fine-tuning.** Both instruct base models score higher on MATH-numeric than any fine-tuned variant: Qwen falls from 31.9% to 23.8% after SFT, and Llama from 22.7% to 19.0%. Fine-tuning also shortened solutions: Qwen's average answer length per difficulty tier fell from roughly 310–450 tokens to 120–240. The study's conclusions concern what DPO and GRPO add *on top of* SFT, not whether SFT beats an instruct model on MATH.
- **The hardest GSM8K tier is small.** It has 91 problems, so differences within that tier are directional only, not individually significant.
- **Limited scale.** The study covers two small models (1–1.5B parameters), attention-only LoRA, greedy decoding and a single GRPO compute budget. "Inert" means inert *at this budget*.

## Citation

```bibtex
@inproceedings{senthilnayaki2026dpocollapse,
  title     = {A Learning-Rate$\times\beta$ Collapse Boundary for {DPO} and Compute-Bounded
               Inertness of {GRPO} in Small-Model Mathematical Reasoning},
  author    = {B. Senthilnayaki and Niteesh Kumar S K and Rahul D.},
  booktitle = {2026 3rd International Conference on Computing and Data Science (ICCDS)},
  address   = {Chennai, India},
  year      = {2026},
  note      = {Best Paper Award}
}
```

## Licence

The adapters inherit their base models' licences: Apache 2.0 for the Qwen2.5-1.5B-Instruct adapters, and the Llama 3.2 Community License for the Llama-3.2-1B-Instruct adapters.