README.md

math_square_README.md

@@ -0,0 +1,169 @@
+---
+license: other
+license_name: mixed-base-model-licenses
+library_name: peft
+base_model:
+- Qwen/Qwen2.5-1.5B-Instruct
+- meta-llama/Llama-3.2-1B-Instruct
+datasets:
+- openai/gsm8k
+- jeggers/competition_math
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- lora
+- sft
+- dpo
+- grpo
+- post-training
+- mathematical-reasoning
+- gsm8k
+- small-language-models
+---
+
+# SFT vs DPO vs GRPO on small models for math reasoning
+
+LoRA checkpoints for the paper
+
+> **A Learning-Rate×β Collapse Boundary for DPO and Compute-Bounded Inertness of GRPO in Small-Model Mathematical Reasoning**
+> B. Senthilnayaki, **Niteesh Kumar S K**, Rahul D. · St. Joseph's Institute of Technology, Chennai
+> 2026 3rd International Conference on Computing and Data Science (ICCDS), Chennai, 9–10 Oct 2026 · technically sponsored by the IEEE Madras Section
+> 🏆 **Best Paper Award**
+
+The study asks a narrow question: once a small model has been supervised fine-tuned on math, do preference optimisation (DPO) or reinforcement learning (GRPO) add anything? Every comparison starts DPO and GRPO from the **same SFT checkpoint**, so neither method is credited with gains SFT already made. Each setting runs with three seeds and is replicated on a second model family.
+
+## Key findings
+
+- **SFT does the work on GSM8K.** It lifts Qwen2.5-1.5B-Instruct from 36.4% to 48.2% and Llama-3.2-1B-Instruct from 19.3% to 34.8%.
+- **DPO never beat its SFT starting point, and the damage follows a boundary.** Accuracy loss grows with learning rate and shrinks with β: about −1 point at learning rate 1e-6, but **−11.4 points** at 5e-6 with β = 0.1 (Qwen, GSM8K). Llama shows the same direction (−5.8). The collapsed models also wrote about 40% shorter solutions (121 → 72 tokens on average).
+- **GRPO is inert at this compute budget.** With 250 steps, group size 4 and a binary correctness reward, GRPO moved accuracy by at most 0.6 points on average, and changing the LoRA rank from 4 to 32 did not change that.
+- **On MATH-numeric, no fine-tuned variant beat the instruct base model.** We report this rather than hide it (see *Limitations*).
+
+## Results
+
+Accuracy (%), greedy decoding. Fine-tuned rows are mean ± s.d. over seeds 42, 1337 and 2024; the base row is a single deterministic run. Each model is evaluated on the dataset it was trained on.
+
+| Model | Eval set | Base (instruct) | SFT | SFT → DPO (lr 5e-6, β 0.1) | SFT → DPO (lr 1e-6, β 0.1) | SFT → GRPO |
+|---|---|---:|---:|---:|---:|---:|
+| Qwen2.5-1.5B | GSM8K | 36.4 | **48.2 ± 0.8** | 36.8 ± 0.6 | 47.3 ± 1.2 | 48.1 ± 1.1 |
+| Qwen2.5-1.5B | MATH-numeric | **31.9** | 23.8 ± 0.4 | 22.0 ± 0.3 | 23.2 ± 0.2 | 23.8 ± 0.4 |
+| Llama-3.2-1B | GSM8K | 19.3 | 34.8 ± 0.1 | 29.0 ± 0.4 | 34.2 ± 0.5 | **35.4 ± 1.0** |
+| Llama-3.2-1B | MATH-numeric | **22.7** | 19.0 ± 0.2 | 16.7 ± 0.5 | 18.5 ± 0.4 | 19.1 ± 0.2 |
+
+### The DPO collapse boundary
+
+Change in GSM8K accuracy relative to SFT (percentage points), Qwen2.5-1.5B, mean of three seeds:
+
+| learning rate \ β | 0.1 | 0.2 | 0.5 |
+|---|---:|---:|---:|
+| 1e-6 | −1.0 | −0.7 | −0.6 |
+| 3e-6 | −3.3 | −2.6 | −1.6 |
+| 5e-6 | **−11.4** | −8.6 | −5.5 |
+
+### GRPO is inert across LoRA ranks
+
+GSM8K accuracy, Qwen2.5-1.5B, seed 42 (SFT starting point: 49.1%):
+
+| LoRA rank | 4 | 8 | 16 | 32 |
+|---|---:|---:|---:|---:|
+| SFT → GRPO | 48.7 | 48.1 | 49.2 | 48.1 |
+
+## Setup
+
+| | |
+|---|---|
+| Base models | `Qwen/Qwen2.5-1.5B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct` |
+| Adapters | LoRA r = 16, α = 32, dropout 0.05 on `q/k/v/o_proj`; bf16 |
+| Batch / schedule | effective batch 32, cosine schedule, 3% warm-up, max length 1,024 |
+| SFT | 3 epochs, learning rate 2e-5 |
+| DPO | 1 epoch, learning rate 5e-6 (swept 1e-6 / 3e-6 / 5e-6), β 0.1 (swept 0.2 / 0.5); on-policy preference pairs sampled from the seed-42 SFT model, 8 samples per prompt |
+| GRPO | 250 steps, learning rate 1e-6, group size 4, binary correctness reward, KL coefficient 0.04 |
+| Seeds | 42, 1337, 2024 |
+| Hardware | one NVIDIA L40S (48 GB) |
+
+**Data and difficulty tiers.**
+- **GSM8K:** the full 1,319-problem test set, tiered by the number of calculator steps in the reference solution (≤3: 794 problems; 4–5: 434; ≥6: 91).
+- **MATH-numeric:** the numeric-answer subset of competition MATH (`jeggers/competition_math`, `numeric` config), with 4,866 training and 3,199 test problems, tiered by official level (1–2: 939; 3: 740; 4–5: 1,520).
+
+**Evaluation.** Greedy decoding with at most 512 new tokens. Final answers are extracted from `\boxed{}`, `#### `, "the answer is", or the last number. A problem counts as correct only on an exact match after numeric normalisation.
+
+**Prompt.** Every model uses the same chat-formatted prompt. The system message reads: *"You are a careful math tutor. Solve the problem step by step, then give the final numeric answer after '#### '."*
+
+## Using the checkpoints
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+
+REPO = "NiteeshSK/math_square"
+BASE = "Qwen/Qwen2.5-1.5B-Instruct"
+
+tok = AutoTokenizer.from_pretrained(BASE)
+
+# SFT and DPO adapters sit on the original base model
+base = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.bfloat16, device_map="auto")
+model = PeftModel.from_pretrained(base, REPO, subfolder="qwen/checkpoints/sft_gsm8k_seed42")
+
+# GRPO adapters sit on the merged SFT weights instead:
+# base = AutoModelForCausalLM.from_pretrained(REPO, subfolder="qwen/checkpoints/sft_gsm8k_seed42_merged",
+#                                             torch_dtype=torch.bfloat16, device_map="auto")
+# model = PeftModel.from_pretrained(base, REPO, subfolder="qwen/checkpoints/grpo_gsm8k_seed42")
+
+messages = [
+    {"role": "system", "content": "You are a careful math tutor. Solve the problem step by step, "
+                                  "then give the final numeric answer after '#### '."},
+    {"role": "user", "content": "A shop sells pens at $3 each. How much do 7 pens cost?"},
+]
+inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+out = model.generate(inputs, max_new_tokens=512, do_sample=False)
+print(tok.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
+```
+
+## Repository layout
+
+```
+qwen/checkpoints/
+  sft_{gsm8k,math_num}_seed{42,1337,2024}          LoRA adapters on Qwen2.5-1.5B-Instruct
+  sft_{gsm8k,math_num}_seed{…}_merged              full SFT weights, merged in fp32
+  dpo_{gsm8k,math_num}_seed{…}                     DPO, lr 5e-6, β 0.1
+  dpo_{gsm8k,math_num}_seed{…}_lr{1e6,3e6,5e6}[_beta{02,05}]   learning-rate × β sweep
+  grpo_{gsm8k,math_num}_seed{…}                    GRPO adapters on the matching *_merged model
+qwen/runs/grpo_rank{4,8,32}_{gsm8k,math_num}       GRPO LoRA-rank ablation (seed 42)
+llama/checkpoints/…                                same structure for Llama-3.2-1B-Instruct
+```
+
+Each adapter's `adapter_config.json` names the base it must be loaded on.
+
+## Reproducibility notes
+
+- **fp32 merging.** LoRA updates are merged into the base weights in fp32 before GRPO. Merging in bf16 can round away the small low-rank update and leave a model that is partly untrained.
+- **Untrained-model check.** An automated check flags any "trained" model whose accuracy and per-tier answer lengths match the untrained base model. That would mean the training had silently not taken effect.
+- **Single source of truth.** One prompt template, one answer extractor and one evaluation script are shared by every method and seed.
+
+Training and evaluation code and per-example evaluation records are in the companion repository `NiteeshSK/sft-dpo-grpo-math-project` (access on request).
+
+## Limitations
+
+- **MATH-numeric regresses after fine-tuning.** Both instruct base models score higher on MATH-numeric than any fine-tuned variant: Qwen falls from 31.9% to 23.8% after SFT, and Llama from 22.7% to 19.0%. Fine-tuning also shortened solutions: Qwen's average answer length per difficulty tier fell from roughly 310–450 tokens to 120–240. The study's conclusions concern what DPO and GRPO add *on top of* SFT, not whether SFT beats an instruct model on MATH.
+- **The hardest GSM8K tier is small.** It has 91 problems, so differences within that tier are directional only, not individually significant.
+- **Limited scale.** The study covers two small models (1–1.5B parameters), attention-only LoRA, greedy decoding and a single GRPO compute budget. "Inert" means inert *at this budget*.
+
+## Citation
+
+```bibtex
+@inproceedings{senthilnayaki2026dpocollapse,
+  title     = {A Learning-Rate$\times\beta$ Collapse Boundary for {DPO} and Compute-Bounded
+               Inertness of {GRPO} in Small-Model Mathematical Reasoning},
+  author    = {B. Senthilnayaki and Niteesh Kumar S K and Rahul D.},
+  booktitle = {2026 3rd International Conference on Computing and Data Science (ICCDS)},
+  address   = {Chennai, India},
+  year      = {2026},
+  note      = {Best Paper Award}
+}
+```
+
+## Licence
+
+The adapters inherit their base models' licences: Apache 2.0 for the Qwen2.5-1.5B-Instruct adapters, and the Llama 3.2 Community License for the Llama-3.2-1B-Instruct adapters.