README.md

---
license: llama3.1
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
tags:
  - education
  - mathematics
  - tutoring
  - socratic
  - lora
language:
  - en
---

# qlm-math-tutor

A LoRA adapter over Llama-3.1-8B-Instruct, fine-tuned for Socratic mathematics
tutoring: asking questions that advance a student's reasoning rather than
supplying answers.

**Read the Limitations section before using this model.** It documents a large
and consequential gap between this model's conversational tutoring behaviour and
its diagnostic accuracy. That gap is the most important fact about this model.

## Model details

| Field | Value |
|---|---|
| Base model | `meta-llama/Llama-3.1-8B-Instruct` |
| Method | LoRA (PEFT) |
| **Published adapter (SFT stage)** | **r=16, alpha=32, dropout=0.05; targets q/k/v/o_proj** |
| Final training loss | 0.3056 |
| Training hardware | 1× NVIDIA L4 (24 GB) |
| Training time | ~4 hours (14,394 s) |
| License | Llama 3.1 Community License |

### Training stages — read this before citing hyperparameters

Two stages exist and they use different LoRA configurations. **The weights in
this repository are the SFT checkpoint.**

| Stage | LoRA config | Target modules | Published here? |
|---|---|---|---|
| SFT | r=16, alpha=32 | q/k/v/o_proj | **Yes** |
| DPO | r=32, alpha=64 | q/k/v/o/gate/up/down_proj | No |

A DPO stage was trained on top of the SFT checkpoint (Bridge dataset, MIT
licence; beta=0.1, lr=5e-6, 3 epochs, effective batch 16, max length 2048,
85/15 train/holdout, seed 20260710). **No post-DPO evaluation results exist.**
Every number reported on this card is the SFT checkpoint. Do not attribute any
metric here to the DPO model.

## Training data

The published checkpoint was trained on a purpose-built set of LLM-generated
Socratic tutoring examples. No real student data was used at any stage.

### Training set for this checkpoint

| Field | Value |
|---|---|
| Training examples | 20,000 |
| Validation examples | 2,000 |
| Design | 10 math topics × 4 grade bands × 5 error types × 100 variations |
| Generation method | LLM-generated targeted Socratic responses |
| Quality control | Automated per-example scoring gate |
| Final training loss | 0.3056 |

The design is factorial rather than sampled: coverage is systematic across topic,
grade band, and error type, with 100 variations per cell. This is a deliberate
curriculum-coverage construction, not a subsample of a larger pool.

### A separate synthetic corpus also exists

| Field | Value |
|---|---|
| Total interactions | 534,536 |
| Split | 427,507 / 53,355 / 53,674 (80/10/10) |
| Simulated students | 10,000 |
| Sessions | 37,647 |
| Mean interactions per session | ~14.2 |
| Provenance | `pipeline_summary.json`, commit `897e2f2d`, 2026-05-17 |

This is an earlier, simulation-based generation run. **It is not the training
data for this checkpoint**, and the two corpora should not be described as one
dataset.

### Availability

The corpus files (`train.jsonl` 413 MB, `val.jsonl` 52 MB, `test.jsonl` 52 MB,
`students.jsonl` 5.6 MB) are retrievable from this repository's git history at
commit `69f74e7a`; Hugging Face retains LFS objects from deleted commits. Access
requires accepting the Llama 3.1 licence gate — unauthenticated requests return
401. The data is synthetic throughout and contains no real student records.

Note that this availability is incidental rather than curated: the files carry no
dataset card and are not discoverable from the repository's file listing. Treat
them as inspectable for verification, not as a maintained public dataset.

## Evaluation

All results are the SFT checkpoint. Strong and weak figures appear in the same
table at the same precision, deliberately.

### MathTutorBench

| Task | Metric | Score |
|---|---|---|
| `pedagogy_following` | match | **1.000** |
| `pedagogy_following_hard` | match | **1.000** |
| `scaffolding_generation` | match | **1.000** |
| `scaffolding_generation_hard` | match | **0.997** |
| `solution_correctness` | recall | 0.783 |
| `solution_correctness` | F1 | 0.632 |
| `solution_correctness` | accuracy | 0.543 |
| `mistake_location` | F1 macro | **0.086** |
| `mistake_correction` | accuracy | **0.044** |

**The two figures in bold at the bottom are the ones that matter for deployment
decisions.** This model scores at or near ceiling on tasks measuring whether its
output *looks like* good tutoring, and 0.086 / 0.044 on tasks measuring whether
it can locate and correct a student's actual mathematical error. That is roughly
a twenty-three-fold gap between form and diagnostic substance. Any single
composite "tutoring quality" score computed over these tasks will conceal it.

### Behavioural evaluation

| Metric | Score | 95% CI | n |
|---|---|---|---|
| Socratic question rate | 100% | [98%, 100%] | 200 |
| Answer avoidance rate | 96% | [92%, 98%] | 200 |
| Grade-appropriate language | 100% | [98%, 100%] | 200 |
| Relevance to the specific student error | 74.5% | [68%, 80%] | 200 |
| Answer leak rate | 1% | [0.2%, 5.4%] | 100 |

Relevance to the specific error (74.5%) is the figure to weigh against the
ceiling scores above: roughly one response in four does not engage the error the
student actually made.

### Integration ablation

Injecting per-turn diagnostic labels from an external misconception classifier
into this model's prompt produced **no measurable improvement** in correction
match rate: 60% bare versus 55% augmented, on 100 paired math word problems.

Stated limitations of that result: the judge was automated string matching
against a reference correction; no confidence interval or significance test was
computed, and at n=100 a 5-point difference is within sampling error; the run was
not repeated and no seed variation was explored (seed 20260710). **This result
should not be cited as evidence that label injection degrades performance.** It
supports only the weaker claim that per-turn injection of a noisy signal did not
help at this scale.

## Limitations

1. **Diagnostic accuracy is poor.** 0.086 F1 on mistake location and 0.044
   accuracy on mistake correction. The model produces well-formed Socratic
   tutoring turns while frequently failing to identify what the student got
   wrong.
2. **Form-substance gap.** Ceiling scores on pedagogy-following and scaffolding
   coexist with the figures above. Evaluating this model on conversational
   quality alone will substantially overstate its instructional value.
3. **Training data is LLM-generated, not merely synthetic.** The examples were
   produced by a language model and filtered by automated scoring. The model
   therefore inherits its generator's conception of what a good Socratic prompt
   is, along with that generator's phrasing habits and blind spots. A tutor
   trained this way can reproduce one model's notion of good pedagogy rather than
   pedagogical ground truth, and no human expert validated the training targets.
4. **Coverage is systematic but bounded.** 20,000 examples across a 10 × 4 × 5
   grid gives 100 variations per cell. Authentic classroom variation in student
   error expression, language background, and dialogue structure substantially
   exceeds this, and generalisation beyond the grid is unmeasured.
5. **No real student data at any stage**, and no claim is made that the generated
   distribution matches authentic classroom language, error patterns, or student
   behaviour.
6. **Corpus availability is incidental.** The training files are retrievable from
   deleted commits behind the Llama licence gate, without a dataset card and
   without discoverability. Third parties can inspect the data but should not
   treat it as a maintained release.
7. **No post-DPO evaluation.** A DPO stage was trained; its results were never
   recorded. All metrics here are pre-DPO.
8. **No human evaluation.** Planned, not performed. All behavioural metrics are
   automated.
9. **No classroom or longitudinal validation.** No evidence exists that this
   model affects learning outcomes.
10. **No adversarial safety results for this model.** The SafeTutors probe suite
   (11 harm dimensions, 37 sub-risks, 24 probe scripts) has been executed
   against Llama 3.3 70B, not against this model. Its robustness to sycophancy,
   answer extraction, emotional pressure, and frame-shift attacks is unmeasured.
11. **English and mathematics only.** No evaluation outside English-language K-12
    mathematics.
12. **Inherits base-model limitations.** All known Llama-3.1-8B-Instruct
    limitations apply, including arithmetic and multi-step reasoning errors.

## Out-of-scope uses

- **Unsupervised student-facing deployment.** Limitations 1, 2, and 9 together
  make this unsuitable without a human educator in the loop.
- **Grading, assessment, or any evaluative judgment about a student.** The model
  cannot reliably locate errors; using it to judge students would produce
  incorrect and unaccountable determinations.
- **Evaluating or ranking teachers.**
- **Any high-stakes decision** about placement, promotion, intervention, or
  diagnosis.
- **Non-mathematics subjects**, or non-English instruction.
- **A source of mathematical truth.** The model is tuned to ask rather than
  answer, and its solution accuracy is 0.543.

## Intended uses

Research on Socratic tutoring behaviour; a baseline for pedagogical evaluation
benchmarks; teacher-supervised practice where an educator reviews the
interaction; and a starting checkpoint for further fine-tuning — particularly on
authentic data, given limitation 4.

## Citation

```
@misc{qlm-math-tutor-2026,
  title  = {qlm-math-tutor: a Socratic mathematics tutoring adapter for Llama-3.1-8B},
  author = {Srivastava, Kumar Sumbhav},
  year   = {2026},
  note   = {Quantum Learning Machines, Inc. LoRA adapter, SFT checkpoint.},
  url    = {https://huggingface.co/QuantumLearningMachines/qlm-math-tutor}
}
```