qlm-math-tutor

A LoRA adapter over Llama-3.1-8B-Instruct, fine-tuned for Socratic mathematics tutoring: asking questions that advance a student's reasoning rather than supplying answers.

Read the Limitations section before using this model. It documents a large and consequential gap between this model's conversational tutoring behaviour and its diagnostic accuracy. That gap is the most important fact about this model.

Model details

Field	Value
Base model	meta-llama/Llama-3.1-8B-Instruct
Method	LoRA (PEFT)
Published adapter (SFT stage)	r=16, alpha=32, dropout=0.05; targets q/k/v/o_proj
Final training loss	0.3056
Training hardware	1× NVIDIA L4 (24 GB)
Training time	~4 hours (14,394 s)
License	Llama 3.1 Community License

Training stages — read this before citing hyperparameters

Two stages exist and they use different LoRA configurations. The weights in this repository are the SFT checkpoint.

Stage	LoRA config	Target modules	Published here?
SFT	r=16, alpha=32	q/k/v/o_proj	Yes
DPO	r=32, alpha=64	q/k/v/o/gate/up/down_proj	No

A DPO stage was trained on top of the SFT checkpoint (Bridge dataset, MIT licence; beta=0.1, lr=5e-6, 3 epochs, effective batch 16, max length 2048, 85/15 train/holdout, seed 20260710). No post-DPO evaluation results exist. Every number reported on this card is the SFT checkpoint. Do not attribute any metric here to the DPO model.

Training data

The published checkpoint was trained on a purpose-built set of LLM-generated Socratic tutoring examples. No real student data was used at any stage.

Training set for this checkpoint

Field	Value
Training examples	20,000
Validation examples	2,000
Design	10 math topics × 4 grade bands × 5 error types × 100 variations
Generation method	LLM-generated targeted Socratic responses
Quality control	Automated per-example scoring gate
Final training loss	0.3056

The design is factorial rather than sampled: coverage is systematic across topic, grade band, and error type, with 100 variations per cell. This is a deliberate curriculum-coverage construction, not a subsample of a larger pool.

A separate synthetic corpus also exists

Field	Value
Total interactions	534,536
Split	427,507 / 53,355 / 53,674 (80/10/10)
Simulated students	10,000
Sessions	37,647
Mean interactions per session	~14.2
Provenance	pipeline_summary.json, commit 897e2f2d, 2026-05-17

This is an earlier, simulation-based generation run. It is not the training data for this checkpoint, and the two corpora should not be described as one dataset.

Availability

The corpus files (train.jsonl 413 MB, val.jsonl 52 MB, test.jsonl 52 MB, students.jsonl 5.6 MB) are retrievable from this repository's git history at commit 69f74e7a; Hugging Face retains LFS objects from deleted commits. Access requires accepting the Llama 3.1 licence gate — unauthenticated requests return 401. The data is synthetic throughout and contains no real student records.

Note that this availability is incidental rather than curated: the files carry no dataset card and are not discoverable from the repository's file listing. Treat them as inspectable for verification, not as a maintained public dataset.

Evaluation

All results are the SFT checkpoint. Strong and weak figures appear in the same table at the same precision, deliberately.

MathTutorBench

Task	Metric	Score
pedagogy_following	match	1.000
pedagogy_following_hard	match	1.000
scaffolding_generation	match	1.000
scaffolding_generation_hard	match	0.997
solution_correctness	recall	0.783
solution_correctness	F1	0.632
solution_correctness	accuracy	0.543
mistake_location	F1 macro	0.086
mistake_correction	accuracy	0.044

The two figures in bold at the bottom are the ones that matter for deployment decisions. This model scores at or near ceiling on tasks measuring whether its output looks like good tutoring, and 0.086 / 0.044 on tasks measuring whether it can locate and correct a student's actual mathematical error. That is roughly a twenty-three-fold gap between form and diagnostic substance. Any single composite "tutoring quality" score computed over these tasks will conceal it.

Behavioural evaluation

Metric	Score	95% CI	n
Socratic question rate	100%	[98%, 100%]	200
Answer avoidance rate	96%	[92%, 98%]	200
Grade-appropriate language	100%	[98%, 100%]	200
Relevance to the specific student error	74.5%	[68%, 80%]	200
Answer leak rate	1%	[0.2%, 5.4%]	100

Relevance to the specific error (74.5%) is the figure to weigh against the ceiling scores above: roughly one response in four does not engage the error the student actually made.

Integration ablation

Injecting per-turn diagnostic labels from an external misconception classifier into this model's prompt produced no measurable improvement in correction match rate: 60% bare versus 55% augmented, on 100 paired math word problems.

Stated limitations of that result: the judge was automated string matching against a reference correction; no confidence interval or significance test was computed, and at n=100 a 5-point difference is within sampling error; the run was not repeated and no seed variation was explored (seed 20260710). This result should not be cited as evidence that label injection degrades performance. It supports only the weaker claim that per-turn injection of a noisy signal did not help at this scale.

Limitations

1. Diagnostic accuracy is poor. 0.086 F1 on mistake location and 0.044 accuracy on mistake correction. The model produces well-formed Socratic tutoring turns while frequently failing to identify what the student got wrong.
2. Form-substance gap. Ceiling scores on pedagogy-following and scaffolding coexist with the figures above. Evaluating this model on conversational quality alone will substantially overstate its instructional value.
3. Training data is LLM-generated, not merely synthetic. The examples were produced by a language model and filtered by automated scoring. The model therefore inherits its generator's conception of what a good Socratic prompt is, along with that generator's phrasing habits and blind spots. A tutor trained this way can reproduce one model's notion of good pedagogy rather than pedagogical ground truth, and no human expert validated the training targets.
4. Coverage is systematic but bounded. 20,000 examples across a 10 × 4 × 5 grid gives 100 variations per cell. Authentic classroom variation in student error expression, language background, and dialogue structure substantially exceeds this, and generalisation beyond the grid is unmeasured.
5. No real student data at any stage, and no claim is made that the generated distribution matches authentic classroom language, error patterns, or student behaviour.
6. Corpus availability is incidental. The training files are retrievable from deleted commits behind the Llama licence gate, without a dataset card and without discoverability. Third parties can inspect the data but should not treat it as a maintained release.
7. No post-DPO evaluation. A DPO stage was trained; its results were never recorded. All metrics here are pre-DPO.
8. No human evaluation. Planned, not performed. All behavioural metrics are automated.
9. No classroom or longitudinal validation. No evidence exists that this model affects learning outcomes.
10. No adversarial safety results for this model. The SafeTutors probe suite (11 harm dimensions, 37 sub-risks, 24 probe scripts) has been executed against Llama 3.3 70B, not against this model. Its robustness to sycophancy, answer extraction, emotional pressure, and frame-shift attacks is unmeasured.
11. English and mathematics only. No evaluation outside English-language K-12 mathematics.
12. Inherits base-model limitations. All known Llama-3.1-8B-Instruct limitations apply, including arithmetic and multi-step reasoning errors.

Out-of-scope uses

• Unsupervised student-facing deployment. Limitations 1, 2, and 9 together make this unsuitable without a human educator in the loop.
• Grading, assessment, or any evaluative judgment about a student. The model cannot reliably locate errors; using it to judge students would produce incorrect and unaccountable determinations.
• Evaluating or ranking teachers.
• Any high-stakes decision about placement, promotion, intervention, or diagnosis.
• Non-mathematics subjects, or non-English instruction.
• A source of mathematical truth. The model is tuned to ask rather than answer, and its solution accuracy is 0.543.

Intended uses

Research on Socratic tutoring behaviour; a baseline for pedagogical evaluation benchmarks; teacher-supervised practice where an educator reviews the interaction; and a starting checkpoint for further fine-tuning — particularly on authentic data, given limitation 4.

Citation

@misc{qlm-math-tutor-2026,
  title  = {qlm-math-tutor: a Socratic mathematics tutoring adapter for Llama-3.1-8B},
  author = {Srivastava, Kumar Sumbhav},
  year   = {2026},
  note   = {Quantum Learning Machines, Inc. LoRA adapter, SFT checkpoint.},
  url    = {https://huggingface.co/QuantumLearningMachines/qlm-math-tutor}
}