QuantumLearningMachines
/

qlm-map-classifier

@@ -1,69 +1,167 @@
 ---
-license: apache-2.0
-language: en
 base_model: microsoft/deberta-v3-base
 pipeline_tag: text-classification
 tags:
-- education
-- tutoring
-- misconception-detection
-- k-12
-- math
 ---
 # qlm-map-classifier
-**A DeBERTa-v3 misconception-detection classifier for K-12 mathematics tutoring, in production at Quantum Learning Machines since April 2026.** It classifies student utterances and worked steps against a taxonomy of documented math misconceptions, and serves as an *observation model* feeding a Bayesian student-state layer — not as a standalone judge of student thinking. We publish it, with its honest numbers, because the field's progress depends on knowing what this class of model can and cannot do.
-## What it does in production
-Given a student turn (utterance, answer, or worked step) in a math tutoring context, the model emits misconception-class probabilities over 34 production classes drawn from QLM's 481-construct misconception ontology (423 constructs public under CC-BY-4.0 via our export API). Its outputs are **not** used to label students directly. They enter a persistent Bayesian posterior alongside triage signals that distinguish misconception-driven errors from slips and disengagement; instructional decisions condition on the accumulated state, not on any single classifier call.
-## Evaluation — read the limitations first
-Evaluated on an internal MathTutorBench-derived error-detection probe (unseen, GSM8K-based). All figures below are from versioned result files.
-| Variant | Detection recall | Mistake-location accuracy | Notes |
-|---|---|---|---|
-| **MAP-34 (this model, deployed)** | **25.1%** | **19.0%** | conservative by design; low over-flagging |
-| Combined-39 (research retrain, not deployed) | 86.2% | 16.3% | 8,207 mixed examples (EEDI public + expert-pattern-derived synthetic); **49.5% solution-correctness accuracy — severe over-flagging** |
-| EEDI-only retrain (not deployed) | 72.1% | 15.7% | same over-flagging pattern |
-| Bare LLM baseline (no classifier) | 78.3% | 14.7% (F1 micro) | flags nearly everything |
-**The pattern that matters:** every high-recall variant achieves its recall by over-flagging — calling correct or merely-slipped work "misconception." The training corpora available to this task (misconception-labeled datasets, expert-authored trigger patterns) contain, by construction, **no negative space**: no slips, no disengagement, no correct-but-unconventional work. A classifier trained only on what misconceptions look like cannot learn what they *don't* look like. This is a data-ecology finding about the field, not an implementation artifact — and it is why the deployed configuration is conservative and why the architecture treats the classifier as one observation source among several.
-## The architecture lesson (ablation)
-In controlled ablation, injecting this classifier's outputs directly into a tutor model's per-turn prompt **reduced** tutoring quality (solution-correctness evaluation: 60% → 55%). Classifier signal helps when integrated as evidence into persistent student state; it hurts when bolted onto generation. Build state layers, not prompt decorations.
 ## Training data
-Production K-12 tutoring interaction data (private; de-identified), labeled against the misconception ontology. We do not publish interaction counts or corpus composition. The ontology's public tier — 423 constructs with descriptions, diagnostic questions, and research citations — is available CC-BY-4.0 via the QLM export API.
-## Intended use
-- Research on misconception detection, error-type triage, and tutoring-system evaluation
-- As an observation component inside stateful student-modeling systems
-- Reproducing and extending the negative-space findings above
-## Out-of-scope use
-- Grading, placement, or any consequential decision about an individual student
-- Standalone "misconception detector" products without state-level integration and human oversight
-- Non-mathematics domains; languages other than English (untested)
-## Bias, risks, and limitations
-Detection behavior has not been separately validated across demographic subgroups; measurement-invariance testing on authentic classroom data is planned, funded work. Low recall means many real misconceptions pass undetected; downstream systems must not treat absence of a flag as evidence of understanding. Trained on production interactions from QLM platforms; distribution shift to other tutoring contexts is untested.
 ## Citation
-@software{qlm_map_classifier_2026,
-author = {Quantum Learning Machines},
-title = {qlm-map-classifier: misconception detection as an observation model},
-year = {2026},
-url = {https://huggingface.co/QuantumLearningMachines/qlm-map-classifier},
-license = {Apache-2.0}
 }
-## Contact: hello@quantumlearningmachines.com · quantumlearningmachines.com

 ---
+license: mit
 base_model: microsoft/deberta-v3-base
+library_name: transformers
 pipeline_tag: text-classification
 tags:
+  - education
+  - mathematics
+  - misconception-detection
+  - evaluation
+language:
+  - en
 ---
 # qlm-map-classifier
+A DeBERTa-v3 classifier that labels a K-12 mathematics student response with the
+misconception it exhibits, drawn from a 39-category grouped taxonomy derived from
+QLM's misconception ontology.
+**Read the Limitations section before using this model.** Its recall is high and
+its precision is near chance: it flags most misconceptions present and is wrong
+about roughly half of what it flags. That trade-off determines where it can and
+cannot be used, and it is documented in full below.
+## Model details
+| Field | Value |
+|---|---|
+| Base model | `microsoft/deberta-v3-base` |
+| Task | Multi-class classification over 39 grouped misconception categories, including a negative class |
+| Unit of prediction | One student-response utterance |
+| Epochs | 5 |
+| Batch size | 16 |
+| Learning rate | 2e-5 |
+| Max sequence length | 256 |
+| Warmup ratio | 0.1 |
+| In production since | April 2026 |
 ## Training data
+| Field | Value |
+|---|---|
+| Training examples | ~8,200 |
+| Categories | 39 (grouped) |
+| Split | 85% train / 10% validation / 5% test |
+| Sources | EEDI misconception-labelled student responses (4,370 real student responses); QLM subject verticals; QLM misconception graph |
+**On the example count.** The figure is stated as approximately 8,200 rather than
+precisely, and that imprecision is deliberate. The number appears in the training
+results record as a free-text descriptor rather than as a count derived from a
+dataset manifest, and it cannot be reproduced from any current build script. A
+later rebuild of the training data on disk contains 10,530 examples under a
+different category grouping. Anyone needing an exact count for replication should
+treat this as unresolved.
+EEDI's 4,370 responses are authentic student work. The remaining examples are
+derived from QLM's subject verticals and misconception graph rather than
+collected from students.
+## Evaluation
+Component models are reported alongside the deployed combined model, at the same
+precision. The weakest figures are the most decision-relevant.
+| Variant | Examples | Categories | Accuracy | Precision | Recall | F1 | Mistake location |
+|---|---|---|---|---|---|---|---|
+| **Combined-39 (deployed)** | ~8,200 | 39 | **49.5%** | **49.7%** | **86.2%** | **63.1%** | **16.3%** |
+| EEDI-only component | 4,370 | 36 | 52.7% | — | 72.1% | 60.4% | — |
+| Retrained component | 308 | 23 | 50.2% | — | **0.5%** | — | 12.9% |
+| MAP-34 detection | — | 34 | — | — | **25.1%** | — | — |
+### How to read these numbers
+**The deployed model over-flags.** 86.2% recall against 49.7% precision means it
+catches most misconceptions that are present while being incorrect on
+approximately half of its positive predictions. It is a high-sensitivity,
+low-specificity instrument.
+**Accuracy of 49.5% is above chance but its baseline is undocumented.** Uniform
+chance on a 39-class problem is 2.6%, so 49.5% is far from random. However, the
+class distribution of the training and test sets is not documented (see
+Limitations), so a majority-class baseline cannot be computed — and without it,
+49.5% cannot be interpreted as skill rather than class imbalance. This is a real
+gap, not a rhetorical hedge.
+**The retrained component is functionally non-detecting.** 0.5% recall on 308
+examples across 23 categories means that component almost never fires. It is
+reported here because it contributes to the deployed combined model and its
+behaviour is a material limitation.
+**MAP-34 detection recall of 25.1% is the figure to weigh** if the intended use
+is detection against the MAP taxonomy rather than the grouped-39 taxonomy. The
+model misses roughly three quarters of those cases.
+### Downstream integration result
+Injecting this classifier's per-turn output into a generative tutor's prompt
+produced **no measurable improvement** in the tutor's correction match rate: 60%
+without labels versus 55% with, on 100 paired math word problems, judged by
+automated string matching. No significance test was computed; at n=100 a
+5-point difference is within sampling error, and the result should not be cited
+as evidence of degradation. The design conclusion drawn internally was that a
+noisy per-turn signal is better integrated across a session into a state
+estimate than injected turn by turn — an architectural inference, not a result
+this experiment establishes.
+## Limitations
+1. **Precision is near chance.** 49.7% on positive predictions. Roughly half of
+   all flags are incorrect.
+2. **Mistake-location performance is poor.** 16.3% on the deployed model, 12.9%
+   on the retrained component.
+3. **MAP-34 detection recall is 25.1%.** Approximately three quarters of MAP-taxonomy
+   cases are missed.
+4. **One component has 0.5% recall** and effectively never fires.
+5. **Class distribution is not documented.** No majority-class baseline can be
+   computed, so accuracy cannot be separated from class imbalance.
+6. **No inter-annotator agreement is documented.** The label reliability of the
+   non-EEDI training examples is unmeasured, so the ceiling on achievable
+   performance is unknown.
+7. **Exact training-set size is not reproducible.** See the note under Training
+   data.
+8. **Per-class performance is not published.** Only aggregate figures exist, so
+   which of the 39 categories the model handles well is unknown.
+9. **No disaggregation by learner profile.** Performance for English learners,
+   students with learning differences, or students behind grade level is
+   unmeasured. Differential performance across these groups is plausible and
+   untested.
+10. **Mathematics and English only.** No evaluation outside English-language K-12
+    mathematics.
+11. **Majority of training data is not authentic student work.** 4,370 of ~8,200
+    examples come from EEDI; the remainder are derived from internal verticals
+    and the misconception graph.
+## Out-of-scope uses
+- **Any determination about an individual student.** Grading, placement,
+  diagnosis, intervention assignment, or referral. With 49.7% precision, roughly
+  half of such determinations would be wrong and none would be contestable.
+- **Any evaluation of a teacher.**
+- **Unsupervised triage** where a flag triggers action without human review. The
+  over-flagging behaviour makes a human reviewer necessary, not optional.
+- **Per-turn injection into a generative tutor's prompt** — see the integration
+  result above.
+- **Detection against the MAP taxonomy**, given 25.1% recall.
+- **Non-mathematics subjects**, or non-English responses.
+## Intended uses
+Research on automated misconception detection; a baseline for misconception
+benchmarks; aggregate, model-level analysis where the false-positive rate is
+tolerable; and a noisy observation feeding a session-level state estimator that
+integrates many signals rather than acting on any single one — the use for which
+it was built.
 ## Citation
+```
+@misc{qlm-map-classifier-2026,
+  title  = {qlm-map-classifier: a DeBERTa-v3 misconception classifier for K-12 mathematics},
+  author = {Srivastava, Kumar Sumbhav},
+  year   = {2026},
+  note   = {Quantum Learning Machines, Inc.},
+  url    = {https://huggingface.co/QuantumLearningMachines/qlm-map-classifier}
 }
+```