README.md

---
license: mit
base_model: microsoft/deberta-v3-base
library_name: transformers
pipeline_tag: text-classification
tags:
  - education
  - mathematics
  - misconception-detection
  - evaluation
language:
  - en
---

# qlm-map-classifier

A DeBERTa-v3 classifier that labels a K-12 mathematics student response with the
misconception it exhibits, drawn from a 39-category grouped taxonomy derived from
QLM's misconception ontology.

**Read the Limitations section before using this model.** Its recall is high and
its precision is near chance: it flags most misconceptions present and is wrong
about roughly half of what it flags. That trade-off determines where it can and
cannot be used, and it is documented in full below.

## Model details

| Field | Value |
|---|---|
| Base model | `microsoft/deberta-v3-base` |
| Task | Multi-class classification over 39 grouped misconception categories, including a negative class |
| Unit of prediction | One student-response utterance |
| Epochs | 5 |
| Batch size | 16 |
| Learning rate | 2e-5 |
| Max sequence length | 256 |
| Warmup ratio | 0.1 |
| In production since | April 2026 |

## Training data

| Field | Value |
|---|---|
| Training examples | ~8,200 |
| Categories | 39 (grouped) |
| Split | 85% train / 10% validation / 5% test |
| Sources | EEDI misconception-labelled student responses (4,370 real student responses); QLM subject verticals; QLM misconception graph |

**On the example count.** The figure is stated as approximately 8,200 rather than
precisely, and that imprecision is deliberate. The number appears in the training
results record as a free-text descriptor rather than as a count derived from a
dataset manifest, and it cannot be reproduced from any current build script. A
later rebuild of the training data on disk contains 10,530 examples under a
different category grouping. Anyone needing an exact count for replication should
treat this as unresolved.

EEDI's 4,370 responses are authentic student work. The remaining examples are
derived from QLM's subject verticals and misconception graph rather than
collected from students.

## Evaluation

Component models are reported alongside the deployed combined model, at the same
precision. The weakest figures are the most decision-relevant.

| Variant | Examples | Categories | Accuracy | Precision | Recall | F1 | Mistake location |
|---|---|---|---|---|---|---|---|
| **Combined-39 (deployed)** | ~8,200 | 39 | **49.5%** | **49.7%** | **86.2%** | **63.1%** | **16.3%** |
| EEDI-only component | 4,370 | 36 | 52.7% | — | 72.1% | 60.4% | — |
| Retrained component | 308 | 23 | 50.2% | — | **0.5%** | — | 12.9% |
| MAP-34 detection | — | 34 | — | — | **25.1%** | — | — |

### How to read these numbers

**The deployed model over-flags.** 86.2% recall against 49.7% precision means it
catches most misconceptions that are present while being incorrect on
approximately half of its positive predictions. It is a high-sensitivity,
low-specificity instrument.

**Accuracy of 49.5% is above chance but its baseline is undocumented.** Uniform
chance on a 39-class problem is 2.6%, so 49.5% is far from random. However, the
class distribution of the training and test sets is not documented (see
Limitations), so a majority-class baseline cannot be computed — and without it,
49.5% cannot be interpreted as skill rather than class imbalance. This is a real
gap, not a rhetorical hedge.

**The retrained component is functionally non-detecting.** 0.5% recall on 308
examples across 23 categories means that component almost never fires. It is
reported here because it contributes to the deployed combined model and its
behaviour is a material limitation.

**MAP-34 detection recall of 25.1% is the figure to weigh** if the intended use
is detection against the MAP taxonomy rather than the grouped-39 taxonomy. The
model misses roughly three quarters of those cases.

### Downstream integration result

Injecting this classifier's per-turn output into a generative tutor's prompt
produced **no measurable improvement** in the tutor's correction match rate: 60%
without labels versus 55% with, on 100 paired math word problems, judged by
automated string matching. No significance test was computed; at n=100 a
5-point difference is within sampling error, and the result should not be cited
as evidence of degradation. The design conclusion drawn internally was that a
noisy per-turn signal is better integrated across a session into a state
estimate than injected turn by turn — an architectural inference, not a result
this experiment establishes.

## Limitations

1. **Precision is near chance.** 49.7% on positive predictions. Roughly half of
   all flags are incorrect.
2. **Mistake-location performance is poor.** 16.3% on the deployed model, 12.9%
   on the retrained component.
3. **MAP-34 detection recall is 25.1%.** Approximately three quarters of MAP-taxonomy
   cases are missed.
4. **One component has 0.5% recall** and effectively never fires.
5. **Class distribution is not documented.** No majority-class baseline can be
   computed, so accuracy cannot be separated from class imbalance.
6. **No inter-annotator agreement is documented.** The label reliability of the
   non-EEDI training examples is unmeasured, so the ceiling on achievable
   performance is unknown.
7. **Exact training-set size is not reproducible.** See the note under Training
   data.
8. **Per-class performance is not published.** Only aggregate figures exist, so
   which of the 39 categories the model handles well is unknown.
9. **No disaggregation by learner profile.** Performance for English learners,
   students with learning differences, or students behind grade level is
   unmeasured. Differential performance across these groups is plausible and
   untested.
10. **Mathematics and English only.** No evaluation outside English-language K-12
    mathematics.
11. **Majority of training data is not authentic student work.** 4,370 of ~8,200
    examples come from EEDI; the remainder are derived from internal verticals
    and the misconception graph.

## Out-of-scope uses

- **Any determination about an individual student.** Grading, placement,
  diagnosis, intervention assignment, or referral. With 49.7% precision, roughly
  half of such determinations would be wrong and none would be contestable.
- **Any evaluation of a teacher.**
- **Unsupervised triage** where a flag triggers action without human review. The
  over-flagging behaviour makes a human reviewer necessary, not optional.
- **Per-turn injection into a generative tutor's prompt** — see the integration
  result above.
- **Detection against the MAP taxonomy**, given 25.1% recall.
- **Non-mathematics subjects**, or non-English responses.

## Intended uses

Research on automated misconception detection; a baseline for misconception
benchmarks; aggregate, model-level analysis where the false-positive rate is
tolerable; and a noisy observation feeding a session-level state estimator that
integrates many signals rather than acting on any single one — the use for which
it was built.

## Citation

```
@misc{qlm-map-classifier-2026,
  title  = {qlm-map-classifier: a DeBERTa-v3 misconception classifier for K-12 mathematics},
  author = {Srivastava, Kumar Sumbhav},
  year   = {2026},
  note   = {Quantum Learning Machines, Inc.},
  url    = {https://huggingface.co/QuantumLearningMachines/qlm-map-classifier}
}
```