Text Classification
Transformers
Safetensors
English
deberta-v2
education
tutoring
dialogue
talk-moves
accountable-talk
evaluation
text-embeddings-inference
Instructions to use QuantumLearningMachines/qlm-nto-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use QuantumLearningMachines/qlm-nto-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="QuantumLearningMachines/qlm-nto-classifier")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("QuantumLearningMachines/qlm-nto-classifier") model = AutoModelForSequenceClassification.from_pretrained("QuantumLearningMachines/qlm-nto-classifier", device_map="auto") - Notebooks
- Google Colab
- Kaggle
docs: limitations-first model card, CC BY-NC-SA 4.0, 6-class labels, eval metrics
Browse files
README.md
CHANGED
|
@@ -1,56 +1,187 @@
|
|
| 1 |
---
|
| 2 |
-
license:
|
| 3 |
-
language: en
|
| 4 |
base_model: microsoft/deberta-v3-base
|
|
|
|
| 5 |
pipeline_tag: text-classification
|
| 6 |
tags:
|
| 7 |
-
- education
|
| 8 |
-
- tutoring
|
| 9 |
-
-
|
| 10 |
-
-
|
| 11 |
-
-
|
| 12 |
-
-
|
|
|
|
|
|
|
| 13 |
---
|
| 14 |
|
| 15 |
# qlm-nto-classifier
|
| 16 |
|
| 17 |
-
|
|
|
|
| 18 |
|
| 19 |
-
|
| 20 |
|
| 21 |
-
|
| 22 |
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
|
|
|
| 30 |
|
| 31 |
-
##
|
| 32 |
|
| 33 |
-
|
| 34 |
-
-
|
| 35 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
|
| 37 |
-
##
|
| 38 |
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
-
##
|
| 44 |
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
## Citation
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
title
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
|
|
|
|
|
|
| 55 |
}
|
| 56 |
-
|
|
|
|
| 1 |
---
|
| 2 |
+
license: cc-by-nc-sa-4.0
|
|
|
|
| 3 |
base_model: microsoft/deberta-v3-base
|
| 4 |
+
library_name: transformers
|
| 5 |
pipeline_tag: text-classification
|
| 6 |
tags:
|
| 7 |
+
- education
|
| 8 |
+
- tutoring
|
| 9 |
+
- dialogue
|
| 10 |
+
- talk-moves
|
| 11 |
+
- accountable-talk
|
| 12 |
+
- evaluation
|
| 13 |
+
language:
|
| 14 |
+
- en
|
| 15 |
---
|
| 16 |
|
| 17 |
# qlm-nto-classifier
|
| 18 |
|
| 19 |
+
A DeBERTa-v3 classifier that labels an utterance from K-12 mathematics classroom
|
| 20 |
+
dialogue with the pedagogical talk move it performs.
|
| 21 |
|
| 22 |
+
**Read the Limitations section before using this model.**
|
| 23 |
|
| 24 |
+
## Model details
|
| 25 |
|
| 26 |
+
| Field | Value |
|
| 27 |
+
|---|---|
|
| 28 |
+
| Base model | `microsoft/deberta-v3-base` (184M parameters) |
|
| 29 |
+
| Task | Multi-class classification of classroom-dialogue utterances by talk move |
|
| 30 |
+
| Unit of prediction | One utterance |
|
| 31 |
+
| Label set | 6 teacher talk-move classes (see table below) |
|
| 32 |
+
| Scope | Teacher moves only — student discursive moves are out of scope |
|
| 33 |
+
| Input construction | Dialogue triples assembled from consecutive utterances |
|
| 34 |
|
| 35 |
+
### Label mapping (`config.id2label`)
|
| 36 |
|
| 37 |
+
| ID | Label |
|
| 38 |
+
|---|---|
|
| 39 |
+
| 0 | Pressing for Accuracy |
|
| 40 |
+
| 1 | Pressing for Reasoning |
|
| 41 |
+
| 2 | Revoicing |
|
| 42 |
+
| 3 | Keeping Everyone Together |
|
| 43 |
+
| 4 | Restating |
|
| 44 |
+
| 5 | Getting Students to Relate to Another's Ideas |
|
| 45 |
|
| 46 |
+
## Training data
|
| 47 |
|
| 48 |
+
Trained on the **TalkMoves dataset** — authentic K-12 mathematics classroom
|
| 49 |
+
transcripts, human-transcribed and human-annotated.
|
| 50 |
+
|
| 51 |
+
| Field | Value |
|
| 52 |
+
|---|---|
|
| 53 |
+
| Corpus | TalkMoves (`TalkMoves_preprocessed.csv`) |
|
| 54 |
+
| Utterances used | 31,263 |
|
| 55 |
+
| Source corpus | 567 human-annotated K-12 mathematics lesson transcripts derived from video recordings |
|
| 56 |
+
| Annotation basis | Sentence-level annotation for ten discursive moves grounded in accountable talk theory; utterance-level dialogue-act labels adapted from the Switchboard Dialog Act Corpus |
|
| 57 |
+
| Fields | `session_id`, `speaker` (T/S), `content`, `Human` (human annotation), `AI` (AI annotation), `Utterance ID` |
|
| 58 |
+
| Licence | CC BY-NC-SA 4.0 — inherited from the TalkMoves dataset |
|
| 59 |
+
| Preprocessing | CSV to dialogue triples; pipeline at `codebook-injected-segmentation/src/seg_Dial_start/` |
|
| 60 |
+
|
| 61 |
+
Unlike some other models in this collection, this corpus is **authentic classroom
|
| 62 |
+
dialogue with human annotations** rather than synthetic or model-generated data.
|
| 63 |
+
That is a genuine strength and the reason the limitations below concern
|
| 64 |
+
documentation rather than data provenance.
|
| 65 |
+
|
| 66 |
+
Each utterance carries both a human and an AI annotation, which means
|
| 67 |
+
human-versus-model agreement is computable directly from the corpus. It has not
|
| 68 |
+
been computed or reported.
|
| 69 |
+
|
| 70 |
+
**Split sizes and methodology are undocumented.** Whether the split is by
|
| 71 |
+
session, by speaker, or random matters: an utterance-level random split would leak
|
| 72 |
+
session context between train and test and inflate any performance figure
|
| 73 |
+
computed later. Establish a session-held-out split *before* generating metrics.
|
| 74 |
+
|
| 75 |
+
### Attribution
|
| 76 |
+
|
| 77 |
+
The TalkMoves dataset is the work of Suresh, Jacobs, Harty, Perkoff, Martin, and
|
| 78 |
+
Sumner (University of Colorado Boulder), published at LREC 2022. Cite it in any
|
| 79 |
+
work using this model:
|
| 80 |
+
|
| 81 |
+
```
|
| 82 |
+
@inproceedings{suresh-etal-2022-talkmoves,
|
| 83 |
+
title = {The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts
|
| 84 |
+
Annotated for Teacher and Student Discursive Moves},
|
| 85 |
+
author = {Suresh, Abhijit and Jacobs, Jennifer and Harty, Charis and
|
| 86 |
+
Perkoff, Margaret and Martin, James H. and Sumner, Tamara},
|
| 87 |
+
booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation
|
| 88 |
+
Conference},
|
| 89 |
+
pages = {4654--4662},
|
| 90 |
+
year = {2022},
|
| 91 |
+
address = {Marseille, France},
|
| 92 |
+
publisher = {European Language Resources Association}
|
| 93 |
+
}
|
| 94 |
+
```
|
| 95 |
|
| 96 |
+
## Evaluation
|
| 97 |
|
| 98 |
+
| Metric | Value |
|
| 99 |
+
|---|---|
|
| 100 |
+
| Accuracy | 0.31 (155 / 500 correct) |
|
| 101 |
+
| Evaluation set size | n = 500 |
|
| 102 |
+
| Number of classes | 6 |
|
| 103 |
+
| Uniform chance baseline | 16.7% (1/6) |
|
| 104 |
+
| Performance vs. chance | ~1.9x uniform chance baseline |
|
| 105 |
+
|
| 106 |
+
This accuracy was measured across all 6 teacher talk-move classes on n=500
|
| 107 |
+
utterances. The model performs roughly 1.9 times better than uniform random
|
| 108 |
+
guessing (16.7%), but 31% accuracy remains low in absolute terms.
|
| 109 |
+
|
| 110 |
+
**Important caveats about these numbers:**
|
| 111 |
+
|
| 112 |
+
- The majority-class baseline is unknown and likely higher than 16.7%, given
|
| 113 |
+
typical class imbalance in talk-move distributions. The model's advantage over
|
| 114 |
+
the *actual* baseline may be smaller than the uniform-chance comparison suggests.
|
| 115 |
+
- Per-class precision, recall, and F1 are not reported.
|
| 116 |
+
- The evaluation split methodology is undocumented — the 500-example set may not
|
| 117 |
+
be session-held-out, which would make this figure optimistic.
|
| 118 |
+
- Human-versus-model agreement on the TalkMoves corpus has not been computed.
|
| 119 |
+
|
| 120 |
+
Source: `classifier_results.json`, field `nto`.
|
| 121 |
+
|
| 122 |
+
## Limitations
|
| 123 |
+
|
| 124 |
+
1. **Accuracy is low (31%).** While above uniform chance, this is insufficient
|
| 125 |
+
for high-stakes individual-level decisions.
|
| 126 |
+
2. **Training corpus is CC BY-NC-SA 4.0** — the TalkMoves dataset's
|
| 127 |
+
NonCommercial and ShareAlike terms apply to this model. This model is
|
| 128 |
+
free and not commercially deployed.
|
| 129 |
+
3. **No inter-annotator or human-model agreement is reported**, despite the corpus
|
| 130 |
+
containing paired human and AI annotations that permit it.
|
| 131 |
+
4. **Split composition and methodology are undocumented.** A random
|
| 132 |
+
utterance-level split would leak session context and make the accuracy figure
|
| 133 |
+
optimistic.
|
| 134 |
+
5. **Class distribution is undocumented.** Talk-move distributions in classroom
|
| 135 |
+
dialogue are typically severely imbalanced, so the majority-class baseline is
|
| 136 |
+
likely high and is currently unknown.
|
| 137 |
+
6. **No disaggregation** across grade bands, teacher experience, student
|
| 138 |
+
population, or classroom composition.
|
| 139 |
+
7. **Transcript-domain dependence.** The source corpus is human-transcribed with
|
| 140 |
+
particular conventions; behaviour on ASR output or differently transcribed
|
| 141 |
+
dialogue is unmeasured.
|
| 142 |
+
8. **Mathematics classrooms, English only.** No evaluation in other subjects or
|
| 143 |
+
languages.
|
| 144 |
+
9. **Teacher moves only.** The model classifies only teacher discursive moves and
|
| 145 |
+
cannot characterise student discourse.
|
| 146 |
+
|
| 147 |
+
## Licence
|
| 148 |
+
|
| 149 |
+
This model is released under **CC BY-NC-SA 4.0** (Creative Commons
|
| 150 |
+
Attribution-NonCommercial-ShareAlike 4.0 International), matching the licence of
|
| 151 |
+
its training data (the TalkMoves dataset). This model is provided free of charge
|
| 152 |
+
and is not commercially deployed.
|
| 153 |
+
|
| 154 |
+
If you use or adapt this model, you must:
|
| 155 |
+
- **Attribute** the TalkMoves dataset (Suresh et al., LREC 2022) and this model
|
| 156 |
+
- **Not use commercially** without separate permission from the dataset authors
|
| 157 |
+
- **Share alike** — distribute adaptations under the same or compatible licence
|
| 158 |
+
|
| 159 |
+
## Out-of-scope uses
|
| 160 |
+
|
| 161 |
+
- **Any evaluation, rating, or ranking of a teacher or tutor.** Talk-move
|
| 162 |
+
classification invites exactly this use, and with 31% accuracy it would
|
| 163 |
+
be indefensible. This is the primary prohibited use, and it is prohibited
|
| 164 |
+
regardless of future performance figures.
|
| 165 |
+
- **Coaching feedback delivered to an educator** without human review.
|
| 166 |
+
- **Any determination about an individual student.**
|
| 167 |
+
- **Unsupervised deployment of any kind**, given the low accuracy.
|
| 168 |
+
- **Non-English dialogue**, or contexts outside K-12 mathematics classrooms.
|
| 169 |
+
|
| 170 |
+
## Intended uses
|
| 171 |
+
|
| 172 |
+
Research on automated talk-move classification; a baseline for
|
| 173 |
+
classroom-discourse benchmarks; and aggregate, corpus-level analysis where no
|
| 174 |
+
individual prediction is acted upon.
|
| 175 |
|
| 176 |
## Citation
|
| 177 |
|
| 178 |
+
```
|
| 179 |
+
@misc{qlm-nto-classifier-2026,
|
| 180 |
+
title = {qlm-nto-classifier: a DeBERTa-v3 pedagogical talk-move classifier},
|
| 181 |
+
author = {Srivastava, Kumar Sumbhav},
|
| 182 |
+
year = {2026},
|
| 183 |
+
note = {Quantum Learning Machines, Inc. Trained on the TalkMoves corpus
|
| 184 |
+
(Suresh et al., LREC 2022).},
|
| 185 |
+
url = {https://huggingface.co/QuantumLearningMachines/qlm-nto-classifier}
|
| 186 |
}
|
| 187 |
+
```
|