QuantumLearningMachines commited on
Commit
1801b0b
·
verified ·
1 Parent(s): 0a7bc0c

docs: limitations-first model card, CC BY-NC-SA 4.0, 6-class labels, eval metrics

Browse files
Files changed (1) hide show
  1. README.md +166 -35
README.md CHANGED
@@ -1,56 +1,187 @@
1
  ---
2
- license: apache-2.0
3
- language: en
4
  base_model: microsoft/deberta-v3-base
 
5
  pipeline_tag: text-classification
6
  tags:
7
- - education
8
- - tutoring
9
- - talk-moves
10
- - pedagogy
11
- - discourse-analysis
12
- - k-12
 
 
13
  ---
14
 
15
  # qlm-nto-classifier
16
 
17
- **A DeBERTa-v3 classifier of pedagogical talk moves in tutoring and classroom discourse, built and deployed at Quantum Learning Machines before June 2026.** Where its sibling model ([qlm-map-classifier](https://huggingface.co/QuantumLearningMachines/qlm-map-classifier)) reads the *student's* side of an exchange, this model reads the *teacher's or tutor's* side: it classifies instructional turns into pedagogical move categories — Socratic questioning, revoicing, scaffolding, redirection, worked-example presentation, praise types, direct instruction, and related moves — so that tutoring quality can be evaluated as *behavior*, not vibes.
 
18
 
19
- ## What it does in production
20
 
21
- The model supports two production surfaces. In **Tutor Lab**, QLM's evaluation infrastructure for AI tutoring models, talk-move classification contributes to the pedagogical-quality dimension of tutor evaluation — identifying whether a tutor's turns exhibit the moves the tutoring literature associates with learning, rather than merely fluent text. In **TeachProof's discourse-analysis engine**, the same move taxonomy underlies per-turn analysis of practice sessions and uploaded lesson transcripts (questioning patterns, revoicing counts, praise specificity), feeding research-cited coaching aligned to established observation frameworks. In both surfaces, classifier outputs are aggregated and reviewed — no single classification drives a consequential judgment.
22
 
23
- ## Evaluation
24
-
25
- We have not yet published formal benchmark figures for this model, and we won't imply any here: per our reporting policy, no number appears in our materials without a versioned result file behind it. Quantitative evaluation — including inter-judge agreement with human raters and reliability-filtered comparison data — is being produced through the Tutor Lab pipeline (Bradley-Terry scoring, Dawid-Skene rater reliability, transitivity auditing) and will be added to this card when it exists. Until then, treat this as a production-deployed component published for transparency and research use, not a benchmarked reference model.
26
-
27
- ## Known limitations
28
-
29
- Talk-move boundaries are genuinely fuzzy: a single turn often performs multiple moves, and single-label classification flattens that. The model is trained on English mathematics-tutoring discourse; transfer to other subjects, registers (whole-class vs. one-on-one), and languages is untested. Move *presence* is not move *quality* — detecting a Socratic question says nothing about whether it was a good one; quality judgments in our systems come from separate evaluation layers with human raters.
 
30
 
31
- ## Intended use
32
 
33
- - Research on pedagogical discourse, tutoring-move analysis, and AI-tutor evaluation
34
- - As a component in aggregate, human-reviewed evaluation pipelines
35
- - Extending or critiquing move-taxonomy approaches to tutoring quality
 
 
 
 
 
36
 
37
- ## Out-of-scope use
38
 
39
- - **Consequential evaluation of individual human teachers** — hiring, retention, compensation, or formal performance ratings. This model is not validated for stakes, and talk-move counts are not teaching quality.
40
- - Standalone automated scoring of tutors or teachers without human raters in the loop
41
- - Non-English or non-instructional discourse
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
43
- ## Training data
44
 
45
- Instructional-discourse data from QLM's production tutoring platforms and expert-authored pedagogical materials (private; de-identified). We do not publish corpus composition or counts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
 
47
  ## Citation
48
 
49
- @software{qlm_nto_classifier_2026,
50
- author = {Quantum Learning Machines},
51
- title = {qlm-nto-classifier: pedagogical talk-move classification},
52
- year = {2026},
53
- url = {https://huggingface.co/QuantumLearningMachines/qlm-nto-classifier},
54
- license = {Apache-2.0}
 
 
55
  }
56
- ## Contact: hello@quantumlearningmachines.com · quantumlearningmachines.com
 
1
  ---
2
+ license: cc-by-nc-sa-4.0
 
3
  base_model: microsoft/deberta-v3-base
4
+ library_name: transformers
5
  pipeline_tag: text-classification
6
  tags:
7
+ - education
8
+ - tutoring
9
+ - dialogue
10
+ - talk-moves
11
+ - accountable-talk
12
+ - evaluation
13
+ language:
14
+ - en
15
  ---
16
 
17
  # qlm-nto-classifier
18
 
19
+ A DeBERTa-v3 classifier that labels an utterance from K-12 mathematics classroom
20
+ dialogue with the pedagogical talk move it performs.
21
 
22
+ **Read the Limitations section before using this model.**
23
 
24
+ ## Model details
25
 
26
+ | Field | Value |
27
+ |---|---|
28
+ | Base model | `microsoft/deberta-v3-base` (184M parameters) |
29
+ | Task | Multi-class classification of classroom-dialogue utterances by talk move |
30
+ | Unit of prediction | One utterance |
31
+ | Label set | 6 teacher talk-move classes (see table below) |
32
+ | Scope | Teacher moves only — student discursive moves are out of scope |
33
+ | Input construction | Dialogue triples assembled from consecutive utterances |
34
 
35
+ ### Label mapping (`config.id2label`)
36
 
37
+ | ID | Label |
38
+ |---|---|
39
+ | 0 | Pressing for Accuracy |
40
+ | 1 | Pressing for Reasoning |
41
+ | 2 | Revoicing |
42
+ | 3 | Keeping Everyone Together |
43
+ | 4 | Restating |
44
+ | 5 | Getting Students to Relate to Another's Ideas |
45
 
46
+ ## Training data
47
 
48
+ Trained on the **TalkMoves dataset** — authentic K-12 mathematics classroom
49
+ transcripts, human-transcribed and human-annotated.
50
+
51
+ | Field | Value |
52
+ |---|---|
53
+ | Corpus | TalkMoves (`TalkMoves_preprocessed.csv`) |
54
+ | Utterances used | 31,263 |
55
+ | Source corpus | 567 human-annotated K-12 mathematics lesson transcripts derived from video recordings |
56
+ | Annotation basis | Sentence-level annotation for ten discursive moves grounded in accountable talk theory; utterance-level dialogue-act labels adapted from the Switchboard Dialog Act Corpus |
57
+ | Fields | `session_id`, `speaker` (T/S), `content`, `Human` (human annotation), `AI` (AI annotation), `Utterance ID` |
58
+ | Licence | CC BY-NC-SA 4.0 — inherited from the TalkMoves dataset |
59
+ | Preprocessing | CSV to dialogue triples; pipeline at `codebook-injected-segmentation/src/seg_Dial_start/` |
60
+
61
+ Unlike some other models in this collection, this corpus is **authentic classroom
62
+ dialogue with human annotations** rather than synthetic or model-generated data.
63
+ That is a genuine strength and the reason the limitations below concern
64
+ documentation rather than data provenance.
65
+
66
+ Each utterance carries both a human and an AI annotation, which means
67
+ human-versus-model agreement is computable directly from the corpus. It has not
68
+ been computed or reported.
69
+
70
+ **Split sizes and methodology are undocumented.** Whether the split is by
71
+ session, by speaker, or random matters: an utterance-level random split would leak
72
+ session context between train and test and inflate any performance figure
73
+ computed later. Establish a session-held-out split *before* generating metrics.
74
+
75
+ ### Attribution
76
+
77
+ The TalkMoves dataset is the work of Suresh, Jacobs, Harty, Perkoff, Martin, and
78
+ Sumner (University of Colorado Boulder), published at LREC 2022. Cite it in any
79
+ work using this model:
80
+
81
+ ```
82
+ @inproceedings{suresh-etal-2022-talkmoves,
83
+ title = {The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts
84
+ Annotated for Teacher and Student Discursive Moves},
85
+ author = {Suresh, Abhijit and Jacobs, Jennifer and Harty, Charis and
86
+ Perkoff, Margaret and Martin, James H. and Sumner, Tamara},
87
+ booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation
88
+ Conference},
89
+ pages = {4654--4662},
90
+ year = {2022},
91
+ address = {Marseille, France},
92
+ publisher = {European Language Resources Association}
93
+ }
94
+ ```
95
 
96
+ ## Evaluation
97
 
98
+ | Metric | Value |
99
+ |---|---|
100
+ | Accuracy | 0.31 (155 / 500 correct) |
101
+ | Evaluation set size | n = 500 |
102
+ | Number of classes | 6 |
103
+ | Uniform chance baseline | 16.7% (1/6) |
104
+ | Performance vs. chance | ~1.9x uniform chance baseline |
105
+
106
+ This accuracy was measured across all 6 teacher talk-move classes on n=500
107
+ utterances. The model performs roughly 1.9 times better than uniform random
108
+ guessing (16.7%), but 31% accuracy remains low in absolute terms.
109
+
110
+ **Important caveats about these numbers:**
111
+
112
+ - The majority-class baseline is unknown and likely higher than 16.7%, given
113
+ typical class imbalance in talk-move distributions. The model's advantage over
114
+ the *actual* baseline may be smaller than the uniform-chance comparison suggests.
115
+ - Per-class precision, recall, and F1 are not reported.
116
+ - The evaluation split methodology is undocumented — the 500-example set may not
117
+ be session-held-out, which would make this figure optimistic.
118
+ - Human-versus-model agreement on the TalkMoves corpus has not been computed.
119
+
120
+ Source: `classifier_results.json`, field `nto`.
121
+
122
+ ## Limitations
123
+
124
+ 1. **Accuracy is low (31%).** While above uniform chance, this is insufficient
125
+ for high-stakes individual-level decisions.
126
+ 2. **Training corpus is CC BY-NC-SA 4.0** — the TalkMoves dataset's
127
+ NonCommercial and ShareAlike terms apply to this model. This model is
128
+ free and not commercially deployed.
129
+ 3. **No inter-annotator or human-model agreement is reported**, despite the corpus
130
+ containing paired human and AI annotations that permit it.
131
+ 4. **Split composition and methodology are undocumented.** A random
132
+ utterance-level split would leak session context and make the accuracy figure
133
+ optimistic.
134
+ 5. **Class distribution is undocumented.** Talk-move distributions in classroom
135
+ dialogue are typically severely imbalanced, so the majority-class baseline is
136
+ likely high and is currently unknown.
137
+ 6. **No disaggregation** across grade bands, teacher experience, student
138
+ population, or classroom composition.
139
+ 7. **Transcript-domain dependence.** The source corpus is human-transcribed with
140
+ particular conventions; behaviour on ASR output or differently transcribed
141
+ dialogue is unmeasured.
142
+ 8. **Mathematics classrooms, English only.** No evaluation in other subjects or
143
+ languages.
144
+ 9. **Teacher moves only.** The model classifies only teacher discursive moves and
145
+ cannot characterise student discourse.
146
+
147
+ ## Licence
148
+
149
+ This model is released under **CC BY-NC-SA 4.0** (Creative Commons
150
+ Attribution-NonCommercial-ShareAlike 4.0 International), matching the licence of
151
+ its training data (the TalkMoves dataset). This model is provided free of charge
152
+ and is not commercially deployed.
153
+
154
+ If you use or adapt this model, you must:
155
+ - **Attribute** the TalkMoves dataset (Suresh et al., LREC 2022) and this model
156
+ - **Not use commercially** without separate permission from the dataset authors
157
+ - **Share alike** — distribute adaptations under the same or compatible licence
158
+
159
+ ## Out-of-scope uses
160
+
161
+ - **Any evaluation, rating, or ranking of a teacher or tutor.** Talk-move
162
+ classification invites exactly this use, and with 31% accuracy it would
163
+ be indefensible. This is the primary prohibited use, and it is prohibited
164
+ regardless of future performance figures.
165
+ - **Coaching feedback delivered to an educator** without human review.
166
+ - **Any determination about an individual student.**
167
+ - **Unsupervised deployment of any kind**, given the low accuracy.
168
+ - **Non-English dialogue**, or contexts outside K-12 mathematics classrooms.
169
+
170
+ ## Intended uses
171
+
172
+ Research on automated talk-move classification; a baseline for
173
+ classroom-discourse benchmarks; and aggregate, corpus-level analysis where no
174
+ individual prediction is acted upon.
175
 
176
  ## Citation
177
 
178
+ ```
179
+ @misc{qlm-nto-classifier-2026,
180
+ title = {qlm-nto-classifier: a DeBERTa-v3 pedagogical talk-move classifier},
181
+ author = {Srivastava, Kumar Sumbhav},
182
+ year = {2026},
183
+ note = {Quantum Learning Machines, Inc. Trained on the TalkMoves corpus
184
+ (Suresh et al., LREC 2022).},
185
+ url = {https://huggingface.co/QuantumLearningMachines/qlm-nto-classifier}
186
  }
187
+ ```