File size: 4,198 Bytes
e771d70 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | ---
language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
- StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
- f1
- precision
- recall
- roc_auc
tags:
- setfit
- text-classification
- education
- classroom-discourse
- edubehaviors
- assertion
model-index:
- name: assertion_is_an_evaluative_statement
results:
- task:
type: text-classification
dataset:
name: assertions_llm_annotated_talkmoves
type: StanfordSCALE/assertions_llm_annotated_talkmoves
split: test
metrics:
- type: f1
value: 0.6406
name: F1 (positive class, test)
- type: precision
value: 0.7301
name: Precision (positive class, test)
- type: recall
value: 0.5707
name: Recall (positive class, test)
- type: roc_auc
value: 0.9181
name: ROC-AUC (test)
---
# Assertion: the utterance is an evaluative statement
This classifier was trained for *EduBehaviors: Assertion-based schemas for auditable dialogue coding* and is usable through the Python package `EduBehaviors-kit`. This classifier was trained on an LLM-annotated subset of teacher utterances from the [TalkMoves Dataset](https://github.com/SumnerLab/TalkMoves). See the Datasets section below for more information.
---
## Training Details
### Datasets
| Dataset | Split | Size |
|---------|-------|------|
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | train | 13,183 (53.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | dev | 3,295 (13.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | test | 8,237 (33.3%) |
This model's columns are `assertion_is_an_evaluative_statement` and `split_is_an_evaluative_statement`.
**Base rate** (share of rows labeled as True): 5.3% overall —
5.4% train, 5.7% dev, 5.1% test.
### Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.678.
### Hyperparameters
| Parameter | Value |
|-----------|-------|
| Base model (body) | [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) |
| Head | `LogisticRegression` |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
---
## Evaluation
### Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|-------|---|-----------|-----------|--------|---------------------|---------|-------------------|
| dev | 3,295 | 5.7% | 0.814 | 0.631 | 0.711 | 0.938 | 0.779 |
| test | 8,237 | 5.1% | 0.730 | 0.571 | 0.641 | 0.918 | 0.674 |
### Limitations
- **Labels come from LLM annotators, not human coders.** Agreement between annotators with Krippendorff's Alpha is 0.678.
- Trained on teacher utterances only. Behaviour on student speech is untested.
---
## How to Use
### Message Structure
The model was trained on text built as:
```
{utterance}
```
The utterance is passed through as-is.
### Running instructions
```bash
pip install setfit
```
```python
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_is_an_evaluative_statement")
text = 'And if somebody came in now with a fish bigger than the boy that would have missed the point right'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
```
## Citation
```bibtex
@misc{assertion_is_an_evaluative_statement,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: the utterance is an evaluative statement},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_is_an_evaluative_statement}
}
```
|