File size: 4,106 Bytes
4f5c6dc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | ---
language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
- StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
- f1
- precision
- recall
- roc_auc
tags:
- setfit
- text-classification
- education
- classroom-discourse
- edubehaviors
- assertion
model-index:
- name: assertion_repeats_a_problem
results:
- task:
type: text-classification
dataset:
name: assertions_llm_annotated_talkmoves
type: StanfordSCALE/assertions_llm_annotated_talkmoves
split: test
metrics:
- type: f1
value: 0.3563
name: F1 (positive class, test)
- type: precision
value: 0.4125
name: Precision (positive class, test)
- type: recall
value: 0.3136
name: Recall (positive class, test)
- type: roc_auc
value: 0.7589
name: ROC-AUC (test)
---
# Assertion: repeats a problem
This classifier was trained for *EduBehaviors: Assertion-based schemas for auditable dialogue coding* and is usable through the Python package `EduBehaviors-kit`. This classifier was trained on an LLM-annotated subset of teacher utterances from the [TalkMoves Dataset](https://github.com/SumnerLab/TalkMoves). See the Datasets section below for more information.
---
## Training Details
### Datasets
| Dataset | Split | Size |
|---------|-------|------|
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | train | 13,178 (53.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | dev | 3,297 (13.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | test | 8,240 (33.3%) |
This model's columns are `assertion_repeats_a_problem` and `split_repeats_a_problem`.
**Base rate** (share of rows labeled as True): 4.2% overall —
4.3% train, 4.0% dev, 4.1% test.
### Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.442.
### Hyperparameters
| Parameter | Value |
|-----------|-------|
| Base model (body) | [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) |
| Head | `LogisticRegression` |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
---
## Evaluation
### Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|-------|---|-----------|-----------|--------|---------------------|---------|-------------------|
| dev | 3,297 | 4.0% | 0.360 | 0.241 | 0.288 | 0.767 | 0.254 |
| test | 8,240 | 4.1% | 0.412 | 0.314 | 0.356 | 0.759 | 0.255 |
### Limitations
- **Labels come from LLM annotators, not human coders.** Agreement between annotators with Krippendorff's Alpha is 0.442.
- Trained on teacher utterances only. Behaviour on student speech is untested.
- Test F1 is 0.356. This model does not work well enough to be used on its own.
---
## How to Use
### Message Structure
The model was trained on text built as:
```
{utterance}
```
The utterance is passed through as-is.
### Running instructions
```bash
pip install setfit
```
```python
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_repeats_a_problem")
text = 'if you use different models'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
```
## Citation
```bibtex
@misc{assertion_repeats_a_problem,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: repeats a problem},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_repeats_a_problem}
}
```
|