File size: 4,321 Bytes
e3fbb87 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | ---
language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
- StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
- f1
- precision
- recall
- roc_auc
tags:
- setfit
- text-classification
- education
- classroom-discourse
- edubehaviors
- assertion
model-index:
- name: assertion_proposes_a_problem_solving_strategy
results:
- task:
type: text-classification
dataset:
name: assertions_llm_annotated_talkmoves
type: StanfordSCALE/assertions_llm_annotated_talkmoves
split: test
metrics:
- type: f1
value: 0.559
name: F1 (positive class, test)
- type: precision
value: 0.595
name: Precision (positive class, test)
- type: recall
value: 0.5272
name: Recall (positive class, test)
- type: roc_auc
value: 0.8768
name: ROC-AUC (test)
---
# Assertion: proposes a problem solving strategy
This classifier was trained for *EduBehaviors: Assertion-based schemas for auditable dialogue coding* and is usable through the Python package `EduBehaviors-kit`. This classifier was trained on an LLM-annotated subset of teacher utterances from the [TalkMoves Dataset](https://github.com/SumnerLab/TalkMoves). See the Datasets section below for more information.
---
## Training Details
### Datasets
| Dataset | Split | Size |
|---------|-------|------|
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | train | 13,180 (53.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | dev | 3,296 (13.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | test | 8,239 (33.3%) |
This model's columns are `assertion_proposes_a_problem_solving_strategy` and `split_proposes_a_problem_solving_strategy`.
**Base rate** (share of rows labeled as True): 11.4% overall —
11.6% train, 10.7% dev, 11.4% test.
### Labels and annotation
Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.569.
### Hyperparameters
| Parameter | Value |
|-----------|-------|
| Base model (body) | [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) |
| Head | `LogisticRegression` |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |
---
## Evaluation
### Results
| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|-------|---|-----------|-----------|--------|---------------------|---------|-------------------|
| dev | 3,296 | 10.7% | 0.570 | 0.425 | 0.487 | 0.861 | 0.480 |
| test | 8,239 | 11.4% | 0.595 | 0.527 | 0.559 | 0.877 | 0.564 |
### Limitations
- **Labels come from LLM annotators, not human coders.** Agreement between annotators with Krippendorff's Alpha is 0.569.
- Trained on teacher utterances only. Behaviour on student speech is untested.
---
## How to Use
### Message Structure
The model was trained on text built as:
```
{utterance}
```
The utterance is passed through as-is.
### Running instructions
```bash
pip install setfit
```
```python
from setfit import SetFitModel
model = SetFitModel.from_pretrained("StanfordSCALE/assertion_proposes_a_problem_solving_strategy")
text = 'But one thing that would be the same is the relative one thing you have to be careful each of you in your own models would be the sizes in relation to the other sizes right'
model.predict([text]) # -> array([1]) when the assertion holds
model.predict_proba([text]) # -> [[P(no), P(yes)]]
```
## Citation
```bibtex
@misc{assertion_proposes_a_problem_solving_strategy,
author = {Stanford SCALE Initiative},
title = {Assertion classifier: proposes a problem solving strategy},
year = {2026},
url = {https://huggingface.co/StanfordSCALE/assertion_proposes_a_problem_solving_strategy}
}
```
|