README.md

---
language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
  - StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
  - f1
  - precision
  - recall
  - roc_auc
tags:
  - setfit
  - text-classification
  - education
  - classroom-discourse
  - edubehaviors
  - assertion
model-index:
- name: assertion_repeats_a_student_answer
  results:
  - task:
      type: text-classification
    dataset:
      name: assertions_llm_annotated_talkmoves
      type: StanfordSCALE/assertions_llm_annotated_talkmoves
      split: test
    metrics:
    - type: f1
      value: 0.3287
      name: F1 (positive class, test)
    - type: precision
      value: 0.5402
      name: Precision (positive class, test)
    - type: recall
      value: 0.2362
      name: Recall (positive class, test)
    - type: roc_auc
      value: 0.7602
      name: ROC-AUC (test)

---

# Assertion: repeats a student answer

This classifier was trained for *EduBehaviors: Assertion-based schemas for auditable dialogue coding* and is usable through the Python package `EduBehaviors-kit`. This classifier was trained on an LLM-annotated subset of teacher utterances from the [TalkMoves Dataset](https://github.com/SumnerLab/TalkMoves). See the Datasets section below for more information.

---

## Training Details

### Datasets

| Dataset | Split | Size |
|---------|-------|------|
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | train | 13,181 (53.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | dev | 3,294 (13.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | test | 8,240 (33.3%) |

This model's columns are `assertion_repeats_a_student_answer` and `split_repeats_a_student_answer`.

**Base rate** (share of rows labeled as True): 5.1% overall —
5.2% train, 5.1% dev, 4.8% test.

### Labels and annotation

Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.713.


### Hyperparameters

| Parameter | Value |
|-----------|-------|
| Base model (body) | [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) |
| Head | `LogisticRegression` |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |

---

## Evaluation

### Results


| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|-------|---|-----------|-----------|--------|---------------------|---------|-------------------|
| dev | 3,294 | 5.1% | 0.449 | 0.186 | 0.263 | 0.752 | 0.296 |
| test | 8,240 | 4.8% | 0.540 | 0.236 | 0.329 | 0.760 | 0.344 |

### Limitations

- **Labels come from LLM annotators, not human coders.** Agreement between annotators with Krippendorff's Alpha is 0.713.
- Trained on teacher utterances only. Behaviour on student speech is untested.
- Test F1 is 0.329. This model does not work well enough to be used on its own.


---

## How to Use

### Message Structure

The model was trained on text built as:

```
{utterance}
```

The utterance is passed through as-is.

### Running instructions

```bash
pip install setfit
```

```python
from setfit import SetFitModel

model = SetFitModel.from_pretrained("StanfordSCALE/assertion_repeats_a_student_answer")

text = 'The boats the biggest'
model.predict([text])        # -> array([1]) when the assertion holds
model.predict_proba([text])  # -> [[P(no), P(yes)]]
```


## Citation

```bibtex
@misc{assertion_repeats_a_student_answer,
  author = {Stanford SCALE Initiative},
  title  = {Assertion classifier: repeats a student answer},
  year   = {2026},
  url    = {https://huggingface.co/StanfordSCALE/assertion_repeats_a_student_answer}
}
```