README.md

---
language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
  - StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
  - f1
  - precision
  - recall
  - roc_auc
tags:
  - setfit
  - text-classification
  - education
  - classroom-discourse
  - edubehaviors
  - assertion
model-index:
- name: assertion_proposes_a_problem_solving_strategy
  results:
  - task:
      type: text-classification
    dataset:
      name: assertions_llm_annotated_talkmoves
      type: StanfordSCALE/assertions_llm_annotated_talkmoves
      split: test
    metrics:
    - type: f1
      value: 0.559
      name: F1 (positive class, test)
    - type: precision
      value: 0.595
      name: Precision (positive class, test)
    - type: recall
      value: 0.5272
      name: Recall (positive class, test)
    - type: roc_auc
      value: 0.8768
      name: ROC-AUC (test)

---

# Assertion: proposes a problem solving strategy

This classifier was trained for *EduBehaviors: Assertion-based schemas for auditable dialogue coding* and is usable through the Python package `EduBehaviors-kit`. This classifier was trained on an LLM-annotated subset of teacher utterances from the [TalkMoves Dataset](https://github.com/SumnerLab/TalkMoves). See the Datasets section below for more information.

---

## Training Details

### Datasets

| Dataset | Split | Size |
|---------|-------|------|
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | train | 13,180 (53.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | dev | 3,296 (13.3%) |
| [StanfordSCALE/assertions_llm_annotated_talkmoves](https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves) | test | 8,239 (33.3%) |

This model's columns are `assertion_proposes_a_problem_solving_strategy` and `split_proposes_a_problem_solving_strategy`.

**Base rate** (share of rows labeled as True): 11.4% overall —
11.6% train, 10.7% dev, 11.4% test.

### Labels and annotation

Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.569.


### Hyperparameters

| Parameter | Value |
|-----------|-------|
| Base model (body) | [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) |
| Head | `LogisticRegression` |
| Body learning rate | 2e-05 |
| Head learning rate | 0.01 |
| Batch size | 16 (contrastive phase) / 32 (head) |
| Epochs | 10 |
| Max steps | 5000 (contrastive phase) |
| Eval max steps | 100 |
| Seed | 20260904 |
| Mixed precision | enabled on GPU |

---

## Evaluation

### Results


| Split | n | Base rate | Precision | Recall | F1 (positive class) | ROC-AUC | Average precision |
|-------|---|-----------|-----------|--------|---------------------|---------|-------------------|
| dev | 3,296 | 10.7% | 0.570 | 0.425 | 0.487 | 0.861 | 0.480 |
| test | 8,239 | 11.4% | 0.595 | 0.527 | 0.559 | 0.877 | 0.564 |

### Limitations

- **Labels come from LLM annotators, not human coders.** Agreement between annotators with Krippendorff's Alpha is 0.569.
- Trained on teacher utterances only. Behaviour on student speech is untested.


---

## How to Use

### Message Structure

The model was trained on text built as:

```
{utterance}
```

The utterance is passed through as-is.

### Running instructions

```bash
pip install setfit
```

```python
from setfit import SetFitModel

model = SetFitModel.from_pretrained("StanfordSCALE/assertion_proposes_a_problem_solving_strategy")

text = 'But one thing that would be  the same is the relative one thing you have to be careful each of you  in your own models would be the sizes in relation to the other sizes  right'
model.predict([text])        # -> array([1]) when the assertion holds
model.predict_proba([text])  # -> [[P(no), P(yes)]]
```


## Citation

```bibtex
@misc{assertion_proposes_a_problem_solving_strategy,
  author = {Stanford SCALE Initiative},
  title  = {Assertion classifier: proposes a problem solving strategy},
  year   = {2026},
  url    = {https://huggingface.co/StanfordSCALE/assertion_proposes_a_problem_solving_strategy}
}
```