README.md

metadata

language: en
library_name: setfit
pipeline_tag: text-classification
base_model: sentence-transformers/paraphrase-mpnet-base-v2
datasets:
  - StanfordSCALE/assertions_llm_annotated_talkmoves
metrics:
  - f1
  - precision
  - recall
  - roc_auc
tags:
  - setfit
  - text-classification
  - education
  - classroom-discourse
  - edubehaviors
  - assertion
model-index:
  - name: assertion_proposes_a_problem_solving_strategy
    results:
      - task:
          type: text-classification
        dataset:
          name: assertions_llm_annotated_talkmoves
          type: StanfordSCALE/assertions_llm_annotated_talkmoves
          split: test
        metrics:
          - type: f1
            value: 0.559
            name: F1 (positive class, test)
          - type: precision
            value: 0.595
            name: Precision (positive class, test)
          - type: recall
            value: 0.5272
            name: Recall (positive class, test)
          - type: roc_auc
            value: 0.8768
            name: ROC-AUC (test)

Assertion: proposes a problem solving strategy

This classifier was trained for EduBehaviors: Assertion-based schemas for auditable dialogue coding and is usable through the Python package EduBehaviors-kit. This classifier was trained on an LLM-annotated subset of teacher utterances from the TalkMoves Dataset. See the Datasets section below for more information.

---

Training Details

Datasets

Dataset	Split	Size
StanfordSCALE/assertions_llm_annotated_talkmoves	train	13,180 (53.3%)
StanfordSCALE/assertions_llm_annotated_talkmoves	dev	3,296 (13.3%)
StanfordSCALE/assertions_llm_annotated_talkmoves	test	8,239 (33.3%)

This model's columns are assertion_proposes_a_problem_solving_strategy and split_proposes_a_problem_solving_strategy.

Base rate (share of rows labeled as True): 11.4% overall — 11.6% train, 10.7% dev, 11.4% test.

Labels and annotation

Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.569.

Hyperparameters

Parameter	Value
Base model (body)	sentence-transformers/paraphrase-mpnet-base-v2
Head	LogisticRegression
Body learning rate	2e-05
Head learning rate	0.01
Batch size	16 (contrastive phase) / 32 (head)
Epochs	10
Max steps	5000 (contrastive phase)
Eval max steps	100
Seed	20260904
Mixed precision	enabled on GPU

---

Evaluation

Results

Split	n	Base rate	Precision	Recall	F1 (positive class)	ROC-AUC	Average precision
dev	3,296	10.7%	0.570	0.425	0.487	0.861	0.480
test	8,239	11.4%	0.595	0.527	0.559	0.877	0.564

Limitations

• Labels come from LLM annotators, not human coders. Agreement between annotators with Krippendorff's Alpha is 0.569.
• Trained on teacher utterances only. Behaviour on student speech is untested.

---

How to Use

Message Structure

The model was trained on text built as:

{utterance}

The utterance is passed through as-is.

Running instructions

pip install setfit

from setfit import SetFitModel

model = SetFitModel.from_pretrained("StanfordSCALE/assertion_proposes_a_problem_solving_strategy")

text = 'But one thing that would be  the same is the relative one thing you have to be careful each of you  in your own models would be the sizes in relation to the other sizes  right'
model.predict([text])        # -> array([1]) when the assertion holds
model.predict_proba([text])  # -> [[P(no), P(yes)]]

Citation

@misc{assertion_proposes_a_problem_solving_strategy,
  author = {Stanford SCALE Initiative},
  title  = {Assertion classifier: proposes a problem solving strategy},
  year   = {2026},
  url    = {https://huggingface.co/StanfordSCALE/assertion_proposes_a_problem_solving_strategy}
}