Cite the arXiv preprint (2609.27059)

1bd22bf verified about 1 month ago

10.5 kB

---
license: cc-by-4.0
language:
- en
base_model:
- FacebookAI/roberta-base
pipeline_tag: text-classification
library_name: transformers
inference: false
tags:
- social-science
- computational-social-science
- situation-entity
- genericity
- discourse-modes
- isaac
- disco
model-index:
- name: isaac-generalization
  results:
  - task:
      type: text-classification
      name: Situation-entity classification (18-way)
    dataset:
      type: disco
      name: "DiSCo gold corpus: held-out 10%"
      split: test
    metrics:
    - type: accuracy
      name: Accuracy (18-way)
      value: 0.737
    - type: f1
      name: Macro F1 (18-way)
      value: 0.514
  - task:
      type: text-classification
      name: Collapsed generalization features
    dataset:
      type: disco
      name: "DiSCo gold corpus: held-out 10%"
      split: test
    metrics:
    - type: f1
      name: Macro F1 (genericity)
      value: 0.852
    - type: accuracy
      name: Accuracy (genericity)
      value: 0.860
    - type: f1
      name: Macro F1 (eventivity)
      value: 0.879
    - type: accuracy
      name: Accuracy (eventivity)
      value: 0.894
    - type: f1
      name: Macro F1 (boundedness / habituality)
      value: 0.804
    - type: accuracy
      name: Accuracy (boundedness / habituality)
      value: 0.850
---
# ISAAC generalization: situation-entity classifier

> **Same weights as the public DiSCo release.** This repository is the
> ISAAC-facing copy of [`BabakScrapes/disco-se-classifier`](https://huggingface.co/BabakScrapes/disco-se-classifier); the checkpoints are
> identical. That card carries the full model description and the DiSCo corpus
> details. This one documents how the model is used inside the ISAAC pipeline and
> restates the performance figures reported in the ISAAC manuscript. Cite
> whichever matches your use; if you are using ISAAC labels, cite both.

Clause-level classifier that assigns one of **18 situation-entity types** to an
English clause, extending Smith's (2003) discourse-mode framework with
boundedness distinctions. In the [Illinois Social Attitudes Aggregate Corpus
(ISAAC)](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) the 18 types are collapsed into three linguistic features
that together capture **how generalized a statement is**:

* **Genericity**: is the main referent a generic category (*gay people*) or a
  specific individual (*my neighbor*)?
* **Eventivity**: a stable state (*God is benevolent*) or a transient event
  (*I went to Nebraska*)?
* **Boundedness / habituality**: for eventive clauses, temporally bounded
  (*I ate this morning*), unbounded (*God loves us*), or habitually recurring
  (*I went there for years*)?

The most anecdotal content is specific entities in bounded, non-habitual events;
the most generalized is generic categories in stative or habitual clauses. This
distinction matters for research on the persuasiveness of attitude-relevant
communication, and no other corpus of social group discourse carries it.

Run this model **after** the
[clause segmenter](https://huggingface.co/ISAAC-corpus/isaac-generalization-segmentation), which splits text into the clauses this model
scores.

## Labels

The released `config.json` in this repository maps indices to readable names.
The full mapping, and the three-feature decomposition ISAAC uses:

| Index | Situation entity | Genericity | Eventivity | Boundedness |
| --- | --- | --- | --- | --- |
| 0 | `BOUNDED EVENT (SPECIFIC)` | specific | dynamic | episodic |
| 1 | `BOUNDED EVENT (GENERIC)` | generic | dynamic | episodic |
| 2 | `UNBOUNDED EVENT (SPECIFIC)` | specific | dynamic | static |
| 3 | `UNBOUNDED EVENT (GENERIC)` | generic | dynamic | static |
| 4 | `BASIC STATE` | specific | stative | static |
| 5 | `COERCED STATE (SPECIFIC)` | specific | dynamic | static |
| 6 | `COERCED STATE (GENERIC)` | generic | dynamic | static |
| 7 | `PERFECT COERCED STATE (SPECIFIC)` | specific | dynamic | episodic |
| 8 | `PERFECT COERCED STATE (GENERIC)` | generic | dynamic | episodic |
| 9 | `GENERIC SENTENCE (DYNAMIC)` | generic | dynamic | habitual |
| 10 | `GENERIC SENTENCE (STATIC)` | generic | stative | static |
| 11 | `GENERIC SENTENCE (HABITUAL)` | generic | stative | habitual |
| 12 | `GENERALIZING SENTENCE (DYNAMIC)` | specific | dynamic | habitual |
| 13 | `GENERALIZING SENTENCE (STATIVE)` | specific | stative | habitual |
| 14 | `QUESTION` | NA | NA | NA |
| 15 | `IMPERATIVE` | NA | NA | NA |
| 16 | `NONSENSE` | NA | NA | NA |
| 17 | `OTHER` | NA | NA | NA |

`QUESTION`, `IMPERATIVE`, `NONSENSE`, and `OTHER` are non-statements, for which
the three features are undefined. ISAAC counts them separately rather than
folding them into the proportions.

## Usage

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
REPO = "ISAAC-corpus/isaac-generalization"
tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True,
                                          add_prefix_space=True)
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()
clauses = ["Gay people are nice", "my gay neighbor watered my plants"]
enc = tokenizer(clauses, padding="max_length", truncation=True, max_length=128,
                return_tensors="pt")
with torch.no_grad():
    pred_ids = model(**enc).logits.argmax(dim=-1).tolist()
for clause, idx in zip(clauses, pred_ids):
    print(model.config.id2label[idx], "|", clause)
```

For the full text-to-features pipeline (segmentation, batching, long-text
splitting, and per-text aggregation), see `code/label_generalization.py` in the
[project repository](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) or `generalization.py` in the
[Space](https://huggingface.co/spaces/BabakScrapes/isaac-classifiers).

## Training data

Pre-trained on the SitEnt corpus, then fine-tuned on the DiSCo corpus of
opinionated, mixed-register English text (Hemmatian, 2022). See the
[DiSCo card](https://huggingface.co/BabakScrapes/disco-se-classifier) for the corpus description.

## Evaluation

Two `FacebookAI/roberta-base` models run in sequence: a clause segmenter, then a
18-way situation-entity classifier. Both are the same weights as
the public DiSCo release; see the
[clause segmenter](https://huggingface.co/BabakScrapes/disco-clause-segmenter) and
[situation-entity classifier](https://huggingface.co/BabakScrapes/disco-se-classifier) cards for the full model
description and the corpus they were trained on.

**Segmentation.** The segmenter covered most of the target clause span for 95.5% of human-verified clauses.

**Classification**, on the held-out 10% of the disco gold corpus (*k* ≈ 2,357):

| Target | Macro F1 | Accuracy |
| --- | --- | --- |
| Full situation entity (18-way) | .514 | .737 |
| Genericity (2-way) | .852 | .860 |
| Eventivity (2-way) | .879 | .894 |
| Boundedness / habituality (4-way) | .804 | .850 |

The 18-way macro F1 of .514 is held down by heavy
label imbalance across the rarer situation-entity types. The three collapsed
features are what ISAAC actually reports, and they are the numbers to rely on.

## Intended use

Clause-level annotation of English text for genericity, eventivity, and
boundedness/habituality, in aggregate research designs, the role it plays in
ISAAC, where it labels clauses across 527M posts.

## Out-of-scope use

* **Clause-level, not document-level.** Feed it clauses from the segmenter, not
  whole posts. Scoring an unsegmented paragraph will not give a meaningful label.
* **Not a measure of truth, quality, or bias.** A generic statement is not
  thereby a stereotype, and a specific one is not thereby accurate.
* **Not for individual-level decisions.** The 18-way macro F1 of .514
  is far too low for any per-clause consequential use; the collapsed features are
  intended for aggregate proportions over large samples.

## Limitations and bias

* **Heavy label imbalance.** The rarer situation-entity types are poorly
  represented in training, which is why 18-way accuracy (.737) far
  exceeds 18-way macro F1 (.514). Per-class performance on rare types is
  correspondingly weak. Use the collapsed features.
* **Single-topic fine-tuning.** DiSCo is opinionated, mixed-register text on one
  controversial policy topic. Cross-genre transfer is expected given the formal,
  content-agnostic nature of the target features, but it has not been separately
  quantified for Reddit discourse about social groups.
* **English only.**
* **Errors compound.** ISAAC's generalization columns are the product of two
  models in series; segmentation errors propagate into classification.

## Links

| | |
| --- | --- |
| Try it without code | [ISAAC Text Classifiers Space](https://huggingface.co/spaces/BabakScrapes/isaac-classifiers) |
| Pipeline source, keyword lists, pattern sets | [GitHub](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) |
| Corpus download, samples, SQL playground | [https://isaac.psychology.illinois.edu/](https://isaac.psychology.illinois.edu/) |
| Data Use Agreement | [Data_Use_Agreement.md](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md) |
| Questions about the models or the corpus | [isaac.corpus.support@gmail.com](mailto:isaac.corpus.support@gmail.com) |

## Citation

Please cite the ISAAC paper. **One citation covers the whole project**: the
corpus, the pipeline, and every model. Please do not cite this model repository
separately; keeping references in one place is what allows the project's
citations to be found together.

```bibtex
@article{hemmatian2026isaac,
  author  = {Hemmatian, Babak and Hadjarab, Sarah and Chen, Jessica and Kurdi, Benedek},
  title   = {The {Illinois} Social Attitudes Aggregate Corpus ({ISAAC}): An Open Tool and Reproducible Pipeline for Analyzing Social Group Discourse at Scale},
  year    = {2026},
  journal = {arXiv},
  eprint  = {2609.27059},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi     = {10.48550/arXiv.2609.27059},
  url     = {https://arxiv.org/abs/2609.27059}
}
```

## License

Released under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).
You may use, share, and adapt these weights, including commercially, provided
you give appropriate credit; see Citation above.

The ISAAC corpus itself is governed separately by the project
[Data Use Agreement](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md).

	---
	license: cc-by-4.0
	language:
	- en
	base_model:
	- FacebookAI/roberta-base
	pipeline_tag: text-classification
	library_name: transformers
	inference: false
	tags:
	- social-science
	- computational-social-science
	- situation-entity
	- genericity
	- discourse-modes
	- isaac
	- disco
	model-index:
	- name: isaac-generalization
	results:
	- task:
	type: text-classification
	name: Situation-entity classification (18-way)
	dataset:
	type: disco
	name: "DiSCo gold corpus: held-out 10%"
	split: test
	metrics:
	- type: accuracy
	name: Accuracy (18-way)
	value: 0.737
	- type: f1
	name: Macro F1 (18-way)
	value: 0.514
	- task:
	type: text-classification
	name: Collapsed generalization features
	dataset:
	type: disco
	name: "DiSCo gold corpus: held-out 10%"
	split: test
	metrics:
	- type: f1
	name: Macro F1 (genericity)
	value: 0.852
	- type: accuracy
	name: Accuracy (genericity)
	value: 0.860
	- type: f1
	name: Macro F1 (eventivity)
	value: 0.879
	- type: accuracy
	name: Accuracy (eventivity)
	value: 0.894
	- type: f1
	name: Macro F1 (boundedness / habituality)
	value: 0.804
	- type: accuracy
	name: Accuracy (boundedness / habituality)
	value: 0.850
	---

	# ISAAC generalization: situation-entity classifier

	> Same weights as the public DiSCo release. This repository is the
	> ISAAC-facing copy of [`BabakScrapes/disco-se-classifier`](https://huggingface.co/BabakScrapes/disco-se-classifier); the checkpoints are
	> identical. That card carries the full model description and the DiSCo corpus
	> details. This one documents how the model is used inside the ISAAC pipeline and
	> restates the performance figures reported in the ISAAC manuscript. Cite
	> whichever matches your use; if you are using ISAAC labels, cite both.

	Clause-level classifier that assigns one of 18 situation-entity types to an
	English clause, extending Smith's (2003) discourse-mode framework with
	boundedness distinctions. In the [Illinois Social Attitudes Aggregate Corpus
	(ISAAC)](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) the 18 types are collapsed into three linguistic features
	that together capture how generalized a statement is:

	* Genericity: is the main referent a generic category (gay people) or a
	specific individual (my neighbor)?
	* Eventivity: a stable state (God is benevolent) or a transient event
	(I went to Nebraska)?
	* Boundedness / habituality: for eventive clauses, temporally bounded
	(I ate this morning), unbounded (God loves us), or habitually recurring
	(I went there for years)?

	The most anecdotal content is specific entities in bounded, non-habitual events;
	the most generalized is generic categories in stative or habitual clauses. This
	distinction matters for research on the persuasiveness of attitude-relevant
	communication, and no other corpus of social group discourse carries it.

	Run this model after the
	[clause segmenter](https://huggingface.co/ISAAC-corpus/isaac-generalization-segmentation), which splits text into the clauses this model
	scores.

	## Labels

	The released `config.json` in this repository maps indices to readable names.
	The full mapping, and the three-feature decomposition ISAAC uses:

	\| Index \| Situation entity \| Genericity \| Eventivity \| Boundedness \|
	\| --- \| --- \| --- \| --- \| --- \|
	\| 0 \| `BOUNDED EVENT (SPECIFIC)` \| specific \| dynamic \| episodic \|
	\| 1 \| `BOUNDED EVENT (GENERIC)` \| generic \| dynamic \| episodic \|
	\| 2 \| `UNBOUNDED EVENT (SPECIFIC)` \| specific \| dynamic \| static \|
	\| 3 \| `UNBOUNDED EVENT (GENERIC)` \| generic \| dynamic \| static \|
	\| 4 \| `BASIC STATE` \| specific \| stative \| static \|
	\| 5 \| `COERCED STATE (SPECIFIC)` \| specific \| dynamic \| static \|
	\| 6 \| `COERCED STATE (GENERIC)` \| generic \| dynamic \| static \|
	\| 7 \| `PERFECT COERCED STATE (SPECIFIC)` \| specific \| dynamic \| episodic \|
	\| 8 \| `PERFECT COERCED STATE (GENERIC)` \| generic \| dynamic \| episodic \|
	\| 9 \| `GENERIC SENTENCE (DYNAMIC)` \| generic \| dynamic \| habitual \|
	\| 10 \| `GENERIC SENTENCE (STATIC)` \| generic \| stative \| static \|
	\| 11 \| `GENERIC SENTENCE (HABITUAL)` \| generic \| stative \| habitual \|
	\| 12 \| `GENERALIZING SENTENCE (DYNAMIC)` \| specific \| dynamic \| habitual \|
	\| 13 \| `GENERALIZING SENTENCE (STATIVE)` \| specific \| stative \| habitual \|
	\| 14 \| `QUESTION` \| NA \| NA \| NA \|
	\| 15 \| `IMPERATIVE` \| NA \| NA \| NA \|
	\| 16 \| `NONSENSE` \| NA \| NA \| NA \|
	\| 17 \| `OTHER` \| NA \| NA \| NA \|

	`QUESTION`, `IMPERATIVE`, `NONSENSE`, and `OTHER` are non-statements, for which
	the three features are undefined. ISAAC counts them separately rather than
	folding them into the proportions.

	## Usage

	```python
	import torch
	from transformers import AutoModelForSequenceClassification, AutoTokenizer

	REPO = "ISAAC-corpus/isaac-generalization"
	tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True,
	add_prefix_space=True)
	model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

	clauses = ["Gay people are nice", "my gay neighbor watered my plants"]
	enc = tokenizer(clauses, padding="max_length", truncation=True, max_length=128,
	return_tensors="pt")
	with torch.no_grad():
	pred_ids = model(**enc).logits.argmax(dim=-1).tolist()

	for clause, idx in zip(clauses, pred_ids):
	print(model.config.id2label[idx], "\|", clause)
	```

	For the full text-to-features pipeline (segmentation, batching, long-text
	splitting, and per-text aggregation), see `code/label_generalization.py` in the
	[project repository](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) or `generalization.py` in the
	[Space](https://huggingface.co/spaces/BabakScrapes/isaac-classifiers).

	## Training data

	Pre-trained on the SitEnt corpus, then fine-tuned on the DiSCo corpus of
	opinionated, mixed-register English text (Hemmatian, 2022). See the
	[DiSCo card](https://huggingface.co/BabakScrapes/disco-se-classifier) for the corpus description.

	## Evaluation

	Two `FacebookAI/roberta-base` models run in sequence: a clause segmenter, then a
	18-way situation-entity classifier. Both are the same weights as
	the public DiSCo release; see the
	[clause segmenter](https://huggingface.co/BabakScrapes/disco-clause-segmenter) and
	[situation-entity classifier](https://huggingface.co/BabakScrapes/disco-se-classifier) cards for the full model
	description and the corpus they were trained on.

	Segmentation. The segmenter covered most of the target clause span for 95.5% of human-verified clauses.

	Classification, on the held-out 10% of the disco gold corpus (k ≈ 2,357):

	\| Target \| Macro F1 \| Accuracy \|
	\| --- \| --- \| --- \|
	\| Full situation entity (18-way) \| .514 \| .737 \|
	\| Genericity (2-way) \| .852 \| .860 \|
	\| Eventivity (2-way) \| .879 \| .894 \|
	\| Boundedness / habituality (4-way) \| .804 \| .850 \|

	The 18-way macro F1 of .514 is held down by heavy
	label imbalance across the rarer situation-entity types. The three collapsed
	features are what ISAAC actually reports, and they are the numbers to rely on.

	## Intended use

	Clause-level annotation of English text for genericity, eventivity, and
	boundedness/habituality, in aggregate research designs, the role it plays in
	ISAAC, where it labels clauses across 527M posts.

	## Out-of-scope use

	* Clause-level, not document-level. Feed it clauses from the segmenter, not
	whole posts. Scoring an unsegmented paragraph will not give a meaningful label.
	* Not a measure of truth, quality, or bias. A generic statement is not
	thereby a stereotype, and a specific one is not thereby accurate.
	* Not for individual-level decisions. The 18-way macro F1 of .514
	is far too low for any per-clause consequential use; the collapsed features are
	intended for aggregate proportions over large samples.

	## Limitations and bias

	* Heavy label imbalance. The rarer situation-entity types are poorly
	represented in training, which is why 18-way accuracy (.737) far
	exceeds 18-way macro F1 (.514). Per-class performance on rare types is
	correspondingly weak. Use the collapsed features.
	* Single-topic fine-tuning. DiSCo is opinionated, mixed-register text on one
	controversial policy topic. Cross-genre transfer is expected given the formal,
	content-agnostic nature of the target features, but it has not been separately
	quantified for Reddit discourse about social groups.
	* English only.
	* Errors compound. ISAAC's generalization columns are the product of two
	models in series; segmentation errors propagate into classification.

	## Links

	\| \| \|
	\| --- \| --- \|
	\| Try it without code \| [ISAAC Text Classifiers Space](https://huggingface.co/spaces/BabakScrapes/isaac-classifiers) \|
	\| Pipeline source, keyword lists, pattern sets \| [GitHub](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) \|
	\| Corpus download, samples, SQL playground \| [https://isaac.psychology.illinois.edu/](https://isaac.psychology.illinois.edu/) \|
	\| Data Use Agreement \| [Data_Use_Agreement.md](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md) \|
	\| Questions about the models or the corpus \| [isaac.corpus.support@gmail.com](mailto:isaac.corpus.support@gmail.com) \|

	## Citation

	Please cite the ISAAC paper. One citation covers the whole project: the
	corpus, the pipeline, and every model. Please do not cite this model repository
	separately; keeping references in one place is what allows the project's
	citations to be found together.

	```bibtex
	@article{hemmatian2026isaac,
	author = {Hemmatian, Babak and Hadjarab, Sarah and Chen, Jessica and Kurdi, Benedek},
	title = {The {Illinois} Social Attitudes Aggregate Corpus ({ISAAC}): An Open Tool and Reproducible Pipeline for Analyzing Social Group Discourse at Scale},
	year = {2026},
	journal = {arXiv},
	eprint = {2609.27059},
	archivePrefix = {arXiv},
	primaryClass = {cs.CL},
	doi = {10.48550/arXiv.2609.27059},
	url = {https://arxiv.org/abs/2609.27059}
	}
	```

	## License

	Released under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).
	You may use, share, and adapt these weights, including commercially, provided
	you give appropriate credit; see Citation above.

	The ISAAC corpus itself is governed separately by the project
	[Data Use Agreement](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md).