--- license: cc-by-4.0 language: - en base_model: - FacebookAI/roberta-base pipeline_tag: text-classification library_name: transformers inference: false tags: - social-science - computational-social-science - situation-entity - genericity - discourse-modes - isaac - disco model-index: - name: isaac-generalization results: - task: type: text-classification name: Situation-entity classification (18-way) dataset: type: disco name: "DiSCo gold corpus: held-out 10%" split: test metrics: - type: accuracy name: Accuracy (18-way) value: 0.737 - type: f1 name: Macro F1 (18-way) value: 0.514 - task: type: text-classification name: Collapsed generalization features dataset: type: disco name: "DiSCo gold corpus: held-out 10%" split: test metrics: - type: f1 name: Macro F1 (genericity) value: 0.852 - type: accuracy name: Accuracy (genericity) value: 0.860 - type: f1 name: Macro F1 (eventivity) value: 0.879 - type: accuracy name: Accuracy (eventivity) value: 0.894 - type: f1 name: Macro F1 (boundedness / habituality) value: 0.804 - type: accuracy name: Accuracy (boundedness / habituality) value: 0.850 --- # ISAAC generalization: situation-entity classifier > **Same weights as the public DiSCo release.** This repository is the > ISAAC-facing copy of [`BabakScrapes/disco-se-classifier`](https://huggingface.co/BabakScrapes/disco-se-classifier); the checkpoints are > identical. That card carries the full model description and the DiSCo corpus > details. This one documents how the model is used inside the ISAAC pipeline and > restates the performance figures reported in the ISAAC manuscript. Cite > whichever matches your use; if you are using ISAAC labels, cite both. Clause-level classifier that assigns one of **18 situation-entity types** to an English clause, extending Smith's (2003) discourse-mode framework with boundedness distinctions. In the [Illinois Social Attitudes Aggregate Corpus (ISAAC)](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) the 18 types are collapsed into three linguistic features that together capture **how generalized a statement is**: * **Genericity**: is the main referent a generic category (*gay people*) or a specific individual (*my neighbor*)? * **Eventivity**: a stable state (*God is benevolent*) or a transient event (*I went to Nebraska*)? * **Boundedness / habituality**: for eventive clauses, temporally bounded (*I ate this morning*), unbounded (*God loves us*), or habitually recurring (*I went there for years*)? The most anecdotal content is specific entities in bounded, non-habitual events; the most generalized is generic categories in stative or habitual clauses. This distinction matters for research on the persuasiveness of attitude-relevant communication, and no other corpus of social group discourse carries it. Run this model **after** the [clause segmenter](https://huggingface.co/ISAAC-corpus/isaac-generalization-segmentation), which splits text into the clauses this model scores. ## Labels The released `config.json` in this repository maps indices to readable names. The full mapping, and the three-feature decomposition ISAAC uses: | Index | Situation entity | Genericity | Eventivity | Boundedness | | --- | --- | --- | --- | --- | | 0 | `BOUNDED EVENT (SPECIFIC)` | specific | dynamic | episodic | | 1 | `BOUNDED EVENT (GENERIC)` | generic | dynamic | episodic | | 2 | `UNBOUNDED EVENT (SPECIFIC)` | specific | dynamic | static | | 3 | `UNBOUNDED EVENT (GENERIC)` | generic | dynamic | static | | 4 | `BASIC STATE` | specific | stative | static | | 5 | `COERCED STATE (SPECIFIC)` | specific | dynamic | static | | 6 | `COERCED STATE (GENERIC)` | generic | dynamic | static | | 7 | `PERFECT COERCED STATE (SPECIFIC)` | specific | dynamic | episodic | | 8 | `PERFECT COERCED STATE (GENERIC)` | generic | dynamic | episodic | | 9 | `GENERIC SENTENCE (DYNAMIC)` | generic | dynamic | habitual | | 10 | `GENERIC SENTENCE (STATIC)` | generic | stative | static | | 11 | `GENERIC SENTENCE (HABITUAL)` | generic | stative | habitual | | 12 | `GENERALIZING SENTENCE (DYNAMIC)` | specific | dynamic | habitual | | 13 | `GENERALIZING SENTENCE (STATIVE)` | specific | stative | habitual | | 14 | `QUESTION` | NA | NA | NA | | 15 | `IMPERATIVE` | NA | NA | NA | | 16 | `NONSENSE` | NA | NA | NA | | 17 | `OTHER` | NA | NA | NA | `QUESTION`, `IMPERATIVE`, `NONSENSE`, and `OTHER` are non-statements, for which the three features are undefined. ISAAC counts them separately rather than folding them into the proportions. ## Usage ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer REPO = "ISAAC-corpus/isaac-generalization" tokenizer = AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_prefix_space=True) model = AutoModelForSequenceClassification.from_pretrained(REPO).eval() clauses = ["Gay people are nice", "my gay neighbor watered my plants"] enc = tokenizer(clauses, padding="max_length", truncation=True, max_length=128, return_tensors="pt") with torch.no_grad(): pred_ids = model(**enc).logits.argmax(dim=-1).tolist() for clause, idx in zip(clauses, pred_ids): print(model.config.id2label[idx], "|", clause) ``` For the full text-to-features pipeline (segmentation, batching, long-text splitting, and per-text aggregation), see `code/label_generalization.py` in the [project repository](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) or `generalization.py` in the [Space](https://huggingface.co/spaces/BabakScrapes/isaac-classifiers). ## Training data Pre-trained on the SitEnt corpus, then fine-tuned on the DiSCo corpus of opinionated, mixed-register English text (Hemmatian, 2022). See the [DiSCo card](https://huggingface.co/BabakScrapes/disco-se-classifier) for the corpus description. ## Evaluation Two `FacebookAI/roberta-base` models run in sequence: a clause segmenter, then a 18-way situation-entity classifier. Both are the same weights as the public DiSCo release; see the [clause segmenter](https://huggingface.co/BabakScrapes/disco-clause-segmenter) and [situation-entity classifier](https://huggingface.co/BabakScrapes/disco-se-classifier) cards for the full model description and the corpus they were trained on. **Segmentation.** The segmenter covered most of the target clause span for 95.5% of human-verified clauses. **Classification**, on the held-out 10% of the disco gold corpus (*k* ≈ 2,357): | Target | Macro F1 | Accuracy | | --- | --- | --- | | Full situation entity (18-way) | .514 | .737 | | Genericity (2-way) | .852 | .860 | | Eventivity (2-way) | .879 | .894 | | Boundedness / habituality (4-way) | .804 | .850 | The 18-way macro F1 of .514 is held down by heavy label imbalance across the rarer situation-entity types. The three collapsed features are what ISAAC actually reports, and they are the numbers to rely on. ## Intended use Clause-level annotation of English text for genericity, eventivity, and boundedness/habituality, in aggregate research designs, the role it plays in ISAAC, where it labels clauses across 527M posts. ## Out-of-scope use * **Clause-level, not document-level.** Feed it clauses from the segmenter, not whole posts. Scoring an unsegmented paragraph will not give a meaningful label. * **Not a measure of truth, quality, or bias.** A generic statement is not thereby a stereotype, and a specific one is not thereby accurate. * **Not for individual-level decisions.** The 18-way macro F1 of .514 is far too low for any per-clause consequential use; the collapsed features are intended for aggregate proportions over large samples. ## Limitations and bias * **Heavy label imbalance.** The rarer situation-entity types are poorly represented in training, which is why 18-way accuracy (.737) far exceeds 18-way macro F1 (.514). Per-class performance on rare types is correspondingly weak. Use the collapsed features. * **Single-topic fine-tuning.** DiSCo is opinionated, mixed-register text on one controversial policy topic. Cross-genre transfer is expected given the formal, content-agnostic nature of the target features, but it has not been separately quantified for Reddit discourse about social groups. * **English only.** * **Errors compound.** ISAAC's generalization columns are the product of two models in series; segmentation errors propagate into classification. ## Links | | | | --- | --- | | Try it without code | [ISAAC Text Classifiers Space](https://huggingface.co/spaces/BabakScrapes/isaac-classifiers) | | Pipeline source, keyword lists, pattern sets | [GitHub](https://github.com/BabakHemmatian/Illinois_Social_Attitudes) | | Corpus download, samples, SQL playground | [https://isaac.psychology.illinois.edu/](https://isaac.psychology.illinois.edu/) | | Data Use Agreement | [Data_Use_Agreement.md](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md) | | Questions about the models or the corpus | [isaac.corpus.support@gmail.com](mailto:isaac.corpus.support@gmail.com) | ## Citation Please cite the ISAAC paper. **One citation covers the whole project**: the corpus, the pipeline, and every model. Please do not cite this model repository separately; keeping references in one place is what allows the project's citations to be found together. ```bibtex @article{hemmatian2026isaac, author = {Hemmatian, Babak and Hadjarab, Sarah and Chen, Jessica and Kurdi, Benedek}, title = {The {Illinois} Social Attitudes Aggregate Corpus ({ISAAC}): An Open Tool and Reproducible Pipeline for Analyzing Social Group Discourse at Scale}, year = {2026}, journal = {arXiv}, eprint = {2609.27059}, archivePrefix = {arXiv}, primaryClass = {cs.CL}, doi = {10.48550/arXiv.2609.27059}, url = {https://arxiv.org/abs/2609.27059} } ``` ## License Released under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/). You may use, share, and adapt these weights, including commercially, provided you give appropriate credit; see Citation above. The ISAAC corpus itself is governed separately by the project [Data Use Agreement](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md).