--- license: cc-by-4.0 language: - en base_model: - FacebookAI/xlm-roberta-large pipeline_tag: text-classification library_name: transformers tags: - text-classification - illiberal-policy-frames - climate model-index: - name: poltextlab/illframes-climate-binary-v2 results: - task: type: text-classification name: Binary illiberal climate-frame detection dataset: name: ILLFRAMES Climate, held-out split of the training pool type: poltextlab/illframes-climate split: test metrics: - name: Accuracy type: accuracy value: 0.8526 - name: Macro F1 type: f1 args: macro value: 0.7462 - name: Weighted F1 type: f1 args: weighted value: 0.8508 - task: type: text-classification name: Binary illiberal climate-frame detection dataset: name: ILLFRAMES Climate, SUPERSET_test_data.csv (the climate-v5 test set) type: poltextlab/illframes-climate split: test_superset metrics: - name: Accuracy type: accuracy value: 0.8115 - name: Macro F1 type: f1 args: macro value: 0.7766 extra_gated_prompt: This model is not accepting new access requests at the moment. Access is still available for our accepted users, which requires the gated access setting to stay active. For inquiries or custom project requests, please contact us at miklos[dot]sebok[at]poltextlab[dot]com. extra_gated_fields: Country: country Institution: text Institution Email: text Full Name: text Please specify your academic project/use case you want to use the models for: text --- # illframes-climate-binary-v2 **Author:** Miklos Sebok (poltextLAB) — miklos[dot]sebok[at]poltextlab[dot]com **© 2026 Miklós Sebők.** Released under CC BY 4.0 — reuse requires attribution to the copyright holder. ## Model Description An **xlm-roberta-large** classifier that decides whether an English sentence carries an illiberal climate-policy frame. It supersedes `poltextlab/illframes-climate-binary`, whose published metrics were computed on a 969-row test set that does not exist in the dataset repository and cannot be verified. This release was trained on a documented split with every evaluation file, prediction file and script included in the repository, so that each number below can be re-derived. Labels: **1 = 710–740** (any illiberal climate frame) · **0 = 799** (None of them). The nine-class codebook is documented on the dataset card ([`poltextlab/illframes-climate`](https://huggingface.co/datasets/poltextlab/illframes-climate)). Use this model when the quantity of interest is whether a text carries an illiberal climate frame at all — frame prevalence, diffusion, time series. For the specific frame use `illframes-climate-v5`, noting that three of its eight frame classes are never recovered (see Limitations). ## How to Use ```python from transformers import AutoTokenizer, pipeline tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large") pipe = pipeline(model="poltextlab/illframes-climate-binary-v2", task="text-classification", tokenizer=tokenizer, use_fast=False, truncation=True, max_length=256, token="") pipe("") ``` The model requires gated access; pass a token when loading. ## Training data and split Source: `poltextlab/illframes-climate`. The three non-test files were concatenated (5177 rows), de-duplicated on normalised text (4740 unique texts; where duplicates disagreed on the label — 205 groups on the nine-class label, 188 on the binary label — the row from `TRAINING_DATA_WITHOUT_SUPERSET_training_data.csv` was kept), and then every row that matched `SUPERSET_test_data.csv` exactly (60) or at TF-IDF cosine ≥ 0.80 (3) was removed, leaving a pool of 4677 rows (18.3% positive). That pool was split 80/10/10, stratified on the binary label, seed 42: train 3742 · validation 467 · test 468. `SUPERSET_test_data.csv` (the published test set of `illframes-climate-v5`, 557 rows after removing 3 within-file duplicates) was held out entirely as a second evaluation set. Split hygiene was re-derived by an independent check: train, validation and test are pairwise disjoint on normalised text; no test row has TF-IDF (1–3-gram) cosine ≥ 0.70 to any training row. The split is at sentence level, not speech level: 97 of 468 test rows (20.7%) share a speech identifier with at least one training row, so the figures below measure generalisation to unseen sentences, not to unseen speeches. Training-row provenance by identifier prefix: AUT 1486, EP 624, DK 549, DEU 310, HUN 284, USA 195, CZ 177, ES 85, syn 21, en 10, pl 1. Rows prefixed `syn`, `en` and `pl` have no documented origin in the dataset repository and are likely synthetic; they are included in training and disclosed here. | Setting | Value | |---|---| | Base model | `xlm-roberta-large` | | Learning rate | 8e-06 (warmup 0.1, weight decay 0.01) | | Epochs | 5 max; early stopping patience 2 on macro-F1; best = epoch 5 | | Effective batch | 16 × 2 | | Max length | 256 tokens | | Precision | fp16 | | Seed | 42 | ## Performance ### Held-out test (n = 468, 18.2% positive) accuracy 0.853 · weighted F1 0.851 · macro F1 0.746 · sensitivity 0.565 · specificity 0.916 | Class | Precision | Recall | F1 | Support | |---|---|---|---|---| | 0: None of them | 0.905 | 0.916 | 0.911 | 383 | | 1: Illiberal frame | 0.600 | 0.565 | 0.582 | 85 | Confusion: TN 351 · FP 32 · FN 37 · TP 48 ### `SUPERSET_test_data.csv` — the climate-v5 test set (n = 557, 36.6% positive) accuracy 0.811 · weighted F1 0.800 · macro F1 0.777 · sensitivity 0.569 · specificity 0.952 | Class | Precision | Recall | F1 | Support | |---|---|---|---|---| | 0: None of them | 0.792 | 0.952 | 0.865 | 353 | | 1: Illiberal frame | 0.872 | 0.569 | 0.688 | 204 | Confusion: TN 336 · FP 17 · FN 88 · TP 116 ### Two operating points The figures above use the default argmax decision (P(frame) ≥ 0.50). A threshold chosen on the validation set to maximise macro-F1 gives P(frame) ≥ **0.4**: | Set | Threshold | Accuracy | Macro F1 | Sensitivity | Specificity | |---|---|---|---|---|---| | test | 0.50 | 0.853 | 0.746 | 0.565 | 0.916 | | test | 0.4 | 0.842 | 0.748 | 0.635 | 0.888 | | SUPERSET | 0.50 | 0.811 | 0.777 | 0.569 | 0.952 | | SUPERSET | 0.4 | 0.822 | 0.795 | 0.627 | 0.935 | `pred__test.csv`, `pred__superset.csv` and `pred__val.csv` in this repository carry P(frame) for every row, so any other threshold can be evaluated directly. Validation (n = 467) was used for checkpoint selection and threshold choice and is **not** a held-out score: accuracy 0.837 · weighted F1 0.835 · macro F1 0.722 · sensitivity 0.523 · specificity 0.908. ### Correcting an observed rate to a prevalence The share of sentences this model flags is not the share that carry a frame. With sensitivity 0.565 and specificity 0.916 (held-out test, threshold 0.50): ``` p_true = (p_obs + specificity − 1) / (sensitivity + specificity − 1) ``` A corpus with no illiberal framing would still return about 8% positives, and a corpus that was entirely framed would return about 56%. The denominator (sensitivity + specificity − 1 = 0.481) is what governs the correction's precision: the smaller it is, the more an error in either rate is amplified. Report corrected prevalence with an interval, never the raw positive rate. ## Comparison with the previous models None of these is a clean head-to-head, and the reasons are stated. | Model | Scored on | Accuracy | Macro F1 | Sensitivity | Specificity | Why it is not a fair baseline | |---|---|---|---|---|---|---| | `poltextlab/illframes-climate-binary` | this held-out test (n = 468) | 0.694 | 0.639 | 0.835 | 0.663 | its training set is undocumented; these test rows come from the same corpus and may have been seen | | `poltextlab/illframes-climate-binary` | SUPERSET (n = 557) | 0.801 | 0.796 | 0.877 | 0.756 | same | | `illframes-climate-v5`, nine classes collapsed to binary | SUPERSET (n = 557) | 0.761 | 0.685 | 0.368 | 0.989 | fair for v5 — SUPERSET is its own published test set | The previous binary model and this one fail in opposite directions: it over-flags (high sensitivity, low specificity), this one under-flags. The published card of `poltextlab/illframes-climate-binary` (accuracy 0.74, weighted F1 0.76, n = 969) refers to a test set that no file or combination of files in the dataset repository reproduces. A second training run on the identical split — class-weighted cross-entropy (weight 4.46 on the frame class), learning rate 1e-05, up to 10 epochs — was tried and rejected: held-out test macro F1 0.724, sensitivity 0.412, specificity 0.966. Its evaluation is kept as `eval_round2.json`. ## Limitations - **Sensitivity is the weak side.** At the default threshold the model misses 44% of framed sentences on the held-out test. The training pool is 18% positive, and the first two epochs predicted no frames at all before the model recovered. - **English, sentence-length input.** Training texts are single sentences (median 26–30 words). Applying the model to whole speeches or articles is a unit mismatch — segment first, aggregate after. Use on other languages is zero-shot transfer through XLM-R and needs a per-language gold set before the output is used substantively. - **Label contestation in the source data.** 205 texts appear in more than one file with different labels; the split resolves them by file priority, not by adjudication. - **Frame-level resolution is not available for every frame.** On its own published test set, `illframes-climate-v5` never recovers classes 721, 722 and 740 (F1 0.00). This binary model does not distinguish frames, so that gap does not affect it, but users who need the specific frame should know it exists. ## Files `model.safetensors` · tokenizer · `config.json` · `eval.json` (all metrics) · `thresholds.json` (validation sweep + applied thresholds) · `pred__*.csv` (per-row P(frame)) · `eval_round2.json` (the rejected second run) · `scripts/` (split, training, baseline, verification and card scripts). ## Cooperation Model performance can be improved by extending the training sets. Coded corpora of any domain and language are welcome at poltextlab{at}poltextlab{dot}com.