File size: 5,906 Bytes
0d81add 339f0bc 0d81add 339f0bc 6e03092 339f0bc 0d81add 339f0bc 6e03092 0d81add 6e03092 ee431ec 6e03092 8ceb87c b96307b 6e03092 8ceb87c 6e03092 97f56e3 6e03092 97f56e3 b96307b 97f56e3 84db301 6e03092 95d21e4 6e03092 713105d 6e03092 991af9d b96307b 6e03092 713105d 6e03092 713105d 6e03092 0d81add 6e03092 0d81add b96307b 0d81add f00ee10 0d81add f00ee10 0d81add 6e03092 b96307b 6e03092 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | ---
license: mit
base_model: microsoft/deberta-v3-base
language:
- en
pipeline_tag: text-classification
tags:
- climate
- un-general-assembly
- political-science
- text-classification
metrics:
- accuracy
- f1
---
# unga-climate-classifier
A fine-tuned version of [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base) that classifies whether an English sentence from a United Nations General Assembly (UNGA) speech is about climate change.
It was developed to build the **Executive Climate Change Attention (ECCA)** indicator, introduced in Grossman and Jan (2026), *Global Environmental Politics*. The indicator measures the share of climate-related sentences in the annual speeches that governments of 195 countries have delivered at the UNGA since 1985.
## Labels
The model returns one of two labels for each sentence:
- `climate`: the sentence is about climate change
- `other`: it is not
## Definition
A sentence is coded as climate-related when it explicitly refers to anthropogenic climate change, including its causes and consequences, related international and national events, and policies addressing it. Broader references to environmental degradation or energy policy (e.g. renewable energy) are not coded as climate-related unless they mention climate change. Extreme weather events are coded only when attributed to climate change.
## Training data
5,600 manually annotated sentences from the [UN General Debate Corpus](https://doi.org/10.1177/2053168017712821) (Baturo, Dasandi and Mikhaylov 2017), which contains the English versions of the speeches. Intercoder reliability: Krippendorff's α = 0.795.
After removing duplicates, the 5,621 sentences were split into training (3,935), validation (843) and test (843) sets, stratified by label. About 31% of sentences in each set are labelled `climate`.
## How to use
```python
from transformers import pipeline
classifier = pipeline("text-classification", model="mljn/unga-climate-classifier")
classifier("Climate change poses a fundamental threat to our future.")
# [{'label': 'climate', 'score': 0.9988}]
```
## From sentences to the ECCA indicator
The ECCA indicator is the share of sentences in a speech that the model classifies as climate-related. To reproduce it, split each speech into sentences, classify every sentence, and compute the proportion labelled `climate`.
## Limitations
- The model was trained and evaluated on English sentences from UNGA speeches. Its performance on other kinds of text, such as manifestos, parliamentary debates or social media posts, has not been tested.
- It classifies single sentences. Longer texts should be split into sentences first; inputs are truncated at 512 tokens.
- It identifies explicit references to climate change only, following the definition above. Implicit references, such as energy or environmental policy that does not mention climate change, are classified as `other` by design.
## Evaluation
Results on the held-out evaluation set:
| Metric | Value |
|:--|--:|
| Accuracy | 0.975 |
| F1 (micro) | 0.975 |
| F1 (macro) | 0.971 |
| Precision (macro) | 0.971 |
| Recall (macro) | 0.972 |
| Balanced accuracy | 0.972 |
| Loss | 0.081 |
The model outperforms a random forest classifier and several keyword dictionaries (see the paper, Figure 1).
<details>
<summary>Training details</summary>
### Training hyperparameters
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 80
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.06
- num_epochs: 5
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Accuracy Balanced | F1 Micro | Precision Macro | Recall Macro | Precision Micro | Recall Micro |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:-----------------:|:--------:|:---------------:|:------------:|:---------------:|:------------:|
| No log | 1.0 | 123 | 0.1057 | 0.9726 | 0.9675 | 0.9583 | 0.9726 | 0.9783 | 0.9583 | 0.9726 | 0.9726 |
| No log | 2.0 | 246 | 0.1102 | 0.9726 | 0.9683 | 0.9697 | 0.9726 | 0.9669 | 0.9697 | 0.9726 | 0.9726 |
| No log | 3.0 | 369 | 0.0894 | 0.9798 | 0.9763 | 0.9729 | 0.9798 | 0.9800 | 0.9729 | 0.9798 | 0.9798 |
| No log | 4.0 | 492 | 0.1098 | 0.9762 | 0.9723 | 0.9723 | 0.9762 | 0.9723 | 0.9723 | 0.9762 | 0.9762 |
| 0.1374 | 5.0 | 615 | 0.1026 | 0.9798 | 0.9763 | 0.9729 | 0.9798 | 0.9800 | 0.9729 | 0.9798 | 0.9798 |
### Framework versions
- Transformers 4.36.2
- Pytorch 2.5.0+cu121
- Datasets 2.6.0
- Tokenizers 0.15.2
</details>
## Data and replication
The ECCA indicator and replication materials are available at [github.com/malojan/executive_climate_change_attention](https://github.com/malojan/executive_climate_change_attention).
## Citation
If you use this model, the underlying dataset or the indicator, please cite:
> Grossman, Emiliano, and Malo Jan. 2026. "Executive Climate Change Attention: Toward an Indicator of Comparative Climate Change Attention." *Global Environmental Politics* 26 (2): 137–150. https://doi.org/10.1162/glep.a.1
```bibtex
@article{grossman2026executive,
title = {Executive Climate Change Attention: Toward an Indicator of Comparative Climate Change Attention},
author = {Grossman, Emiliano and Jan, Malo},
journal = {Global Environmental Politics},
volume = {26},
number = {2},
pages = {137--150},
year = {2026},
doi = {10.1162/glep.a.1}
}
```
|