File size: 5,906 Bytes
0d81add
 
 
339f0bc
 
 
0d81add
339f0bc
 
6e03092
339f0bc
0d81add
339f0bc
6e03092
0d81add
 
 
 
6e03092
ee431ec
6e03092
8ceb87c
b96307b
 
 
 
 
 
 
6e03092
8ceb87c
6e03092
97f56e3
6e03092
97f56e3
b96307b
97f56e3
84db301
 
6e03092
95d21e4
6e03092
 
713105d
6e03092
 
 
991af9d
 
b96307b
 
 
 
 
 
 
 
 
 
6e03092
713105d
6e03092
713105d
6e03092
 
 
 
 
 
 
 
 
0d81add
6e03092
0d81add
b96307b
 
 
0d81add
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f00ee10
 
 
 
 
0d81add
 
 
 
f00ee10
0d81add
6e03092
 
b96307b
 
6e03092
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
---
license: mit
base_model: microsoft/deberta-v3-base
language:
  - en
pipeline_tag: text-classification
tags:
  - climate
  - un-general-assembly
  - political-science
  - text-classification
metrics:
  - accuracy
  - f1
---

# unga-climate-classifier

A fine-tuned version of [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base) that classifies whether an English sentence from a United Nations General Assembly (UNGA) speech is about climate change.

It was developed to build the **Executive Climate Change Attention (ECCA)** indicator, introduced in Grossman and Jan (2026), *Global Environmental Politics*. The indicator measures the share of climate-related sentences in the annual speeches that governments of 195 countries have delivered at the UNGA since 1985.

## Labels

The model returns one of two labels for each sentence:

- `climate`: the sentence is about climate change
- `other`: it is not

## Definition

A sentence is coded as climate-related when it explicitly refers to anthropogenic climate change, including its causes and consequences, related international and national events, and policies addressing it. Broader references to environmental degradation or energy policy (e.g. renewable energy) are not coded as climate-related unless they mention climate change. Extreme weather events are coded only when attributed to climate change.

## Training data

5,600 manually annotated sentences from the [UN General Debate Corpus](https://doi.org/10.1177/2053168017712821) (Baturo, Dasandi and Mikhaylov 2017), which contains the English versions of the speeches. Intercoder reliability: Krippendorff's α = 0.795.

After removing duplicates, the 5,621 sentences were split into training (3,935), validation (843) and test (843) sets, stratified by label. About 31% of sentences in each set are labelled `climate`.

## How to use

```python
from transformers import pipeline

classifier = pipeline("text-classification", model="mljn/unga-climate-classifier")
classifier("Climate change poses a fundamental threat to our future.")
# [{'label': 'climate', 'score': 0.9988}]
```

## From sentences to the ECCA indicator

The ECCA indicator is the share of sentences in a speech that the model classifies as climate-related. To reproduce it, split each speech into sentences, classify every sentence, and compute the proportion labelled `climate`.

## Limitations

- The model was trained and evaluated on English sentences from UNGA speeches. Its performance on other kinds of text, such as manifestos, parliamentary debates or social media posts, has not been tested.
- It classifies single sentences. Longer texts should be split into sentences first; inputs are truncated at 512 tokens.
- It identifies explicit references to climate change only, following the definition above. Implicit references, such as energy or environmental policy that does not mention climate change, are classified as `other` by design.

## Evaluation

Results on the held-out evaluation set:

| Metric | Value |
|:--|--:|
| Accuracy | 0.975 |
| F1 (micro) | 0.975 |
| F1 (macro) | 0.971 |
| Precision (macro) | 0.971 |
| Recall (macro) | 0.972 |
| Balanced accuracy | 0.972 |
| Loss | 0.081 |

The model outperforms a random forest classifier and several keyword dictionaries (see the paper, Figure 1).

<details>
<summary>Training details</summary>

### Training hyperparameters

- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 80
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.06
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Accuracy Balanced | F1 Micro | Precision Macro | Recall Macro | Precision Micro | Recall Micro |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:-----------------:|:--------:|:---------------:|:------------:|:---------------:|:------------:|
| No log        | 1.0   | 123  | 0.1057          | 0.9726   | 0.9675   | 0.9583            | 0.9726   | 0.9783          | 0.9583       | 0.9726          | 0.9726       |
| No log        | 2.0   | 246  | 0.1102          | 0.9726   | 0.9683   | 0.9697            | 0.9726   | 0.9669          | 0.9697       | 0.9726          | 0.9726       |
| No log        | 3.0   | 369  | 0.0894          | 0.9798   | 0.9763   | 0.9729            | 0.9798   | 0.9800          | 0.9729       | 0.9798          | 0.9798       |
| No log        | 4.0   | 492  | 0.1098          | 0.9762   | 0.9723   | 0.9723            | 0.9762   | 0.9723          | 0.9723       | 0.9762          | 0.9762       |
| 0.1374        | 5.0   | 615  | 0.1026          | 0.9798   | 0.9763   | 0.9729            | 0.9798   | 0.9800          | 0.9729       | 0.9798          | 0.9798       |

### Framework versions

- Transformers 4.36.2
- Pytorch 2.5.0+cu121
- Datasets 2.6.0
- Tokenizers 0.15.2

</details>

## Data and replication

The ECCA indicator and replication materials are available at [github.com/malojan/executive_climate_change_attention](https://github.com/malojan/executive_climate_change_attention).

## Citation

If you use this model, the underlying dataset or the indicator, please cite:

> Grossman, Emiliano, and Malo Jan. 2026. "Executive Climate Change Attention: Toward an Indicator of Comparative Climate Change Attention." *Global Environmental Politics* 26 (2): 137–150. https://doi.org/10.1162/glep.a.1

```bibtex
@article{grossman2026executive,
  title   = {Executive Climate Change Attention: Toward an Indicator of Comparative Climate Change Attention},
  author  = {Grossman, Emiliano and Jan, Malo},
  journal = {Global Environmental Politics},
  volume  = {26},
  number  = {2},
  pages   = {137--150},
  year    = {2026},
  doi     = {10.1162/glep.a.1}
}
```