File size: 4,769 Bytes
59b49d8 5f61af8 59b49d8 5f61af8 59b49d8 5f61af8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 | ---
language: en
license: mit
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert-base-uncased
datasets:
- albertmartinez/OSDG
tags:
- sdg
- sustainable-development-goals
- policy
- text-classification
metrics:
- accuracy
- f1
---
# sdg-classifier
A 16-class text classifier that assigns a passage to one UN Sustainable Development
Goal. Fine-tuned from `distilbert-base-uncased` on the OSDG Community Dataset.
Built primarily as an investigation into the benchmark itself. **Read the
limitations section before using the per-class scores to compare against other
models** — one class in this dataset is separable by writing style rather than
topic, which inflates aggregate metrics.
## Intended use
- First-pass SDG tagging of policy text, dataset descriptions, and report excerpts,
for human review
- A reproducible baseline for OSDG-CD experiments
## Out of scope
- Any use where the output is acted on without review
- SDG 17 (Partnerships), which is absent from the training data
- Detecting that a text is *not* SDG-related — there is no negative class, so the
model will assign a goal to any input, including unrelated text
- Short inputs. Training excerpts average ~100 words; performance on titles or
single phrases is untested
## Results
Test set: 8,605 held-out examples. Training set filtered to agreement ≥ 0.5.
| Configuration | Accuracy | Macro F1 |
|---|---|---|
| 3 epochs | 0.793 | 0.762 |
| **6 epochs (this model)** | **0.799** | **0.772** |
Per-class (6 epochs):
| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| SDG 1 — No Poverty | 0.729 | 0.619 | 0.670 | 494 |
| SDG 2 — Zero Hunger | 0.777 | 0.800 | 0.789 | 506 |
| SDG 3 — Good Health | 0.904 | 0.861 | 0.882 | 526 |
| SDG 4 — Quality Education | 0.887 | 0.829 | 0.857 | 736 |
| SDG 5 — Gender Equality | 0.814 | 0.848 | 0.831 | 875 |
| SDG 6 — Clean Water | 0.764 | 0.807 | 0.785 | 549 |
| SDG 7 — Affordable Energy | 0.785 | 0.838 | 0.810 | 610 |
| SDG 8 — Decent Work | 0.399 | 0.532 | 0.456 | 284 |
| SDG 9 — Industry & Infrastructure | 0.782 | 0.753 | 0.767 | 543 |
| SDG 10 — Reduced Inequalities | 0.613 | 0.648 | 0.630 | 457 |
| SDG 11 — Sustainable Cities | 0.743 | 0.793 | 0.767 | 497 |
| SDG 12 — Responsible Consumption | 0.755 | 0.687 | 0.719 | 233 |
| SDG 13 — Climate Action | 0.779 | 0.726 | 0.751 | 452 |
| SDG 14 — Life Below Water | 0.871 | 0.792 | 0.830 | 231 |
| SDG 15 — Life on Land | 0.851 | 0.794 | 0.822 | 554 |
| SDG 16 — Peace & Justice | 0.984 | 0.985 | 0.984 | 1058 |
## Limitations
**SDG 16's score is partly an artefact.** At 0.984 F1 it is far above every other
class, on arguably the framework's most abstract goal. A TF-IDF + logistic
regression model trained on only the **first eight words** of each excerpt reaches
0.739 F1 on SDG 16 against a macro average of 0.327 — eight words carry no topical
content, so the class is identifiable from stylistic register alone. Sampled SDG 16
excerpts are overwhelmingly academic law and political-science abstracts
("This article provides a theoretically informed analysis…"). The signal is
distributed through the text, not confined to the opening: removing the lead
sentence cost SDG 16 only 0.016 F1 while costing other classes 0.05–0.10.
SDG 16 is 12.3% of the test set, so this inflates the aggregate numbers above.
Per-class scores on this benchmark are not comparable across goals.
**SDG 8 is unreliable.** Precision 0.399 — when this model predicts SDG 8, it is
wrong roughly 6 times in 10. SDG 1, 8, and 10 form a confusion cluster
(1→10 at 0.16, 8→10 at 0.12, 10→8 at 0.11) reflecting genuine conceptual overlap
between poverty, decent work, and inequality. Treat SDG 8 predictions as low
confidence.
**Single-label on a multi-label problem.** Real passages often address several
goals at once. This model returns one.
**No confidence intervals.** Single seed, single run.
## Training
| | |
|---|---|
| Base model | `distilbert-base-uncased` |
| Dataset | `albertmartinez/OSDG`, config `2024-04-01` |
| Filter | agreement ≥ 0.5 |
| Splits | 30,978 train / 3,442 val / 8,605 test |
| Max length | 256 tokens |
| Epochs | 6 |
| Batch size | 32 |
| Learning rate | 3e-5, 10% warmup |
| Hardware | 1 Kaggle session, T4 |
| Model selection | best val macro-F1 |
## Usage
```python
from transformers import pipeline
clf = pipeline("text-classification", model="USERNAME/sdg-classifier")
clf("Rural households gained access to piped drinking water and sanitation.")
# [{'label': 'SDG 6', 'score': 0.59}]
```
## Citation
Dataset: OSDG, UNDP IICPSD SDG AI Lab, & PPMI. *OSDG Community Dataset (OSDG-CD)*.
Zenodo. https://doi.org/10.5281/zenodo.5550238 (CC BY 4.0) |