File size: 1,980 Bytes
d3f193c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
---
license: mit
library_name: pytorch
pipeline_tag: feature-extraction
tags:
- demography
- population-pyramid
- transformer
- global-health
datasets:
- SallySims/wpp2024-single-age-pyramids
---

# PyramidFormer base
Small transformer encoder (0.8M parameters) over the 101 single-age tokens of a pyramid, pretrained by hiding 30% of ages or one 5-20 year span and reconstructing them. Held-out masked-age MSE (standardised log-share units): 0.0098 after 2000 steps.
The similarity, heaping-detector and forecaster models are each fine-tuned from this backbone and are published as separate repos.

## Training run
quick run: 2000 optimisation steps.

## Training data
UN World Population Prospects 2024, single-age population by sex, countries/areas, 1950-2023 (estimates), as processed in
[SallySims/wpp2024-single-age-pyramids](https://huggingface.co/datasets/SallySims/wpp2024-single-age-pyramids). Split by country: 80% train / 10% validation / 10% test,
so every number above is on countries the model never saw. WPP pyramids are modelled estimates, not raw census counts.
Source: United Nations, DESA, Population Division (2024). World Population Prospects 2024. https://population.un.org/wpp/
Check the UN terms of use before reuse. Research prototype; not for official statistics.
This repo is self-contained: `pyramidformer.py` (model code), `model.safetensors`, `config.json`. There is no hosted inference widget because the architecture is custom; use the snippet below.

## Usage
```python
from huggingface_hub import snapshot_download
import importlib.util, os
p = snapshot_download("SallySims/pyramidformer-base")
spec = importlib.util.spec_from_file_location("pyramidformer", os.path.join(p, "pyramidformer.py"))
P = importlib.util.module_from_spec(spec); spec.loader.exec_module(P)
model, norm = P.load_pretrained(p)
# start a new task model from this backbone
new = P.init_from_base(P.PyramidFormer(task="forecast"), model)   # or "similarity" / "heaping"

```