--- license: mit library_name: pytorch pipeline_tag: feature-extraction tags: - demography - population-pyramid - transformer - global-health datasets: - SallySims/wpp2024-single-age-pyramids --- # PyramidFormer base Small transformer encoder (0.8M parameters) over the 101 single-age tokens of a pyramid, pretrained by hiding 30% of ages or one 5-20 year span and reconstructing them. Held-out masked-age MSE (standardised log-share units): 0.0098 after 2000 steps. The similarity, heaping-detector and forecaster models are each fine-tuned from this backbone and are published as separate repos. ## Training run quick run: 2000 optimisation steps. ## Training data UN World Population Prospects 2024, single-age population by sex, countries/areas, 1950-2023 (estimates), as processed in [SallySims/wpp2024-single-age-pyramids](https://huggingface.co/datasets/SallySims/wpp2024-single-age-pyramids). Split by country: 80% train / 10% validation / 10% test, so every number above is on countries the model never saw. WPP pyramids are modelled estimates, not raw census counts. Source: United Nations, DESA, Population Division (2024). World Population Prospects 2024. https://population.un.org/wpp/ Check the UN terms of use before reuse. Research prototype; not for official statistics. This repo is self-contained: `pyramidformer.py` (model code), `model.safetensors`, `config.json`. There is no hosted inference widget because the architecture is custom; use the snippet below. ## Usage ```python from huggingface_hub import snapshot_download import importlib.util, os p = snapshot_download("SallySims/pyramidformer-base") spec = importlib.util.spec_from_file_location("pyramidformer", os.path.join(p, "pyramidformer.py")) P = importlib.util.module_from_spec(spec); spec.loader.exec_module(P) model, norm = P.load_pretrained(p) # start a new task model from this backbone new = P.init_from_base(P.PyramidFormer(task="forecast"), model) # or "similarity" / "heaping" ```