Alignment without Transfer β€” model checkpoints

Checkpoints for the paper Alignment without Transfer: Tokenization Impacts Cross-lingual Representations during Multilingual Language Model Pretraining (preprint forthcoming; not yet on arXiv).

About 1B-parameter Llama-style decoders (16 layers, d=2048, SwiGLU, RoPE, untied embeddings, 65,536-piece vocabulary) trained from scratch on one or two of English, German, French, Arabic and Chinese (FineWeb-HQ / FineWeb2-HQ). Model pairs differ only in the tokenizer, i.e. in how many languages share the 65k vocabulary: 419, 20, 5 or 2.

164 checkpoints: 96 bilingual (English + one partner language) and 68 monolingual, each at several token budgets.

Everything else for the paper is in the Hugging Face collection and on GitHub:

code, experiment log github.com/jvonrad/XScript-Pretraining
cached alignment embeddings jvonrad/alignment-without-transfer-embeddings
data provenance (pool file lists, holdout text) jvonrad/alignment-without-transfer-data

Layout

model-checkpoints/
  bilingual/<en-X>/<N>-languages/<budget>/      X in de, fr, ar, zh
  monolingual/<lang>/<N>-languages/<budget>/    lang in en, de, fr, ar, zh
      final.pt.part000 ... part004              one torch.save file split in 5 parts
      n_parts.txt                               number of parts
tokenizers/<tokenizer>/{sp.model, meta.json}
models.json                                     model name -> path, tokenizer, languages, budget, LR state
training-states/                                optimizer/resume states of some runs (not needed for evaluation)
  • <N>-languages β€” how many languages the tokenizer was trained on:

    folder tokenizer trained on
    2-languages unigram_bi_<X> English + the partner language only
    5-languages unigram_destarved the 5 study languages (byte-premium adjusted)
    20-languages unigram_20lang the 5 study languages + ces, fas, hun, ind, ita, jpn, kor, nld, pol, por, ron, rus, swe, ukr, vie
    419-languages unigram_starved ~419 FineWeb/FineWeb2 languages (T=100)

    All are SentencePiece Unigram, 65,536 pieces, byte fallback, ids 0–3 = <unk>/<bos>/<eos>/<pad>.

  • <budget> β€” total training tokens (for bilinguals half per language): 2b … 23b are mid-stable snapshots at peak learning rate; 30b-final is the cooled final model.

⚠️ Compare like with like

All runs share one warmup–stable–decay schedule: warmup to 1B tokens, peak LR 3.0e-3 until 24B, decay to 3.0e-4 by 30B. A 30b-final checkpoint has been through the decay; every other budget has not. Pairing the two hands the cooled model the whole decay phase for free. Exceptions: monolingual/de/419-languages/16b (a re-seeded retrain that stops at 16.1B, not cooled) is the largest German 419-language model, and Chinese monolinguals exist only up to 15b.

Model names

models.json keeps the short names the code and result files use: <langs>-<tok>[-<budget>], with <tok> = starved (419 languages), 20lang (20), fair (5) or bi (2), and no budget suffix for the cooled final, e.g. en-ar-fair-12b β†’ model-checkpoints/bilingual/en-ar/5-languages/12b.

Loading a checkpoint

import json, shutil
from huggingface_hub import hf_hub_download

repo, name = "jvonrad/alignment-without-transfer", "en-ar-fair"           # any key of models.json
spec = json.load(open(hf_hub_download(repo, "models.json")))[name]
d = spec["path"]                                 # model-checkpoints/bilingual/en-ar/5-languages/30b-final
n = int(open(hf_hub_download(repo, f"{d}/n_parts.txt")).read())
with open("final.pt", "wb") as out:
    for i in range(n):
        with open(hf_hub_download(repo, f"{d}/final.pt.part{i:03d}"), "rb") as part:
            shutil.copyfileobj(part, out)
sp_model = hf_hub_download(repo, f"tokenizers/{spec['tok']}/sp.model")

# with the GitHub repo's src/ on sys.path:
import torch
from xscript.model import ModelConfig, Transformer
ck = torch.load("final.pt", map_location="cpu", weights_only=False)
model = Transformer(ModelConfig(**ck["cfg"]["model"]))
model.load_state_dict(ck["model"])

The GitHub repo's evaluation scripts do this automatically (scripts/external_bench/hf_layout.py maps model names to these paths).

Checkpoint index

Each cell links to the checkpoint folder.

Bilingual

en-de

en-fr

en-ar

en-zh

Monolingual

en

de

fr

ar

zh

Full table (path, exact token count, LR state, original run name)
model path tokenizer tokens LR state original run
en-ar-bi-10b model-checkpoints/bilingual/en-ar/2-languages/10b 2 languages (unigram_bi_ar) 9.76B mid-stable (peak LR 3e-3) en-ar__unigram_bi_ar__step9929_9760M
en-ar-bi-15b model-checkpoints/bilingual/en-ar/2-languages/15b 2 languages (unigram_bi_ar) 14.764B mid-stable (peak LR 3e-3) en-ar__unigram_bi_ar__step15019_14764M
en-ar-bi-23b model-checkpoints/bilingual/en-ar/2-languages/23b 2 languages (unigram_bi_ar) 22.77B mid-stable (peak LR 3e-3) en-ar__unigram_bi_ar__step23163_22770M
en-ar-bi-2b model-checkpoints/bilingual/en-ar/2-languages/2b 2 languages (unigram_bi_ar) 1.754B mid-stable (peak LR 3e-3) en-ar__unigram_bi_ar__step1785_1754M
en-ar-bi model-checkpoints/bilingual/en-ar/2-languages/30b-final 2 languages (unigram_bi_ar) 30B cooled final (LR 3e-4) en-ar__unigram_bi_ar
en-ar-bi-5b model-checkpoints/bilingual/en-ar/2-languages/5b 2 languages (unigram_bi_ar) 4.756B mid-stable (peak LR 3e-3) en-ar__unigram_bi_ar__step4839_4756M
en-ar-20lang-10b model-checkpoints/bilingual/en-ar/20-languages/10b 20 languages (unigram_20lang) 9.76B mid-stable (peak LR 3e-3) en-ar__unigram_20lang__step9929_9760M
en-ar-20lang-15b model-checkpoints/bilingual/en-ar/20-languages/15b 20 languages (unigram_20lang) 14.764B mid-stable (peak LR 3e-3) en-ar__unigram_20lang__step15019_14764M
en-ar-20lang-23b model-checkpoints/bilingual/en-ar/20-languages/23b 20 languages (unigram_20lang) 22.766B mid-stable (peak LR 3e-3) en-ar__unigram_20lang__step23159_22766M
en-ar-20lang-2b model-checkpoints/bilingual/en-ar/20-languages/2b 20 languages (unigram_20lang) 1.754B mid-stable (peak LR 3e-3) en-ar__unigram_20lang__step1785_1754M
en-ar-20lang model-checkpoints/bilingual/en-ar/20-languages/30b-final 20 languages (unigram_20lang) 30B cooled final (LR 3e-4) en-ar__unigram_20lang
en-ar-20lang-5b model-checkpoints/bilingual/en-ar/20-languages/5b 20 languages (unigram_20lang) 4.756B mid-stable (peak LR 3e-3) en-ar__unigram_20lang__step4839_4756M
en-ar-starved-10b model-checkpoints/bilingual/en-ar/419-languages/10b 419 languages (unigram_starved) 9.753B mid-stable (peak LR 3e-3) en-ar__unigram_starved__step10631_9753M
en-ar-starved-15b model-checkpoints/bilingual/en-ar/419-languages/15b 419 languages (unigram_starved) 14.757B mid-stable (peak LR 3e-3) en-ar__unigram_starved__step15793_14757M
en-ar-starved-23b model-checkpoints/bilingual/en-ar/419-languages/23b 419 languages (unigram_starved) 22.758B mid-stable (peak LR 3e-3) en-ar__unigram_starved__step23933_22758M
en-ar-starved-2b model-checkpoints/bilingual/en-ar/419-languages/2b 419 languages (unigram_starved) 1.753B mid-stable (peak LR 3e-3) en-ar__unigram_starved__step1911_1753M
en-ar-starved model-checkpoints/bilingual/en-ar/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) en-ar__unigram_starved
en-ar-starved-5b model-checkpoints/bilingual/en-ar/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) en-ar__unigram_starved__step5181_4753M
en-ar-fair-10b model-checkpoints/bilingual/en-ar/5-languages/10b 5 languages (unigram_destarved) 9.753B mid-stable (peak LR 3e-3) en-ar__unigram_destarved__step10631_9753M
en-ar-fair-15b model-checkpoints/bilingual/en-ar/5-languages/15b 5 languages (unigram_destarved) 14.755B mid-stable (peak LR 3e-3) en-ar__unigram_destarved__step15937_14755M
en-ar-fair-23b model-checkpoints/bilingual/en-ar/5-languages/23b 5 languages (unigram_destarved) 22.757B mid-stable (peak LR 3e-3) en-ar__unigram_destarved__step24077_22757M
en-ar-fair-2b model-checkpoints/bilingual/en-ar/5-languages/2b 5 languages (unigram_destarved) 1.753B mid-stable (peak LR 3e-3) en-ar__unigram_destarved__step1911_1753M
en-ar-fair model-checkpoints/bilingual/en-ar/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) en-ar__unigram_destarved
en-ar-fair-5b model-checkpoints/bilingual/en-ar/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) en-ar__unigram_destarved__step5181_4753M
en-de-bi-10b model-checkpoints/bilingual/en-de/2-languages/10b 2 languages (unigram_bi_de) 9.76B mid-stable (peak LR 3e-3) en-de__unigram_bi_de__step9929_9760M
en-de-bi-15b model-checkpoints/bilingual/en-de/2-languages/15b 2 languages (unigram_bi_de) 14.764B mid-stable (peak LR 3e-3) en-de__unigram_bi_de__step15019_14764M
en-de-bi-23b model-checkpoints/bilingual/en-de/2-languages/23b 2 languages (unigram_bi_de) 22.77B mid-stable (peak LR 3e-3) en-de__unigram_bi_de__step23163_22770M
en-de-bi-2b model-checkpoints/bilingual/en-de/2-languages/2b 2 languages (unigram_bi_de) 1.754B mid-stable (peak LR 3e-3) en-de__unigram_bi_de__step1785_1754M
en-de-bi model-checkpoints/bilingual/en-de/2-languages/30b-final 2 languages (unigram_bi_de) 30B cooled final (LR 3e-4) en-de__unigram_bi_de
en-de-bi-5b model-checkpoints/bilingual/en-de/2-languages/5b 2 languages (unigram_bi_de) 4.756B mid-stable (peak LR 3e-3) en-de__unigram_bi_de__step4839_4756M
en-de-20lang-10b model-checkpoints/bilingual/en-de/20-languages/10b 20 languages (unigram_20lang) 9.76B mid-stable (peak LR 3e-3) en-de__unigram_20lang__step9929_9760M
en-de-20lang-15b model-checkpoints/bilingual/en-de/20-languages/15b 20 languages (unigram_20lang) 14.764B mid-stable (peak LR 3e-3) en-de__unigram_20lang__step15019_14764M
en-de-20lang-23b model-checkpoints/bilingual/en-de/20-languages/23b 20 languages (unigram_20lang) 22.766B mid-stable (peak LR 3e-3) en-de__unigram_20lang__step23159_22766M
en-de-20lang-2b model-checkpoints/bilingual/en-de/20-languages/2b 20 languages (unigram_20lang) 1.754B mid-stable (peak LR 3e-3) en-de__unigram_20lang__step1785_1754M
en-de-20lang model-checkpoints/bilingual/en-de/20-languages/30b-final 20 languages (unigram_20lang) 30B cooled final (LR 3e-4) en-de__unigram_20lang
en-de-20lang-5b model-checkpoints/bilingual/en-de/20-languages/5b 20 languages (unigram_20lang) 4.756B mid-stable (peak LR 3e-3) en-de__unigram_20lang__step4839_4756M
en-de-starved-10b model-checkpoints/bilingual/en-de/419-languages/10b 419 languages (unigram_starved) 9.753B mid-stable (peak LR 3e-3) en-de__unigram_starved__step10631_9753M
en-de-starved-15b model-checkpoints/bilingual/en-de/419-languages/15b 419 languages (unigram_starved) 14.756B mid-stable (peak LR 3e-3) en-de__unigram_starved__step15865_14756M
en-de-starved-23b model-checkpoints/bilingual/en-de/419-languages/23b 419 languages (unigram_starved) 22.758B mid-stable (peak LR 3e-3) en-de__unigram_starved__step24005_22758M
en-de-starved-2b model-checkpoints/bilingual/en-de/419-languages/2b 419 languages (unigram_starved) 1.753B mid-stable (peak LR 3e-3) en-de__unigram_starved__step1911_1753M
en-de-starved model-checkpoints/bilingual/en-de/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) en-de__unigram_starved
en-de-starved-5b model-checkpoints/bilingual/en-de/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) en-de__unigram_starved__step5181_4753M
en-de-fair-10b model-checkpoints/bilingual/en-de/5-languages/10b 5 languages (unigram_destarved) 9.753B mid-stable (peak LR 3e-3) en-de__unigram_destarved__step10631_9753M
en-de-fair-15b model-checkpoints/bilingual/en-de/5-languages/15b 5 languages (unigram_destarved) 14.755B mid-stable (peak LR 3e-3) en-de__unigram_destarved__step15937_14755M
en-de-fair-23b model-checkpoints/bilingual/en-de/5-languages/23b 5 languages (unigram_destarved) 22.757B mid-stable (peak LR 3e-3) en-de__unigram_destarved__step24077_22757M
en-de-fair-2b model-checkpoints/bilingual/en-de/5-languages/2b 5 languages (unigram_destarved) 1.753B mid-stable (peak LR 3e-3) en-de__unigram_destarved__step1911_1753M
en-de-fair model-checkpoints/bilingual/en-de/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) en-de__unigram_destarved
en-de-fair-5b model-checkpoints/bilingual/en-de/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) en-de__unigram_destarved__step5181_4753M
en-fr-bi-10b model-checkpoints/bilingual/en-fr/2-languages/10b 2 languages (unigram_bi_fr) 9.76B mid-stable (peak LR 3e-3) en-fr__unigram_bi_fr__step9929_9760M
en-fr-bi-15b model-checkpoints/bilingual/en-fr/2-languages/15b 2 languages (unigram_bi_fr) 14.764B mid-stable (peak LR 3e-3) en-fr__unigram_bi_fr__step15019_14764M
en-fr-bi-23b model-checkpoints/bilingual/en-fr/2-languages/23b 2 languages (unigram_bi_fr) 22.77B mid-stable (peak LR 3e-3) en-fr__unigram_bi_fr__step23163_22770M
en-fr-bi-2b model-checkpoints/bilingual/en-fr/2-languages/2b 2 languages (unigram_bi_fr) 1.754B mid-stable (peak LR 3e-3) en-fr__unigram_bi_fr__step1785_1754M
en-fr-bi model-checkpoints/bilingual/en-fr/2-languages/30b-final 2 languages (unigram_bi_fr) 30B cooled final (LR 3e-4) en-fr__unigram_bi_fr
en-fr-bi-5b model-checkpoints/bilingual/en-fr/2-languages/5b 2 languages (unigram_bi_fr) 4.756B mid-stable (peak LR 3e-3) en-fr__unigram_bi_fr__step4839_4756M
en-fr-20lang-10b model-checkpoints/bilingual/en-fr/20-languages/10b 20 languages (unigram_20lang) 9.76B mid-stable (peak LR 3e-3) en-fr__unigram_20lang__step9929_9760M
en-fr-20lang-15b model-checkpoints/bilingual/en-fr/20-languages/15b 20 languages (unigram_20lang) 14.764B mid-stable (peak LR 3e-3) en-fr__unigram_20lang__step15019_14764M
en-fr-20lang-23b model-checkpoints/bilingual/en-fr/20-languages/23b 20 languages (unigram_20lang) 22.766B mid-stable (peak LR 3e-3) en-fr__unigram_20lang__step23159_22766M
en-fr-20lang-2b model-checkpoints/bilingual/en-fr/20-languages/2b 20 languages (unigram_20lang) 1.754B mid-stable (peak LR 3e-3) en-fr__unigram_20lang__step1785_1754M
en-fr-20lang model-checkpoints/bilingual/en-fr/20-languages/30b-final 20 languages (unigram_20lang) 30B cooled final (LR 3e-4) en-fr__unigram_20lang
en-fr-20lang-5b model-checkpoints/bilingual/en-fr/20-languages/5b 20 languages (unigram_20lang) 4.756B mid-stable (peak LR 3e-3) en-fr__unigram_20lang__step4839_4756M
en-fr-starved-10b model-checkpoints/bilingual/en-fr/419-languages/10b 419 languages (unigram_starved) 9.753B mid-stable (peak LR 3e-3) en-fr__unigram_starved__step10631_9753M
en-fr-starved-15b model-checkpoints/bilingual/en-fr/419-languages/15b 419 languages (unigram_starved) 14.756B mid-stable (peak LR 3e-3) en-fr__unigram_starved__step15865_14756M
en-fr-starved-23b model-checkpoints/bilingual/en-fr/419-languages/23b 419 languages (unigram_starved) 22.758B mid-stable (peak LR 3e-3) en-fr__unigram_starved__step24005_22758M
en-fr-starved-2b model-checkpoints/bilingual/en-fr/419-languages/2b 419 languages (unigram_starved) 1.753B mid-stable (peak LR 3e-3) en-fr__unigram_starved__step1911_1753M
en-fr-starved model-checkpoints/bilingual/en-fr/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) en-fr__unigram_starved
en-fr-starved-5b model-checkpoints/bilingual/en-fr/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) en-fr__unigram_starved__step5181_4753M
en-fr-fair-10b model-checkpoints/bilingual/en-fr/5-languages/10b 5 languages (unigram_destarved) 9.753B mid-stable (peak LR 3e-3) en-fr__unigram_destarved__step10631_9753M
en-fr-fair-15b model-checkpoints/bilingual/en-fr/5-languages/15b 5 languages (unigram_destarved) 14.756B mid-stable (peak LR 3e-3) en-fr__unigram_destarved__step15865_14756M
en-fr-fair-23b model-checkpoints/bilingual/en-fr/5-languages/23b 5 languages (unigram_destarved) 22.758B mid-stable (peak LR 3e-3) en-fr__unigram_destarved__step24005_22758M
en-fr-fair-2b model-checkpoints/bilingual/en-fr/5-languages/2b 5 languages (unigram_destarved) 1.753B mid-stable (peak LR 3e-3) en-fr__unigram_destarved__step1911_1753M
en-fr-fair model-checkpoints/bilingual/en-fr/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) en-fr__unigram_destarved
en-fr-fair-5b model-checkpoints/bilingual/en-fr/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) en-fr__unigram_destarved__step5181_4753M
en-zh-bi-10b model-checkpoints/bilingual/en-zh/2-languages/10b 2 languages (unigram_bi_zh) 9.76B mid-stable (peak LR 3e-3) en-zh__unigram_bi_zh__step9929_9760M
en-zh-bi-15b model-checkpoints/bilingual/en-zh/2-languages/15b 2 languages (unigram_bi_zh) 14.764B mid-stable (peak LR 3e-3) en-zh__unigram_bi_zh__step15019_14764M
en-zh-bi-23b model-checkpoints/bilingual/en-zh/2-languages/23b 2 languages (unigram_bi_zh) 22.77B mid-stable (peak LR 3e-3) en-zh__unigram_bi_zh__step23163_22770M
en-zh-bi-2b model-checkpoints/bilingual/en-zh/2-languages/2b 2 languages (unigram_bi_zh) 1.754B mid-stable (peak LR 3e-3) en-zh__unigram_bi_zh__step1785_1754M
en-zh-bi model-checkpoints/bilingual/en-zh/2-languages/30b-final 2 languages (unigram_bi_zh) 30B cooled final (LR 3e-4) en-zh__unigram_bi_zh
en-zh-bi-5b model-checkpoints/bilingual/en-zh/2-languages/5b 2 languages (unigram_bi_zh) 4.756B mid-stable (peak LR 3e-3) en-zh__unigram_bi_zh__step4839_4756M
en-zh-20lang-10b model-checkpoints/bilingual/en-zh/20-languages/10b 20 languages (unigram_20lang) 9.76B mid-stable (peak LR 3e-3) en-zh__unigram_20lang__step9929_9760M
en-zh-20lang-15b model-checkpoints/bilingual/en-zh/20-languages/15b 20 languages (unigram_20lang) 14.764B mid-stable (peak LR 3e-3) en-zh__unigram_20lang__step15019_14764M
en-zh-20lang-23b model-checkpoints/bilingual/en-zh/20-languages/23b 20 languages (unigram_20lang) 22.766B mid-stable (peak LR 3e-3) en-zh__unigram_20lang__step23159_22766M
en-zh-20lang-2b model-checkpoints/bilingual/en-zh/20-languages/2b 20 languages (unigram_20lang) 1.754B mid-stable (peak LR 3e-3) en-zh__unigram_20lang__step1785_1754M
en-zh-20lang model-checkpoints/bilingual/en-zh/20-languages/30b-final 20 languages (unigram_20lang) 30B cooled final (LR 3e-4) en-zh__unigram_20lang
en-zh-20lang-5b model-checkpoints/bilingual/en-zh/20-languages/5b 20 languages (unigram_20lang) 4.756B mid-stable (peak LR 3e-3) en-zh__unigram_20lang__step4839_4756M
en-zh-starved-10b model-checkpoints/bilingual/en-zh/419-languages/10b 419 languages (unigram_starved) 9.753B mid-stable (peak LR 3e-3) en-zh__unigram_starved__step10631_9753M
en-zh-starved-15b model-checkpoints/bilingual/en-zh/419-languages/15b 419 languages (unigram_starved) 14.756B mid-stable (peak LR 3e-3) en-zh__unigram_starved__step15865_14756M
en-zh-starved-23b model-checkpoints/bilingual/en-zh/419-languages/23b 419 languages (unigram_starved) 22.758B mid-stable (peak LR 3e-3) en-zh__unigram_starved__step24005_22758M
en-zh-starved-2b model-checkpoints/bilingual/en-zh/419-languages/2b 419 languages (unigram_starved) 1.753B mid-stable (peak LR 3e-3) en-zh__unigram_starved__step1911_1753M
en-zh-starved model-checkpoints/bilingual/en-zh/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) en-zh__unigram_starved
en-zh-starved-5b model-checkpoints/bilingual/en-zh/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) en-zh__unigram_starved__step5181_4753M
en-zh-fair-10b model-checkpoints/bilingual/en-zh/5-languages/10b 5 languages (unigram_destarved) 9.753B mid-stable (peak LR 3e-3) en-zh__unigram_destarved__step10631_9753M
en-zh-fair-15b model-checkpoints/bilingual/en-zh/5-languages/15b 5 languages (unigram_destarved) 14.755B mid-stable (peak LR 3e-3) en-zh__unigram_destarved__step15937_14755M
en-zh-fair-23b model-checkpoints/bilingual/en-zh/5-languages/23b 5 languages (unigram_destarved) 22.757B mid-stable (peak LR 3e-3) en-zh__unigram_destarved__step24077_22757M
en-zh-fair-2b model-checkpoints/bilingual/en-zh/5-languages/2b 5 languages (unigram_destarved) 1.753B mid-stable (peak LR 3e-3) en-zh__unigram_destarved__step1911_1753M
en-zh-fair model-checkpoints/bilingual/en-zh/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) en-zh__unigram_destarved
en-zh-fair-5b model-checkpoints/bilingual/en-zh/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) en-zh__unigram_destarved__step5181_4753M
ar-starved-12b model-checkpoints/monolingual/ar/419-languages/12b 419 languages (unigram_starved) 11.754B mid-stable (peak LR 3e-3) ar__unigram_starved__step12811_11754M
ar-starved-15b model-checkpoints/monolingual/ar/419-languages/15b 419 languages (unigram_starved) 14.754B mid-stable (peak LR 3e-3) ar__unigram_starved__step16081_14754M
ar-starved-1b model-checkpoints/monolingual/ar/419-languages/1b 419 languages (unigram_starved) 1.001B mid-stable (peak LR 3e-3) ar__unigram_starved__step1092_1001M
ar-starved-2b model-checkpoints/monolingual/ar/419-languages/2b 419 languages (unigram_starved) 2.253B mid-stable (peak LR 3e-3) ar__unigram_starved__step2456_2253M
ar-starved model-checkpoints/monolingual/ar/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) ar__unigram_starved
ar-starved-5b model-checkpoints/monolingual/ar/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) ar__unigram_starved__step5181_4753M
ar-starved-8b model-checkpoints/monolingual/ar/419-languages/8b 419 languages (unigram_starved) 7.753B mid-stable (peak LR 3e-3) ar__unigram_starved__step8451_7753M
ar-fair-12b model-checkpoints/monolingual/ar/5-languages/12b 5 languages (unigram_destarved) 11.754B mid-stable (peak LR 3e-3) ar__unigram_destarved__step12811_11754M
ar-fair-15b model-checkpoints/monolingual/ar/5-languages/15b 5 languages (unigram_destarved) 14.754B mid-stable (peak LR 3e-3) ar__unigram_destarved__step16081_14754M
ar-fair-1b model-checkpoints/monolingual/ar/5-languages/1b 5 languages (unigram_destarved) 1.001B mid-stable (peak LR 3e-3) ar__unigram_destarved__step1092_1001M
ar-fair-2b model-checkpoints/monolingual/ar/5-languages/2b 5 languages (unigram_destarved) 2.253B mid-stable (peak LR 3e-3) ar__unigram_destarved__step2456_2253M
ar-fair model-checkpoints/monolingual/ar/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) ar__unigram_destarved
ar-fair-5b model-checkpoints/monolingual/ar/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) ar__unigram_destarved__step5181_4753M
ar-fair-8b model-checkpoints/monolingual/ar/5-languages/8b 5 languages (unigram_destarved) 7.753B mid-stable (peak LR 3e-3) ar__unigram_destarved__step8451_7753M
de-starved-12b model-checkpoints/monolingual/de/419-languages/12b 419 languages (unigram_starved) 11.754B mid-stable (peak LR 3e-3) de__unigram_starved__step12811_11754M
de-starved-15b model-checkpoints/monolingual/de/419-languages/15b 419 languages (unigram_starved) 14.754B mid-stable (peak LR 3e-3) de__unigram_starved__step16081_14754M
de-starved-16b model-checkpoints/monolingual/de/419-languages/16b 419 languages (unigram_starved) 16.1B mid-stable (peak LR 3e-3) de__unigram_starved
de-starved-1b model-checkpoints/monolingual/de/419-languages/1b 419 languages (unigram_starved) 1.001B mid-stable (peak LR 3e-3) de__unigram_starved__step1092_1001M
de-starved-2b model-checkpoints/monolingual/de/419-languages/2b 419 languages (unigram_starved) 2.253B mid-stable (peak LR 3e-3) de__unigram_starved__step2456_2253M
de-starved-5b model-checkpoints/monolingual/de/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) de__unigram_starved__step5181_4753M
de-starved-8b model-checkpoints/monolingual/de/419-languages/8b 419 languages (unigram_starved) 7.753B mid-stable (peak LR 3e-3) de__unigram_starved__step8451_7753M
de-fair-12b model-checkpoints/monolingual/de/5-languages/12b 5 languages (unigram_destarved) 11.754B mid-stable (peak LR 3e-3) de__unigram_destarved__step12811_11754M
de-fair-15b model-checkpoints/monolingual/de/5-languages/15b 5 languages (unigram_destarved) 14.755B mid-stable (peak LR 3e-3) de__unigram_destarved__step16009_14755M
de-fair-1b model-checkpoints/monolingual/de/5-languages/1b 5 languages (unigram_destarved) 1.001B mid-stable (peak LR 3e-3) de__unigram_destarved__step1092_1001M
de-fair-2b model-checkpoints/monolingual/de/5-languages/2b 5 languages (unigram_destarved) 2.253B mid-stable (peak LR 3e-3) de__unigram_destarved__step2456_2253M
de-fair model-checkpoints/monolingual/de/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) de__unigram_destarved
de-fair-5b model-checkpoints/monolingual/de/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) de__unigram_destarved__step5181_4753M
de-fair-8b model-checkpoints/monolingual/de/5-languages/8b 5 languages (unigram_destarved) 7.753B mid-stable (peak LR 3e-3) de__unigram_destarved__step8451_7753M
en-starved-12b model-checkpoints/monolingual/en/419-languages/12b 419 languages (unigram_starved) 11.754B mid-stable (peak LR 3e-3) en__unigram_starved__step12739_11754M
en-starved-15b model-checkpoints/monolingual/en/419-languages/15b 419 languages (unigram_starved) 14.757B mid-stable (peak LR 3e-3) en__unigram_starved__step15793_14757M
en-starved-1b model-checkpoints/monolingual/en/419-languages/1b 419 languages (unigram_starved) 1.001B mid-stable (peak LR 3e-3) en__unigram_starved__step1092_1001M
en-starved-2b model-checkpoints/monolingual/en/419-languages/2b 419 languages (unigram_starved) 2.253B mid-stable (peak LR 3e-3) en__unigram_starved__step2456_2253M
en-starved model-checkpoints/monolingual/en/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) en__unigram_starved
en-starved-5b model-checkpoints/monolingual/en/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) en__unigram_starved__step5181_4753M
en-starved-8b model-checkpoints/monolingual/en/419-languages/8b 419 languages (unigram_starved) 7.753B mid-stable (peak LR 3e-3) en__unigram_starved__step8451_7753M
en-fair-12b model-checkpoints/monolingual/en/5-languages/12b 5 languages (unigram_destarved) 11.754B mid-stable (peak LR 3e-3) en__unigram_destarved__step12811_11754M
en-fair-15b model-checkpoints/monolingual/en/5-languages/15b 5 languages (unigram_destarved) 14.755B mid-stable (peak LR 3e-3) en__unigram_destarved__step15937_14755M
en-fair-1b model-checkpoints/monolingual/en/5-languages/1b 5 languages (unigram_destarved) 1.001B mid-stable (peak LR 3e-3) en__unigram_destarved__step1092_1001M
en-fair-2b model-checkpoints/monolingual/en/5-languages/2b 5 languages (unigram_destarved) 2.253B mid-stable (peak LR 3e-3) en__unigram_destarved__step2456_2253M
en-fair model-checkpoints/monolingual/en/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) en__unigram_destarved
en-fair-5b model-checkpoints/monolingual/en/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) en__unigram_destarved__step5181_4753M
en-fair-8b model-checkpoints/monolingual/en/5-languages/8b 5 languages (unigram_destarved) 7.753B mid-stable (peak LR 3e-3) en__unigram_destarved__step8451_7753M
fr-starved-12b model-checkpoints/monolingual/fr/419-languages/12b 419 languages (unigram_starved) 11.754B mid-stable (peak LR 3e-3) fr__unigram_starved__step12811_11754M
fr-starved-15b model-checkpoints/monolingual/fr/419-languages/15b 419 languages (unigram_starved) 14.755B mid-stable (peak LR 3e-3) fr__unigram_starved__step16009_14755M
fr-starved-1b model-checkpoints/monolingual/fr/419-languages/1b 419 languages (unigram_starved) 1.001B mid-stable (peak LR 3e-3) fr__unigram_starved__step1092_1001M
fr-starved-2b model-checkpoints/monolingual/fr/419-languages/2b 419 languages (unigram_starved) 2.253B mid-stable (peak LR 3e-3) fr__unigram_starved__step2456_2253M
fr-starved model-checkpoints/monolingual/fr/419-languages/30b-final 419 languages (unigram_starved) 30B cooled final (LR 3e-4) fr__unigram_starved
fr-starved-5b model-checkpoints/monolingual/fr/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) fr__unigram_starved__step5181_4753M
fr-starved-8b model-checkpoints/monolingual/fr/419-languages/8b 419 languages (unigram_starved) 7.753B mid-stable (peak LR 3e-3) fr__unigram_starved__step8451_7753M
fr-fair-12b model-checkpoints/monolingual/fr/5-languages/12b 5 languages (unigram_destarved) 11.754B mid-stable (peak LR 3e-3) fr__unigram_destarved__step12811_11754M
fr-fair-15b model-checkpoints/monolingual/fr/5-languages/15b 5 languages (unigram_destarved) 14.755B mid-stable (peak LR 3e-3) fr__unigram_destarved__step16009_14755M
fr-fair-1b model-checkpoints/monolingual/fr/5-languages/1b 5 languages (unigram_destarved) 1.001B mid-stable (peak LR 3e-3) fr__unigram_destarved__step1092_1001M
fr-fair-2b model-checkpoints/monolingual/fr/5-languages/2b 5 languages (unigram_destarved) 2.253B mid-stable (peak LR 3e-3) fr__unigram_destarved__step2456_2253M
fr-fair model-checkpoints/monolingual/fr/5-languages/30b-final 5 languages (unigram_destarved) 30B cooled final (LR 3e-4) fr__unigram_destarved
fr-fair-5b model-checkpoints/monolingual/fr/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) fr__unigram_destarved__step5181_4753M
fr-fair-8b model-checkpoints/monolingual/fr/5-languages/8b 5 languages (unigram_destarved) 7.753B mid-stable (peak LR 3e-3) fr__unigram_destarved__step8451_7753M
zh-starved-12b model-checkpoints/monolingual/zh/419-languages/12b 419 languages (unigram_starved) 11.754B mid-stable (peak LR 3e-3) zh__unigram_starved__step12811_11754M
zh-starved-15b model-checkpoints/monolingual/zh/419-languages/15b 419 languages (unigram_starved) 14.756B mid-stable (peak LR 3e-3) zh__unigram_starved__step15865_14756M
zh-starved-1b model-checkpoints/monolingual/zh/419-languages/1b 419 languages (unigram_starved) 1.001B mid-stable (peak LR 3e-3) zh__unigram_starved__step1092_1001M
zh-starved-2b model-checkpoints/monolingual/zh/419-languages/2b 419 languages (unigram_starved) 2.253B mid-stable (peak LR 3e-3) zh__unigram_starved__step2456_2253M
zh-starved-5b model-checkpoints/monolingual/zh/419-languages/5b 419 languages (unigram_starved) 4.753B mid-stable (peak LR 3e-3) zh__unigram_starved__step5181_4753M
zh-starved-8b model-checkpoints/monolingual/zh/419-languages/8b 419 languages (unigram_starved) 7.753B mid-stable (peak LR 3e-3) zh__unigram_starved__step8451_7753M
zh-fair-12b model-checkpoints/monolingual/zh/5-languages/12b 5 languages (unigram_destarved) 11.754B mid-stable (peak LR 3e-3) zh__unigram_destarved__step12811_11754M
zh-fair-15b model-checkpoints/monolingual/zh/5-languages/15b 5 languages (unigram_destarved) 14.756B mid-stable (peak LR 3e-3) zh__unigram_destarved__step15865_14756M
zh-fair-1b model-checkpoints/monolingual/zh/5-languages/1b 5 languages (unigram_destarved) 1.001B mid-stable (peak LR 3e-3) zh__unigram_destarved__step1092_1001M
zh-fair-2b model-checkpoints/monolingual/zh/5-languages/2b 5 languages (unigram_destarved) 2.253B mid-stable (peak LR 3e-3) zh__unigram_destarved__step2456_2253M
zh-fair-5b model-checkpoints/monolingual/zh/5-languages/5b 5 languages (unigram_destarved) 4.753B mid-stable (peak LR 3e-3) zh__unigram_destarved__step5181_4753M
zh-fair-8b model-checkpoints/monolingual/zh/5-languages/8b 5 languages (unigram_destarved) 7.753B mid-stable (peak LR 3e-3) zh__unigram_destarved__step8451_7753M

Training states

training-states/ holds the full training-state archives (step checkpoints, last.pt, per-rank optimizer shards) of the 20- and 2-language bilingual runs (en-X__unigram_20lang, en-X__unigram_bi_X) and of the abandoned 100B-token extensions (en-{de,ar}__unigram_{destarved,starved}__100b, en-ar__*__100b_scratch). They are only needed to resume or re-analyse training, not for evaluation.

Older material

src/xscript/, run_benchmarks.py and requirements.txt at the top level are an early export of the code and predate the split-checkpoint layout; use the GitHub repo instead.

Original runbook (outdated)

The README this repository shipped with is kept below; its heading levels are shifted down one, nothing else is changed. It describes the first 15-checkpoint export (whole final.pt files, three benchmarks) and is out of date on the layout and the model count.

External benchmark evaluation

Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer (src/xscript/model.py) + SentencePiece tokenizer β€” pure PyTorch, using F.scaled_dot_product_attention, no flash-attn / triton / custom kernels β€” so they run on any stock GPU (or CPU, slowly). Each model is ~1B params (fits any 16GB GPU).

The benchmark harness (src/xscript/eval/bench.py) wraps our model into lm-evaluation-harness and scores Global-MMLU, Belebele, and XNLI on each run's training languages. It is the same harness we would have run on-cluster, so numbers are directly comparable.

1. Export from Isambard (already done by upload_to_hf.py)

The private HF repo mirrors the on-cluster layout:

src/xscript/**                       # bundled model + harness code
tokenizers/unigram_{starved,destarved}/{sp.model,meta.json}
runs/<name>/checkpoints/final.pt     # 15 checkpoints, fp32, ~4GB each
models.json                          # friendly name -> tokenizer + langs + orig run
run_benchmarks.py  requirements.txt  README.md

Models use friendly names <mixture>-<starved|fair> (e.g. en-fair, en-ar-starved). models.json maps each to its real tokenizer.

2. Run on your GPU

## clone just the runner (or download run_benchmarks.py + requirements.txt from the repo)
pip install torch --index-url https://download.pytorch.org/whl/cu121   # match your CUDA
pip install -r requirements.txt
export HF_TOKEN=hf_...        # while the repo is private

## quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST
python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200

## full suite once the quick pass looks sane
python run_benchmarks.py --repo jvonrad/xscript-eval

The runner downloads one checkpoint at a time and deletes it after eval (--keep-checkpoints to retain), so peak disk is ~5GB. Results:

xscript_bench/results/bench/<run>_final.json    # per-run task accuracies
xscript_bench/results/summary.json              # everything combined

Send those JSONs back for analysis.

Notes

  • --runs en-starved en-fair limits to a subset (friendly names).
  • --tasks xnli_en xnli_de overrides the task list (default = the run's langs).
  • Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs).
  • Scores are ordinary accuracy (acc,none); raw harness output is preserved in each per-run JSON for length-normalized variants.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including jvonrad/alignment-without-transfer