Download README.md from jvonrad/alignment-without-transfer: direct link, hf CLI and curl.
- Browser
- Download file 63.5 kB
-
https://huggingface.co/jvonrad/alignment-without-transfer/resolve/main/README.md
- Command line
-
hf download hf://jvonrad/alignment-without-transfer/README.md
-
curl -L -o README.md https://huggingface.co/jvonrad/alignment-without-transfer/resolve/main/README.md
language:
- en
- de
- fr
- ar
- zh
tags:
- multilingual
- pretraining
- tokenization
- cross-lingual-transfer
- xscript-pretraining
Alignment without Transfer — model checkpoints
Checkpoints for the paper Alignment without Transfer: Tokenization Impacts Cross-lingual Representations during Multilingual Language Model Pretraining (preprint forthcoming; not yet on arXiv).
About 1B-parameter Llama-style decoders (16 layers, d=2048, SwiGLU, RoPE, untied embeddings, 65,536-piece vocabulary) trained from scratch on one or two of English, German, French, Arabic and Chinese (FineWeb-HQ / FineWeb2-HQ). Model pairs differ only in the tokenizer, i.e. in how many languages share the 65k vocabulary: 419, 20, 5 or 2.
164 checkpoints: 96 bilingual (English + one partner language) and 68 monolingual, each at several token budgets.
Everything else for the paper is in the Hugging Face collection and on GitHub:
| code, experiment log | github.com/jvonrad/XScript-Pretraining |
| cached alignment embeddings | jvonrad/alignment-without-transfer-embeddings |
| data provenance (pool file lists, holdout text) | jvonrad/alignment-without-transfer-data |
Layout
model-checkpoints/
bilingual/<en-X>/<N>-languages/<budget>/ X in de, fr, ar, zh
monolingual/<lang>/<N>-languages/<budget>/ lang in en, de, fr, ar, zh
final.pt.part000 ... part004 one torch.save file split in 5 parts
n_parts.txt number of parts
tokenizers/<tokenizer>/{sp.model, meta.json}
models.json model name -> path, tokenizer, languages, budget, LR state
training-states/ optimizer/resume states of some runs (not needed for evaluation)
<N>-languages— how many languages the tokenizer was trained on:folder tokenizer trained on 2-languagesunigram_bi_<X>English + the partner language only 5-languagesunigram_destarvedthe 5 study languages (byte-premium adjusted) 20-languagesunigram_20langthe 5 study languages + ces, fas, hun, ind, ita, jpn, kor, nld, pol, por, ron, rus, swe, ukr, vie 419-languagesunigram_starved~419 FineWeb/FineWeb2 languages (T=100) All are SentencePiece Unigram, 65,536 pieces, byte fallback, ids 0–3 =
<unk>/<bos>/<eos>/<pad>.<budget>— total training tokens (for bilinguals half per language):2b…23bare mid-stable snapshots at peak learning rate;30b-finalis the cooled final model.
⚠️ Compare like with like
All runs share one warmup–stable–decay schedule: warmup to 1B tokens, peak LR
3.0e-3 until 24B, decay to 3.0e-4 by 30B. A 30b-final checkpoint has been
through the decay; every other budget has not. Pairing the two hands the cooled
model the whole decay phase for free. Exceptions: monolingual/de/419-languages/16b
(a re-seeded retrain that stops at 16.1B, not cooled) is the largest German
419-language model, and Chinese monolinguals exist only up to 15b.
Model names
models.json keeps the short names the code and result files use:
<langs>-<tok>[-<budget>], with <tok> = starved (419 languages), 20lang
(20), fair (5) or bi (2), and no budget suffix for the cooled final, e.g.
en-ar-fair-12b → model-checkpoints/bilingual/en-ar/5-languages/12b.
Loading a checkpoint
import json, shutil
from huggingface_hub import hf_hub_download
repo, name = "jvonrad/alignment-without-transfer", "en-ar-fair" # any key of models.json
spec = json.load(open(hf_hub_download(repo, "models.json")))[name]
d = spec["path"] # model-checkpoints/bilingual/en-ar/5-languages/30b-final
n = int(open(hf_hub_download(repo, f"{d}/n_parts.txt")).read())
with open("final.pt", "wb") as out:
for i in range(n):
with open(hf_hub_download(repo, f"{d}/final.pt.part{i:03d}"), "rb") as part:
shutil.copyfileobj(part, out)
sp_model = hf_hub_download(repo, f"tokenizers/{spec['tok']}/sp.model")
# with the GitHub repo's src/ on sys.path:
import torch
from xscript.model import ModelConfig, Transformer
ck = torch.load("final.pt", map_location="cpu", weights_only=False)
model = Transformer(ModelConfig(**ck["cfg"]["model"]))
model.load_state_dict(ck["model"])
The GitHub repo's evaluation scripts do this automatically
(scripts/external_bench/hf_layout.py maps model names to these paths).
Checkpoint index
Each cell links to the checkpoint folder.
Bilingual
en-de
| tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final |
|---|---|---|---|---|---|---|
| 2 languages | en-de-bi-2b |
en-de-bi-5b |
en-de-bi-10b |
en-de-bi-15b |
en-de-bi-23b |
en-de-bi |
| 5 languages | en-de-fair-2b |
en-de-fair-5b |
en-de-fair-10b |
en-de-fair-15b |
en-de-fair-23b |
en-de-fair |
| 20 languages | en-de-20lang-2b |
en-de-20lang-5b |
en-de-20lang-10b |
en-de-20lang-15b |
en-de-20lang-23b |
en-de-20lang |
| 419 languages | en-de-starved-2b |
en-de-starved-5b |
en-de-starved-10b |
en-de-starved-15b |
en-de-starved-23b |
en-de-starved |
en-fr
| tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final |
|---|---|---|---|---|---|---|
| 2 languages | en-fr-bi-2b |
en-fr-bi-5b |
en-fr-bi-10b |
en-fr-bi-15b |
en-fr-bi-23b |
en-fr-bi |
| 5 languages | en-fr-fair-2b |
en-fr-fair-5b |
en-fr-fair-10b |
en-fr-fair-15b |
en-fr-fair-23b |
en-fr-fair |
| 20 languages | en-fr-20lang-2b |
en-fr-20lang-5b |
en-fr-20lang-10b |
en-fr-20lang-15b |
en-fr-20lang-23b |
en-fr-20lang |
| 419 languages | en-fr-starved-2b |
en-fr-starved-5b |
en-fr-starved-10b |
en-fr-starved-15b |
en-fr-starved-23b |
en-fr-starved |
en-ar
| tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final |
|---|---|---|---|---|---|---|
| 2 languages | en-ar-bi-2b |
en-ar-bi-5b |
en-ar-bi-10b |
en-ar-bi-15b |
en-ar-bi-23b |
en-ar-bi |
| 5 languages | en-ar-fair-2b |
en-ar-fair-5b |
en-ar-fair-10b |
en-ar-fair-15b |
en-ar-fair-23b |
en-ar-fair |
| 20 languages | en-ar-20lang-2b |
en-ar-20lang-5b |
en-ar-20lang-10b |
en-ar-20lang-15b |
en-ar-20lang-23b |
en-ar-20lang |
| 419 languages | en-ar-starved-2b |
en-ar-starved-5b |
en-ar-starved-10b |
en-ar-starved-15b |
en-ar-starved-23b |
en-ar-starved |
en-zh
| tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final |
|---|---|---|---|---|---|---|
| 2 languages | en-zh-bi-2b |
en-zh-bi-5b |
en-zh-bi-10b |
en-zh-bi-15b |
en-zh-bi-23b |
en-zh-bi |
| 5 languages | en-zh-fair-2b |
en-zh-fair-5b |
en-zh-fair-10b |
en-zh-fair-15b |
en-zh-fair-23b |
en-zh-fair |
| 20 languages | en-zh-20lang-2b |
en-zh-20lang-5b |
en-zh-20lang-10b |
en-zh-20lang-15b |
en-zh-20lang-23b |
en-zh-20lang |
| 419 languages | en-zh-starved-2b |
en-zh-starved-5b |
en-zh-starved-10b |
en-zh-starved-15b |
en-zh-starved-23b |
en-zh-starved |
Monolingual
en
| tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 30b-final |
|---|---|---|---|---|---|---|---|
| 5 languages | en-fair-1b |
en-fair-2b |
en-fair-5b |
en-fair-8b |
en-fair-12b |
en-fair-15b |
en-fair |
| 419 languages | en-starved-1b |
en-starved-2b |
en-starved-5b |
en-starved-8b |
en-starved-12b |
en-starved-15b |
en-starved |
de
| tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 16b | 30b-final |
|---|---|---|---|---|---|---|---|---|
| 5 languages | de-fair-1b |
de-fair-2b |
de-fair-5b |
de-fair-8b |
de-fair-12b |
de-fair-15b |
de-fair |
|
| 419 languages | de-starved-1b |
de-starved-2b |
de-starved-5b |
de-starved-8b |
de-starved-12b |
de-starved-15b |
de-starved-16b |
fr
| tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 30b-final |
|---|---|---|---|---|---|---|---|
| 5 languages | fr-fair-1b |
fr-fair-2b |
fr-fair-5b |
fr-fair-8b |
fr-fair-12b |
fr-fair-15b |
fr-fair |
| 419 languages | fr-starved-1b |
fr-starved-2b |
fr-starved-5b |
fr-starved-8b |
fr-starved-12b |
fr-starved-15b |
fr-starved |
ar
| tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 30b-final |
|---|---|---|---|---|---|---|---|
| 5 languages | ar-fair-1b |
ar-fair-2b |
ar-fair-5b |
ar-fair-8b |
ar-fair-12b |
ar-fair-15b |
ar-fair |
| 419 languages | ar-starved-1b |
ar-starved-2b |
ar-starved-5b |
ar-starved-8b |
ar-starved-12b |
ar-starved-15b |
ar-starved |
zh
| tokenizer | 1b | 2b | 5b | 8b | 12b | 15b |
|---|---|---|---|---|---|---|
| 5 languages | zh-fair-1b |
zh-fair-2b |
zh-fair-5b |
zh-fair-8b |
zh-fair-12b |
zh-fair-15b |
| 419 languages | zh-starved-1b |
zh-starved-2b |
zh-starved-5b |
zh-starved-8b |
zh-starved-12b |
zh-starved-15b |
Full table (path, exact token count, LR state, original run name)
| model | path | tokenizer | tokens | LR state | original run |
|---|---|---|---|---|---|
en-ar-bi-10b |
model-checkpoints/bilingual/en-ar/2-languages/10b |
2 languages (unigram_bi_ar) |
9.76B | mid-stable (peak LR 3e-3) | en-ar__unigram_bi_ar__step9929_9760M |
en-ar-bi-15b |
model-checkpoints/bilingual/en-ar/2-languages/15b |
2 languages (unigram_bi_ar) |
14.764B | mid-stable (peak LR 3e-3) | en-ar__unigram_bi_ar__step15019_14764M |
en-ar-bi-23b |
model-checkpoints/bilingual/en-ar/2-languages/23b |
2 languages (unigram_bi_ar) |
22.77B | mid-stable (peak LR 3e-3) | en-ar__unigram_bi_ar__step23163_22770M |
en-ar-bi-2b |
model-checkpoints/bilingual/en-ar/2-languages/2b |
2 languages (unigram_bi_ar) |
1.754B | mid-stable (peak LR 3e-3) | en-ar__unigram_bi_ar__step1785_1754M |
en-ar-bi |
model-checkpoints/bilingual/en-ar/2-languages/30b-final |
2 languages (unigram_bi_ar) |
30B | cooled final (LR 3e-4) | en-ar__unigram_bi_ar |
en-ar-bi-5b |
model-checkpoints/bilingual/en-ar/2-languages/5b |
2 languages (unigram_bi_ar) |
4.756B | mid-stable (peak LR 3e-3) | en-ar__unigram_bi_ar__step4839_4756M |
en-ar-20lang-10b |
model-checkpoints/bilingual/en-ar/20-languages/10b |
20 languages (unigram_20lang) |
9.76B | mid-stable (peak LR 3e-3) | en-ar__unigram_20lang__step9929_9760M |
en-ar-20lang-15b |
model-checkpoints/bilingual/en-ar/20-languages/15b |
20 languages (unigram_20lang) |
14.764B | mid-stable (peak LR 3e-3) | en-ar__unigram_20lang__step15019_14764M |
en-ar-20lang-23b |
model-checkpoints/bilingual/en-ar/20-languages/23b |
20 languages (unigram_20lang) |
22.766B | mid-stable (peak LR 3e-3) | en-ar__unigram_20lang__step23159_22766M |
en-ar-20lang-2b |
model-checkpoints/bilingual/en-ar/20-languages/2b |
20 languages (unigram_20lang) |
1.754B | mid-stable (peak LR 3e-3) | en-ar__unigram_20lang__step1785_1754M |
en-ar-20lang |
model-checkpoints/bilingual/en-ar/20-languages/30b-final |
20 languages (unigram_20lang) |
30B | cooled final (LR 3e-4) | en-ar__unigram_20lang |
en-ar-20lang-5b |
model-checkpoints/bilingual/en-ar/20-languages/5b |
20 languages (unigram_20lang) |
4.756B | mid-stable (peak LR 3e-3) | en-ar__unigram_20lang__step4839_4756M |
en-ar-starved-10b |
model-checkpoints/bilingual/en-ar/419-languages/10b |
419 languages (unigram_starved) |
9.753B | mid-stable (peak LR 3e-3) | en-ar__unigram_starved__step10631_9753M |
en-ar-starved-15b |
model-checkpoints/bilingual/en-ar/419-languages/15b |
419 languages (unigram_starved) |
14.757B | mid-stable (peak LR 3e-3) | en-ar__unigram_starved__step15793_14757M |
en-ar-starved-23b |
model-checkpoints/bilingual/en-ar/419-languages/23b |
419 languages (unigram_starved) |
22.758B | mid-stable (peak LR 3e-3) | en-ar__unigram_starved__step23933_22758M |
en-ar-starved-2b |
model-checkpoints/bilingual/en-ar/419-languages/2b |
419 languages (unigram_starved) |
1.753B | mid-stable (peak LR 3e-3) | en-ar__unigram_starved__step1911_1753M |
en-ar-starved |
model-checkpoints/bilingual/en-ar/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | en-ar__unigram_starved |
en-ar-starved-5b |
model-checkpoints/bilingual/en-ar/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | en-ar__unigram_starved__step5181_4753M |
en-ar-fair-10b |
model-checkpoints/bilingual/en-ar/5-languages/10b |
5 languages (unigram_destarved) |
9.753B | mid-stable (peak LR 3e-3) | en-ar__unigram_destarved__step10631_9753M |
en-ar-fair-15b |
model-checkpoints/bilingual/en-ar/5-languages/15b |
5 languages (unigram_destarved) |
14.755B | mid-stable (peak LR 3e-3) | en-ar__unigram_destarved__step15937_14755M |
en-ar-fair-23b |
model-checkpoints/bilingual/en-ar/5-languages/23b |
5 languages (unigram_destarved) |
22.757B | mid-stable (peak LR 3e-3) | en-ar__unigram_destarved__step24077_22757M |
en-ar-fair-2b |
model-checkpoints/bilingual/en-ar/5-languages/2b |
5 languages (unigram_destarved) |
1.753B | mid-stable (peak LR 3e-3) | en-ar__unigram_destarved__step1911_1753M |
en-ar-fair |
model-checkpoints/bilingual/en-ar/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | en-ar__unigram_destarved |
en-ar-fair-5b |
model-checkpoints/bilingual/en-ar/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | en-ar__unigram_destarved__step5181_4753M |
en-de-bi-10b |
model-checkpoints/bilingual/en-de/2-languages/10b |
2 languages (unigram_bi_de) |
9.76B | mid-stable (peak LR 3e-3) | en-de__unigram_bi_de__step9929_9760M |
en-de-bi-15b |
model-checkpoints/bilingual/en-de/2-languages/15b |
2 languages (unigram_bi_de) |
14.764B | mid-stable (peak LR 3e-3) | en-de__unigram_bi_de__step15019_14764M |
en-de-bi-23b |
model-checkpoints/bilingual/en-de/2-languages/23b |
2 languages (unigram_bi_de) |
22.77B | mid-stable (peak LR 3e-3) | en-de__unigram_bi_de__step23163_22770M |
en-de-bi-2b |
model-checkpoints/bilingual/en-de/2-languages/2b |
2 languages (unigram_bi_de) |
1.754B | mid-stable (peak LR 3e-3) | en-de__unigram_bi_de__step1785_1754M |
en-de-bi |
model-checkpoints/bilingual/en-de/2-languages/30b-final |
2 languages (unigram_bi_de) |
30B | cooled final (LR 3e-4) | en-de__unigram_bi_de |
en-de-bi-5b |
model-checkpoints/bilingual/en-de/2-languages/5b |
2 languages (unigram_bi_de) |
4.756B | mid-stable (peak LR 3e-3) | en-de__unigram_bi_de__step4839_4756M |
en-de-20lang-10b |
model-checkpoints/bilingual/en-de/20-languages/10b |
20 languages (unigram_20lang) |
9.76B | mid-stable (peak LR 3e-3) | en-de__unigram_20lang__step9929_9760M |
en-de-20lang-15b |
model-checkpoints/bilingual/en-de/20-languages/15b |
20 languages (unigram_20lang) |
14.764B | mid-stable (peak LR 3e-3) | en-de__unigram_20lang__step15019_14764M |
en-de-20lang-23b |
model-checkpoints/bilingual/en-de/20-languages/23b |
20 languages (unigram_20lang) |
22.766B | mid-stable (peak LR 3e-3) | en-de__unigram_20lang__step23159_22766M |
en-de-20lang-2b |
model-checkpoints/bilingual/en-de/20-languages/2b |
20 languages (unigram_20lang) |
1.754B | mid-stable (peak LR 3e-3) | en-de__unigram_20lang__step1785_1754M |
en-de-20lang |
model-checkpoints/bilingual/en-de/20-languages/30b-final |
20 languages (unigram_20lang) |
30B | cooled final (LR 3e-4) | en-de__unigram_20lang |
en-de-20lang-5b |
model-checkpoints/bilingual/en-de/20-languages/5b |
20 languages (unigram_20lang) |
4.756B | mid-stable (peak LR 3e-3) | en-de__unigram_20lang__step4839_4756M |
en-de-starved-10b |
model-checkpoints/bilingual/en-de/419-languages/10b |
419 languages (unigram_starved) |
9.753B | mid-stable (peak LR 3e-3) | en-de__unigram_starved__step10631_9753M |
en-de-starved-15b |
model-checkpoints/bilingual/en-de/419-languages/15b |
419 languages (unigram_starved) |
14.756B | mid-stable (peak LR 3e-3) | en-de__unigram_starved__step15865_14756M |
en-de-starved-23b |
model-checkpoints/bilingual/en-de/419-languages/23b |
419 languages (unigram_starved) |
22.758B | mid-stable (peak LR 3e-3) | en-de__unigram_starved__step24005_22758M |
en-de-starved-2b |
model-checkpoints/bilingual/en-de/419-languages/2b |
419 languages (unigram_starved) |
1.753B | mid-stable (peak LR 3e-3) | en-de__unigram_starved__step1911_1753M |
en-de-starved |
model-checkpoints/bilingual/en-de/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | en-de__unigram_starved |
en-de-starved-5b |
model-checkpoints/bilingual/en-de/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | en-de__unigram_starved__step5181_4753M |
en-de-fair-10b |
model-checkpoints/bilingual/en-de/5-languages/10b |
5 languages (unigram_destarved) |
9.753B | mid-stable (peak LR 3e-3) | en-de__unigram_destarved__step10631_9753M |
en-de-fair-15b |
model-checkpoints/bilingual/en-de/5-languages/15b |
5 languages (unigram_destarved) |
14.755B | mid-stable (peak LR 3e-3) | en-de__unigram_destarved__step15937_14755M |
en-de-fair-23b |
model-checkpoints/bilingual/en-de/5-languages/23b |
5 languages (unigram_destarved) |
22.757B | mid-stable (peak LR 3e-3) | en-de__unigram_destarved__step24077_22757M |
en-de-fair-2b |
model-checkpoints/bilingual/en-de/5-languages/2b |
5 languages (unigram_destarved) |
1.753B | mid-stable (peak LR 3e-3) | en-de__unigram_destarved__step1911_1753M |
en-de-fair |
model-checkpoints/bilingual/en-de/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | en-de__unigram_destarved |
en-de-fair-5b |
model-checkpoints/bilingual/en-de/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | en-de__unigram_destarved__step5181_4753M |
en-fr-bi-10b |
model-checkpoints/bilingual/en-fr/2-languages/10b |
2 languages (unigram_bi_fr) |
9.76B | mid-stable (peak LR 3e-3) | en-fr__unigram_bi_fr__step9929_9760M |
en-fr-bi-15b |
model-checkpoints/bilingual/en-fr/2-languages/15b |
2 languages (unigram_bi_fr) |
14.764B | mid-stable (peak LR 3e-3) | en-fr__unigram_bi_fr__step15019_14764M |
en-fr-bi-23b |
model-checkpoints/bilingual/en-fr/2-languages/23b |
2 languages (unigram_bi_fr) |
22.77B | mid-stable (peak LR 3e-3) | en-fr__unigram_bi_fr__step23163_22770M |
en-fr-bi-2b |
model-checkpoints/bilingual/en-fr/2-languages/2b |
2 languages (unigram_bi_fr) |
1.754B | mid-stable (peak LR 3e-3) | en-fr__unigram_bi_fr__step1785_1754M |
en-fr-bi |
model-checkpoints/bilingual/en-fr/2-languages/30b-final |
2 languages (unigram_bi_fr) |
30B | cooled final (LR 3e-4) | en-fr__unigram_bi_fr |
en-fr-bi-5b |
model-checkpoints/bilingual/en-fr/2-languages/5b |
2 languages (unigram_bi_fr) |
4.756B | mid-stable (peak LR 3e-3) | en-fr__unigram_bi_fr__step4839_4756M |
en-fr-20lang-10b |
model-checkpoints/bilingual/en-fr/20-languages/10b |
20 languages (unigram_20lang) |
9.76B | mid-stable (peak LR 3e-3) | en-fr__unigram_20lang__step9929_9760M |
en-fr-20lang-15b |
model-checkpoints/bilingual/en-fr/20-languages/15b |
20 languages (unigram_20lang) |
14.764B | mid-stable (peak LR 3e-3) | en-fr__unigram_20lang__step15019_14764M |
en-fr-20lang-23b |
model-checkpoints/bilingual/en-fr/20-languages/23b |
20 languages (unigram_20lang) |
22.766B | mid-stable (peak LR 3e-3) | en-fr__unigram_20lang__step23159_22766M |
en-fr-20lang-2b |
model-checkpoints/bilingual/en-fr/20-languages/2b |
20 languages (unigram_20lang) |
1.754B | mid-stable (peak LR 3e-3) | en-fr__unigram_20lang__step1785_1754M |
en-fr-20lang |
model-checkpoints/bilingual/en-fr/20-languages/30b-final |
20 languages (unigram_20lang) |
30B | cooled final (LR 3e-4) | en-fr__unigram_20lang |
en-fr-20lang-5b |
model-checkpoints/bilingual/en-fr/20-languages/5b |
20 languages (unigram_20lang) |
4.756B | mid-stable (peak LR 3e-3) | en-fr__unigram_20lang__step4839_4756M |
en-fr-starved-10b |
model-checkpoints/bilingual/en-fr/419-languages/10b |
419 languages (unigram_starved) |
9.753B | mid-stable (peak LR 3e-3) | en-fr__unigram_starved__step10631_9753M |
en-fr-starved-15b |
model-checkpoints/bilingual/en-fr/419-languages/15b |
419 languages (unigram_starved) |
14.756B | mid-stable (peak LR 3e-3) | en-fr__unigram_starved__step15865_14756M |
en-fr-starved-23b |
model-checkpoints/bilingual/en-fr/419-languages/23b |
419 languages (unigram_starved) |
22.758B | mid-stable (peak LR 3e-3) | en-fr__unigram_starved__step24005_22758M |
en-fr-starved-2b |
model-checkpoints/bilingual/en-fr/419-languages/2b |
419 languages (unigram_starved) |
1.753B | mid-stable (peak LR 3e-3) | en-fr__unigram_starved__step1911_1753M |
en-fr-starved |
model-checkpoints/bilingual/en-fr/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | en-fr__unigram_starved |
en-fr-starved-5b |
model-checkpoints/bilingual/en-fr/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | en-fr__unigram_starved__step5181_4753M |
en-fr-fair-10b |
model-checkpoints/bilingual/en-fr/5-languages/10b |
5 languages (unigram_destarved) |
9.753B | mid-stable (peak LR 3e-3) | en-fr__unigram_destarved__step10631_9753M |
en-fr-fair-15b |
model-checkpoints/bilingual/en-fr/5-languages/15b |
5 languages (unigram_destarved) |
14.756B | mid-stable (peak LR 3e-3) | en-fr__unigram_destarved__step15865_14756M |
en-fr-fair-23b |
model-checkpoints/bilingual/en-fr/5-languages/23b |
5 languages (unigram_destarved) |
22.758B | mid-stable (peak LR 3e-3) | en-fr__unigram_destarved__step24005_22758M |
en-fr-fair-2b |
model-checkpoints/bilingual/en-fr/5-languages/2b |
5 languages (unigram_destarved) |
1.753B | mid-stable (peak LR 3e-3) | en-fr__unigram_destarved__step1911_1753M |
en-fr-fair |
model-checkpoints/bilingual/en-fr/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | en-fr__unigram_destarved |
en-fr-fair-5b |
model-checkpoints/bilingual/en-fr/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | en-fr__unigram_destarved__step5181_4753M |
en-zh-bi-10b |
model-checkpoints/bilingual/en-zh/2-languages/10b |
2 languages (unigram_bi_zh) |
9.76B | mid-stable (peak LR 3e-3) | en-zh__unigram_bi_zh__step9929_9760M |
en-zh-bi-15b |
model-checkpoints/bilingual/en-zh/2-languages/15b |
2 languages (unigram_bi_zh) |
14.764B | mid-stable (peak LR 3e-3) | en-zh__unigram_bi_zh__step15019_14764M |
en-zh-bi-23b |
model-checkpoints/bilingual/en-zh/2-languages/23b |
2 languages (unigram_bi_zh) |
22.77B | mid-stable (peak LR 3e-3) | en-zh__unigram_bi_zh__step23163_22770M |
en-zh-bi-2b |
model-checkpoints/bilingual/en-zh/2-languages/2b |
2 languages (unigram_bi_zh) |
1.754B | mid-stable (peak LR 3e-3) | en-zh__unigram_bi_zh__step1785_1754M |
en-zh-bi |
model-checkpoints/bilingual/en-zh/2-languages/30b-final |
2 languages (unigram_bi_zh) |
30B | cooled final (LR 3e-4) | en-zh__unigram_bi_zh |
en-zh-bi-5b |
model-checkpoints/bilingual/en-zh/2-languages/5b |
2 languages (unigram_bi_zh) |
4.756B | mid-stable (peak LR 3e-3) | en-zh__unigram_bi_zh__step4839_4756M |
en-zh-20lang-10b |
model-checkpoints/bilingual/en-zh/20-languages/10b |
20 languages (unigram_20lang) |
9.76B | mid-stable (peak LR 3e-3) | en-zh__unigram_20lang__step9929_9760M |
en-zh-20lang-15b |
model-checkpoints/bilingual/en-zh/20-languages/15b |
20 languages (unigram_20lang) |
14.764B | mid-stable (peak LR 3e-3) | en-zh__unigram_20lang__step15019_14764M |
en-zh-20lang-23b |
model-checkpoints/bilingual/en-zh/20-languages/23b |
20 languages (unigram_20lang) |
22.766B | mid-stable (peak LR 3e-3) | en-zh__unigram_20lang__step23159_22766M |
en-zh-20lang-2b |
model-checkpoints/bilingual/en-zh/20-languages/2b |
20 languages (unigram_20lang) |
1.754B | mid-stable (peak LR 3e-3) | en-zh__unigram_20lang__step1785_1754M |
en-zh-20lang |
model-checkpoints/bilingual/en-zh/20-languages/30b-final |
20 languages (unigram_20lang) |
30B | cooled final (LR 3e-4) | en-zh__unigram_20lang |
en-zh-20lang-5b |
model-checkpoints/bilingual/en-zh/20-languages/5b |
20 languages (unigram_20lang) |
4.756B | mid-stable (peak LR 3e-3) | en-zh__unigram_20lang__step4839_4756M |
en-zh-starved-10b |
model-checkpoints/bilingual/en-zh/419-languages/10b |
419 languages (unigram_starved) |
9.753B | mid-stable (peak LR 3e-3) | en-zh__unigram_starved__step10631_9753M |
en-zh-starved-15b |
model-checkpoints/bilingual/en-zh/419-languages/15b |
419 languages (unigram_starved) |
14.756B | mid-stable (peak LR 3e-3) | en-zh__unigram_starved__step15865_14756M |
en-zh-starved-23b |
model-checkpoints/bilingual/en-zh/419-languages/23b |
419 languages (unigram_starved) |
22.758B | mid-stable (peak LR 3e-3) | en-zh__unigram_starved__step24005_22758M |
en-zh-starved-2b |
model-checkpoints/bilingual/en-zh/419-languages/2b |
419 languages (unigram_starved) |
1.753B | mid-stable (peak LR 3e-3) | en-zh__unigram_starved__step1911_1753M |
en-zh-starved |
model-checkpoints/bilingual/en-zh/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | en-zh__unigram_starved |
en-zh-starved-5b |
model-checkpoints/bilingual/en-zh/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | en-zh__unigram_starved__step5181_4753M |
en-zh-fair-10b |
model-checkpoints/bilingual/en-zh/5-languages/10b |
5 languages (unigram_destarved) |
9.753B | mid-stable (peak LR 3e-3) | en-zh__unigram_destarved__step10631_9753M |
en-zh-fair-15b |
model-checkpoints/bilingual/en-zh/5-languages/15b |
5 languages (unigram_destarved) |
14.755B | mid-stable (peak LR 3e-3) | en-zh__unigram_destarved__step15937_14755M |
en-zh-fair-23b |
model-checkpoints/bilingual/en-zh/5-languages/23b |
5 languages (unigram_destarved) |
22.757B | mid-stable (peak LR 3e-3) | en-zh__unigram_destarved__step24077_22757M |
en-zh-fair-2b |
model-checkpoints/bilingual/en-zh/5-languages/2b |
5 languages (unigram_destarved) |
1.753B | mid-stable (peak LR 3e-3) | en-zh__unigram_destarved__step1911_1753M |
en-zh-fair |
model-checkpoints/bilingual/en-zh/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | en-zh__unigram_destarved |
en-zh-fair-5b |
model-checkpoints/bilingual/en-zh/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | en-zh__unigram_destarved__step5181_4753M |
ar-starved-12b |
model-checkpoints/monolingual/ar/419-languages/12b |
419 languages (unigram_starved) |
11.754B | mid-stable (peak LR 3e-3) | ar__unigram_starved__step12811_11754M |
ar-starved-15b |
model-checkpoints/monolingual/ar/419-languages/15b |
419 languages (unigram_starved) |
14.754B | mid-stable (peak LR 3e-3) | ar__unigram_starved__step16081_14754M |
ar-starved-1b |
model-checkpoints/monolingual/ar/419-languages/1b |
419 languages (unigram_starved) |
1.001B | mid-stable (peak LR 3e-3) | ar__unigram_starved__step1092_1001M |
ar-starved-2b |
model-checkpoints/monolingual/ar/419-languages/2b |
419 languages (unigram_starved) |
2.253B | mid-stable (peak LR 3e-3) | ar__unigram_starved__step2456_2253M |
ar-starved |
model-checkpoints/monolingual/ar/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | ar__unigram_starved |
ar-starved-5b |
model-checkpoints/monolingual/ar/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | ar__unigram_starved__step5181_4753M |
ar-starved-8b |
model-checkpoints/monolingual/ar/419-languages/8b |
419 languages (unigram_starved) |
7.753B | mid-stable (peak LR 3e-3) | ar__unigram_starved__step8451_7753M |
ar-fair-12b |
model-checkpoints/monolingual/ar/5-languages/12b |
5 languages (unigram_destarved) |
11.754B | mid-stable (peak LR 3e-3) | ar__unigram_destarved__step12811_11754M |
ar-fair-15b |
model-checkpoints/monolingual/ar/5-languages/15b |
5 languages (unigram_destarved) |
14.754B | mid-stable (peak LR 3e-3) | ar__unigram_destarved__step16081_14754M |
ar-fair-1b |
model-checkpoints/monolingual/ar/5-languages/1b |
5 languages (unigram_destarved) |
1.001B | mid-stable (peak LR 3e-3) | ar__unigram_destarved__step1092_1001M |
ar-fair-2b |
model-checkpoints/monolingual/ar/5-languages/2b |
5 languages (unigram_destarved) |
2.253B | mid-stable (peak LR 3e-3) | ar__unigram_destarved__step2456_2253M |
ar-fair |
model-checkpoints/monolingual/ar/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | ar__unigram_destarved |
ar-fair-5b |
model-checkpoints/monolingual/ar/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | ar__unigram_destarved__step5181_4753M |
ar-fair-8b |
model-checkpoints/monolingual/ar/5-languages/8b |
5 languages (unigram_destarved) |
7.753B | mid-stable (peak LR 3e-3) | ar__unigram_destarved__step8451_7753M |
de-starved-12b |
model-checkpoints/monolingual/de/419-languages/12b |
419 languages (unigram_starved) |
11.754B | mid-stable (peak LR 3e-3) | de__unigram_starved__step12811_11754M |
de-starved-15b |
model-checkpoints/monolingual/de/419-languages/15b |
419 languages (unigram_starved) |
14.754B | mid-stable (peak LR 3e-3) | de__unigram_starved__step16081_14754M |
de-starved-16b |
model-checkpoints/monolingual/de/419-languages/16b |
419 languages (unigram_starved) |
16.1B | mid-stable (peak LR 3e-3) | de__unigram_starved |
de-starved-1b |
model-checkpoints/monolingual/de/419-languages/1b |
419 languages (unigram_starved) |
1.001B | mid-stable (peak LR 3e-3) | de__unigram_starved__step1092_1001M |
de-starved-2b |
model-checkpoints/monolingual/de/419-languages/2b |
419 languages (unigram_starved) |
2.253B | mid-stable (peak LR 3e-3) | de__unigram_starved__step2456_2253M |
de-starved-5b |
model-checkpoints/monolingual/de/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | de__unigram_starved__step5181_4753M |
de-starved-8b |
model-checkpoints/monolingual/de/419-languages/8b |
419 languages (unigram_starved) |
7.753B | mid-stable (peak LR 3e-3) | de__unigram_starved__step8451_7753M |
de-fair-12b |
model-checkpoints/monolingual/de/5-languages/12b |
5 languages (unigram_destarved) |
11.754B | mid-stable (peak LR 3e-3) | de__unigram_destarved__step12811_11754M |
de-fair-15b |
model-checkpoints/monolingual/de/5-languages/15b |
5 languages (unigram_destarved) |
14.755B | mid-stable (peak LR 3e-3) | de__unigram_destarved__step16009_14755M |
de-fair-1b |
model-checkpoints/monolingual/de/5-languages/1b |
5 languages (unigram_destarved) |
1.001B | mid-stable (peak LR 3e-3) | de__unigram_destarved__step1092_1001M |
de-fair-2b |
model-checkpoints/monolingual/de/5-languages/2b |
5 languages (unigram_destarved) |
2.253B | mid-stable (peak LR 3e-3) | de__unigram_destarved__step2456_2253M |
de-fair |
model-checkpoints/monolingual/de/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | de__unigram_destarved |
de-fair-5b |
model-checkpoints/monolingual/de/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | de__unigram_destarved__step5181_4753M |
de-fair-8b |
model-checkpoints/monolingual/de/5-languages/8b |
5 languages (unigram_destarved) |
7.753B | mid-stable (peak LR 3e-3) | de__unigram_destarved__step8451_7753M |
en-starved-12b |
model-checkpoints/monolingual/en/419-languages/12b |
419 languages (unigram_starved) |
11.754B | mid-stable (peak LR 3e-3) | en__unigram_starved__step12739_11754M |
en-starved-15b |
model-checkpoints/monolingual/en/419-languages/15b |
419 languages (unigram_starved) |
14.757B | mid-stable (peak LR 3e-3) | en__unigram_starved__step15793_14757M |
en-starved-1b |
model-checkpoints/monolingual/en/419-languages/1b |
419 languages (unigram_starved) |
1.001B | mid-stable (peak LR 3e-3) | en__unigram_starved__step1092_1001M |
en-starved-2b |
model-checkpoints/monolingual/en/419-languages/2b |
419 languages (unigram_starved) |
2.253B | mid-stable (peak LR 3e-3) | en__unigram_starved__step2456_2253M |
en-starved |
model-checkpoints/monolingual/en/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | en__unigram_starved |
en-starved-5b |
model-checkpoints/monolingual/en/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | en__unigram_starved__step5181_4753M |
en-starved-8b |
model-checkpoints/monolingual/en/419-languages/8b |
419 languages (unigram_starved) |
7.753B | mid-stable (peak LR 3e-3) | en__unigram_starved__step8451_7753M |
en-fair-12b |
model-checkpoints/monolingual/en/5-languages/12b |
5 languages (unigram_destarved) |
11.754B | mid-stable (peak LR 3e-3) | en__unigram_destarved__step12811_11754M |
en-fair-15b |
model-checkpoints/monolingual/en/5-languages/15b |
5 languages (unigram_destarved) |
14.755B | mid-stable (peak LR 3e-3) | en__unigram_destarved__step15937_14755M |
en-fair-1b |
model-checkpoints/monolingual/en/5-languages/1b |
5 languages (unigram_destarved) |
1.001B | mid-stable (peak LR 3e-3) | en__unigram_destarved__step1092_1001M |
en-fair-2b |
model-checkpoints/monolingual/en/5-languages/2b |
5 languages (unigram_destarved) |
2.253B | mid-stable (peak LR 3e-3) | en__unigram_destarved__step2456_2253M |
en-fair |
model-checkpoints/monolingual/en/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | en__unigram_destarved |
en-fair-5b |
model-checkpoints/monolingual/en/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | en__unigram_destarved__step5181_4753M |
en-fair-8b |
model-checkpoints/monolingual/en/5-languages/8b |
5 languages (unigram_destarved) |
7.753B | mid-stable (peak LR 3e-3) | en__unigram_destarved__step8451_7753M |
fr-starved-12b |
model-checkpoints/monolingual/fr/419-languages/12b |
419 languages (unigram_starved) |
11.754B | mid-stable (peak LR 3e-3) | fr__unigram_starved__step12811_11754M |
fr-starved-15b |
model-checkpoints/monolingual/fr/419-languages/15b |
419 languages (unigram_starved) |
14.755B | mid-stable (peak LR 3e-3) | fr__unigram_starved__step16009_14755M |
fr-starved-1b |
model-checkpoints/monolingual/fr/419-languages/1b |
419 languages (unigram_starved) |
1.001B | mid-stable (peak LR 3e-3) | fr__unigram_starved__step1092_1001M |
fr-starved-2b |
model-checkpoints/monolingual/fr/419-languages/2b |
419 languages (unigram_starved) |
2.253B | mid-stable (peak LR 3e-3) | fr__unigram_starved__step2456_2253M |
fr-starved |
model-checkpoints/monolingual/fr/419-languages/30b-final |
419 languages (unigram_starved) |
30B | cooled final (LR 3e-4) | fr__unigram_starved |
fr-starved-5b |
model-checkpoints/monolingual/fr/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | fr__unigram_starved__step5181_4753M |
fr-starved-8b |
model-checkpoints/monolingual/fr/419-languages/8b |
419 languages (unigram_starved) |
7.753B | mid-stable (peak LR 3e-3) | fr__unigram_starved__step8451_7753M |
fr-fair-12b |
model-checkpoints/monolingual/fr/5-languages/12b |
5 languages (unigram_destarved) |
11.754B | mid-stable (peak LR 3e-3) | fr__unigram_destarved__step12811_11754M |
fr-fair-15b |
model-checkpoints/monolingual/fr/5-languages/15b |
5 languages (unigram_destarved) |
14.755B | mid-stable (peak LR 3e-3) | fr__unigram_destarved__step16009_14755M |
fr-fair-1b |
model-checkpoints/monolingual/fr/5-languages/1b |
5 languages (unigram_destarved) |
1.001B | mid-stable (peak LR 3e-3) | fr__unigram_destarved__step1092_1001M |
fr-fair-2b |
model-checkpoints/monolingual/fr/5-languages/2b |
5 languages (unigram_destarved) |
2.253B | mid-stable (peak LR 3e-3) | fr__unigram_destarved__step2456_2253M |
fr-fair |
model-checkpoints/monolingual/fr/5-languages/30b-final |
5 languages (unigram_destarved) |
30B | cooled final (LR 3e-4) | fr__unigram_destarved |
fr-fair-5b |
model-checkpoints/monolingual/fr/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | fr__unigram_destarved__step5181_4753M |
fr-fair-8b |
model-checkpoints/monolingual/fr/5-languages/8b |
5 languages (unigram_destarved) |
7.753B | mid-stable (peak LR 3e-3) | fr__unigram_destarved__step8451_7753M |
zh-starved-12b |
model-checkpoints/monolingual/zh/419-languages/12b |
419 languages (unigram_starved) |
11.754B | mid-stable (peak LR 3e-3) | zh__unigram_starved__step12811_11754M |
zh-starved-15b |
model-checkpoints/monolingual/zh/419-languages/15b |
419 languages (unigram_starved) |
14.756B | mid-stable (peak LR 3e-3) | zh__unigram_starved__step15865_14756M |
zh-starved-1b |
model-checkpoints/monolingual/zh/419-languages/1b |
419 languages (unigram_starved) |
1.001B | mid-stable (peak LR 3e-3) | zh__unigram_starved__step1092_1001M |
zh-starved-2b |
model-checkpoints/monolingual/zh/419-languages/2b |
419 languages (unigram_starved) |
2.253B | mid-stable (peak LR 3e-3) | zh__unigram_starved__step2456_2253M |
zh-starved-5b |
model-checkpoints/monolingual/zh/419-languages/5b |
419 languages (unigram_starved) |
4.753B | mid-stable (peak LR 3e-3) | zh__unigram_starved__step5181_4753M |
zh-starved-8b |
model-checkpoints/monolingual/zh/419-languages/8b |
419 languages (unigram_starved) |
7.753B | mid-stable (peak LR 3e-3) | zh__unigram_starved__step8451_7753M |
zh-fair-12b |
model-checkpoints/monolingual/zh/5-languages/12b |
5 languages (unigram_destarved) |
11.754B | mid-stable (peak LR 3e-3) | zh__unigram_destarved__step12811_11754M |
zh-fair-15b |
model-checkpoints/monolingual/zh/5-languages/15b |
5 languages (unigram_destarved) |
14.756B | mid-stable (peak LR 3e-3) | zh__unigram_destarved__step15865_14756M |
zh-fair-1b |
model-checkpoints/monolingual/zh/5-languages/1b |
5 languages (unigram_destarved) |
1.001B | mid-stable (peak LR 3e-3) | zh__unigram_destarved__step1092_1001M |
zh-fair-2b |
model-checkpoints/monolingual/zh/5-languages/2b |
5 languages (unigram_destarved) |
2.253B | mid-stable (peak LR 3e-3) | zh__unigram_destarved__step2456_2253M |
zh-fair-5b |
model-checkpoints/monolingual/zh/5-languages/5b |
5 languages (unigram_destarved) |
4.753B | mid-stable (peak LR 3e-3) | zh__unigram_destarved__step5181_4753M |
zh-fair-8b |
model-checkpoints/monolingual/zh/5-languages/8b |
5 languages (unigram_destarved) |
7.753B | mid-stable (peak LR 3e-3) | zh__unigram_destarved__step8451_7753M |
Training states
training-states/ holds the full training-state archives (step checkpoints,
last.pt, per-rank optimizer shards) of the 20- and 2-language bilingual runs
(en-X__unigram_20lang, en-X__unigram_bi_X) and of the abandoned 100B-token
extensions (en-{de,ar}__unigram_{destarved,starved}__100b,
en-ar__*__100b_scratch). They are only needed to resume or re-analyse
training, not for evaluation.
Older material
src/xscript/, run_benchmarks.py and requirements.txt at the top level are
an early export of the code and predate the split-checkpoint layout; use the
GitHub repo instead.
Original runbook (outdated)
The README this repository shipped with is kept below; its heading levels are
shifted down one, nothing else is changed. It describes the first 15-checkpoint
export (whole final.pt files, three benchmarks) and is out of date on the
layout and the model count.
External benchmark evaluation
Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained
checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer
(src/xscript/model.py) + SentencePiece tokenizer — pure PyTorch, using
F.scaled_dot_product_attention, no flash-attn / triton / custom kernels — so
they run on any stock GPU (or CPU, slowly). Each model is ~1B params (fits any
16GB GPU).
The benchmark harness (src/xscript/eval/bench.py) wraps our model into
lm-evaluation-harness and scores Global-MMLU, Belebele, and XNLI on each run's
training languages. It is the same harness we would have run on-cluster, so
numbers are directly comparable.
1. Export from Isambard (already done by upload_to_hf.py)
The private HF repo mirrors the on-cluster layout:
src/xscript/** # bundled model + harness code
tokenizers/unigram_{starved,destarved}/{sp.model,meta.json}
runs/<name>/checkpoints/final.pt # 15 checkpoints, fp32, ~4GB each
models.json # friendly name -> tokenizer + langs + orig run
run_benchmarks.py requirements.txt README.md
Models use friendly names <mixture>-<starved|fair> (e.g. en-fair,
en-ar-starved). models.json maps each to its real tokenizer.
2. Run on your GPU
## clone just the runner (or download run_benchmarks.py + requirements.txt from the repo)
pip install torch --index-url https://download.pytorch.org/whl/cu121 # match your CUDA
pip install -r requirements.txt
export HF_TOKEN=hf_... # while the repo is private
## quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST
python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200
## full suite once the quick pass looks sane
python run_benchmarks.py --repo jvonrad/xscript-eval
The runner downloads one checkpoint at a time and deletes it after eval
(--keep-checkpoints to retain), so peak disk is ~5GB. Results:
xscript_bench/results/bench/<run>_final.json # per-run task accuracies
xscript_bench/results/summary.json # everything combined
Send those JSONs back for analysis.
Notes
--runs en-starved en-fairlimits to a subset (friendly names).--tasks xnli_en xnli_deoverrides the task list (default = the run's langs).- Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs).
- Scores are ordinary accuracy (
acc,none); raw harness output is preserved in each per-run JSON for length-normalized variants.