|
Download README.md from jvonrad/alignment-without-transfer: direct link, hf CLI and curl.
- Browser
- Download file 63.5 kB
-
https://huggingface.co/jvonrad/alignment-without-transfer/resolve/main/README.md
- Command line
-
hf download hf://jvonrad/alignment-without-transfer/README.md
-
curl -L -o README.md https://huggingface.co/jvonrad/alignment-without-transfer/resolve/main/README.md
63.5 kB
| language: | |
| - en | |
| - de | |
| - fr | |
| - ar | |
| - zh | |
| tags: | |
| - multilingual | |
| - pretraining | |
| - tokenization | |
| - cross-lingual-transfer | |
| - xscript-pretraining | |
| # Alignment without Transfer — model checkpoints | |
| Checkpoints for the paper **Alignment without Transfer: Tokenization Impacts Cross-lingual Representations during Multilingual Language Model Pretraining** (preprint forthcoming; not yet on arXiv). | |
| About 1B-parameter Llama-style decoders (16 layers, d=2048, SwiGLU, RoPE, untied | |
| embeddings, 65,536-piece vocabulary) trained from scratch on one or two of | |
| English, German, French, Arabic and Chinese (FineWeb-HQ / FineWeb2-HQ). Model | |
| pairs differ **only in the tokenizer**, i.e. in how many languages share the | |
| 65k vocabulary: 419, 20, 5 or 2. | |
| **164 checkpoints**: 96 bilingual (English + one partner language) and | |
| 68 monolingual, each at several token budgets. | |
| Everything else for the paper is in the | |
| [Hugging Face collection](https://huggingface.co/collections/jvonrad/alignment-without-transfer-6afdbe0f532d2dd94353ca31) and on GitHub: | |
| | | | | |
| |---|---| | |
| | code, experiment log | [github.com/jvonrad/XScript-Pretraining](https://github.com/jvonrad/XScript-Pretraining) | | |
| | cached alignment embeddings | [`jvonrad/alignment-without-transfer-embeddings`](https://huggingface.co/datasets/jvonrad/alignment-without-transfer-embeddings) | | |
| | data provenance (pool file lists, holdout text) | [`jvonrad/alignment-without-transfer-data`](https://huggingface.co/datasets/jvonrad/alignment-without-transfer-data) | | |
| ## Layout | |
| ``` | |
| model-checkpoints/ | |
| bilingual/<en-X>/<N>-languages/<budget>/ X in de, fr, ar, zh | |
| monolingual/<lang>/<N>-languages/<budget>/ lang in en, de, fr, ar, zh | |
| final.pt.part000 ... part004 one torch.save file split in 5 parts | |
| n_parts.txt number of parts | |
| tokenizers/<tokenizer>/{sp.model, meta.json} | |
| models.json model name -> path, tokenizer, languages, budget, LR state | |
| training-states/ optimizer/resume states of some runs (not needed for evaluation) | |
| ``` | |
| * **`<N>-languages`** — how many languages the tokenizer was trained on: | |
| | folder | tokenizer | trained on | | |
| |---|---|---| | |
| | `2-languages` | `unigram_bi_<X>` | English + the partner language only | | |
| | `5-languages` | `unigram_destarved` | the 5 study languages (byte-premium adjusted) | | |
| | `20-languages` | `unigram_20lang` | the 5 study languages + ces, fas, hun, ind, ita, jpn, kor, nld, pol, por, ron, rus, swe, ukr, vie | | |
| | `419-languages` | `unigram_starved` | ~419 FineWeb/FineWeb2 languages (T=100) | | |
| All are SentencePiece Unigram, 65,536 pieces, byte fallback, ids 0–3 = | |
| `<unk>/<bos>/<eos>/<pad>`. | |
| * **`<budget>`** — total training tokens (for bilinguals half per language): | |
| `2b` … `23b` are **mid-stable** snapshots at peak learning rate; | |
| `30b-final` is the **cooled** final model. | |
| ### ⚠️ Compare like with like | |
| All runs share one warmup–stable–decay schedule: warmup to 1B tokens, peak LR | |
| 3.0e-3 until 24B, decay to 3.0e-4 by 30B. A `30b-final` checkpoint has been | |
| through the decay; every other budget has not. Pairing the two hands the cooled | |
| model the whole decay phase for free. Exceptions: `monolingual/de/419-languages/16b` | |
| (a re-seeded retrain that stops at 16.1B, not cooled) is the largest German | |
| 419-language model, and Chinese monolinguals exist only up to `15b`. | |
| ### Model names | |
| `models.json` keeps the short names the code and result files use: | |
| `<langs>-<tok>[-<budget>]`, with `<tok>` = `starved` (419 languages), `20lang` | |
| (20), `fair` (5) or `bi` (2), and no budget suffix for the cooled final, e.g. | |
| `en-ar-fair-12b` → `model-checkpoints/bilingual/en-ar/5-languages/12b`. | |
| ## Loading a checkpoint | |
| ```python | |
| import json, shutil | |
| from huggingface_hub import hf_hub_download | |
| repo, name = "jvonrad/alignment-without-transfer", "en-ar-fair" # any key of models.json | |
| spec = json.load(open(hf_hub_download(repo, "models.json")))[name] | |
| d = spec["path"] # model-checkpoints/bilingual/en-ar/5-languages/30b-final | |
| n = int(open(hf_hub_download(repo, f"{d}/n_parts.txt")).read()) | |
| with open("final.pt", "wb") as out: | |
| for i in range(n): | |
| with open(hf_hub_download(repo, f"{d}/final.pt.part{i:03d}"), "rb") as part: | |
| shutil.copyfileobj(part, out) | |
| sp_model = hf_hub_download(repo, f"tokenizers/{spec['tok']}/sp.model") | |
| # with the GitHub repo's src/ on sys.path: | |
| import torch | |
| from xscript.model import ModelConfig, Transformer | |
| ck = torch.load("final.pt", map_location="cpu", weights_only=False) | |
| model = Transformer(ModelConfig(**ck["cfg"]["model"])) | |
| model.load_state_dict(ck["model"]) | |
| ``` | |
| The GitHub repo's evaluation scripts do this automatically | |
| (`scripts/external_bench/hf_layout.py` maps model names to these paths). | |
| ## Checkpoint index | |
| Each cell links to the checkpoint folder. | |
| ### Bilingual | |
| #### en-de | |
| | tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final | | |
| |---|---|---|---|---|---|---| | |
| | 2 languages | [`en-de-bi-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/2-languages/2b) | [`en-de-bi-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/2-languages/5b) | [`en-de-bi-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/2-languages/10b) | [`en-de-bi-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/2-languages/15b) | [`en-de-bi-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/2-languages/23b) | [`en-de-bi`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/2-languages/30b-final) | | |
| | 5 languages | [`en-de-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/5-languages/2b) | [`en-de-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/5-languages/5b) | [`en-de-fair-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/5-languages/10b) | [`en-de-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/5-languages/15b) | [`en-de-fair-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/5-languages/23b) | [`en-de-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/5-languages/30b-final) | | |
| | 20 languages | [`en-de-20lang-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/20-languages/2b) | [`en-de-20lang-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/20-languages/5b) | [`en-de-20lang-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/20-languages/10b) | [`en-de-20lang-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/20-languages/15b) | [`en-de-20lang-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/20-languages/23b) | [`en-de-20lang`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/20-languages/30b-final) | | |
| | 419 languages | [`en-de-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/419-languages/2b) | [`en-de-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/419-languages/5b) | [`en-de-starved-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/419-languages/10b) | [`en-de-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/419-languages/15b) | [`en-de-starved-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/419-languages/23b) | [`en-de-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-de/419-languages/30b-final) | | |
| #### en-fr | |
| | tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final | | |
| |---|---|---|---|---|---|---| | |
| | 2 languages | [`en-fr-bi-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/2-languages/2b) | [`en-fr-bi-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/2-languages/5b) | [`en-fr-bi-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/2-languages/10b) | [`en-fr-bi-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/2-languages/15b) | [`en-fr-bi-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/2-languages/23b) | [`en-fr-bi`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/2-languages/30b-final) | | |
| | 5 languages | [`en-fr-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/5-languages/2b) | [`en-fr-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/5-languages/5b) | [`en-fr-fair-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/5-languages/10b) | [`en-fr-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/5-languages/15b) | [`en-fr-fair-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/5-languages/23b) | [`en-fr-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/5-languages/30b-final) | | |
| | 20 languages | [`en-fr-20lang-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/20-languages/2b) | [`en-fr-20lang-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/20-languages/5b) | [`en-fr-20lang-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/20-languages/10b) | [`en-fr-20lang-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/20-languages/15b) | [`en-fr-20lang-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/20-languages/23b) | [`en-fr-20lang`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/20-languages/30b-final) | | |
| | 419 languages | [`en-fr-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/419-languages/2b) | [`en-fr-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/419-languages/5b) | [`en-fr-starved-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/419-languages/10b) | [`en-fr-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/419-languages/15b) | [`en-fr-starved-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/419-languages/23b) | [`en-fr-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-fr/419-languages/30b-final) | | |
| #### en-ar | |
| | tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final | | |
| |---|---|---|---|---|---|---| | |
| | 2 languages | [`en-ar-bi-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/2-languages/2b) | [`en-ar-bi-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/2-languages/5b) | [`en-ar-bi-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/2-languages/10b) | [`en-ar-bi-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/2-languages/15b) | [`en-ar-bi-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/2-languages/23b) | [`en-ar-bi`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/2-languages/30b-final) | | |
| | 5 languages | [`en-ar-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/5-languages/2b) | [`en-ar-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/5-languages/5b) | [`en-ar-fair-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/5-languages/10b) | [`en-ar-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/5-languages/15b) | [`en-ar-fair-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/5-languages/23b) | [`en-ar-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/5-languages/30b-final) | | |
| | 20 languages | [`en-ar-20lang-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/20-languages/2b) | [`en-ar-20lang-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/20-languages/5b) | [`en-ar-20lang-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/20-languages/10b) | [`en-ar-20lang-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/20-languages/15b) | [`en-ar-20lang-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/20-languages/23b) | [`en-ar-20lang`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/20-languages/30b-final) | | |
| | 419 languages | [`en-ar-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/419-languages/2b) | [`en-ar-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/419-languages/5b) | [`en-ar-starved-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/419-languages/10b) | [`en-ar-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/419-languages/15b) | [`en-ar-starved-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/419-languages/23b) | [`en-ar-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-ar/419-languages/30b-final) | | |
| #### en-zh | |
| | tokenizer | 2b | 5b | 10b | 15b | 23b | 30b-final | | |
| |---|---|---|---|---|---|---| | |
| | 2 languages | [`en-zh-bi-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/2-languages/2b) | [`en-zh-bi-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/2-languages/5b) | [`en-zh-bi-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/2-languages/10b) | [`en-zh-bi-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/2-languages/15b) | [`en-zh-bi-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/2-languages/23b) | [`en-zh-bi`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/2-languages/30b-final) | | |
| | 5 languages | [`en-zh-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/5-languages/2b) | [`en-zh-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/5-languages/5b) | [`en-zh-fair-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/5-languages/10b) | [`en-zh-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/5-languages/15b) | [`en-zh-fair-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/5-languages/23b) | [`en-zh-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/5-languages/30b-final) | | |
| | 20 languages | [`en-zh-20lang-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/20-languages/2b) | [`en-zh-20lang-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/20-languages/5b) | [`en-zh-20lang-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/20-languages/10b) | [`en-zh-20lang-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/20-languages/15b) | [`en-zh-20lang-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/20-languages/23b) | [`en-zh-20lang`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/20-languages/30b-final) | | |
| | 419 languages | [`en-zh-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/419-languages/2b) | [`en-zh-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/419-languages/5b) | [`en-zh-starved-10b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/419-languages/10b) | [`en-zh-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/419-languages/15b) | [`en-zh-starved-23b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/419-languages/23b) | [`en-zh-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/bilingual/en-zh/419-languages/30b-final) | | |
| ### Monolingual | |
| #### en | |
| | tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 30b-final | | |
| |---|---|---|---|---|---|---|---| | |
| | 5 languages | [`en-fair-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/1b) | [`en-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/2b) | [`en-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/5b) | [`en-fair-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/8b) | [`en-fair-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/12b) | [`en-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/15b) | [`en-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/5-languages/30b-final) | | |
| | 419 languages | [`en-starved-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/1b) | [`en-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/2b) | [`en-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/5b) | [`en-starved-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/8b) | [`en-starved-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/12b) | [`en-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/15b) | [`en-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/en/419-languages/30b-final) | | |
| #### de | |
| | tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 16b | 30b-final | | |
| |---|---|---|---|---|---|---|---|---| | |
| | 5 languages | [`de-fair-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/1b) | [`de-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/2b) | [`de-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/5b) | [`de-fair-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/8b) | [`de-fair-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/12b) | [`de-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/15b) | | [`de-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/5-languages/30b-final) | | |
| | 419 languages | [`de-starved-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/1b) | [`de-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/2b) | [`de-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/5b) | [`de-starved-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/8b) | [`de-starved-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/12b) | [`de-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/15b) | [`de-starved-16b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/de/419-languages/16b) | | | |
| #### fr | |
| | tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 30b-final | | |
| |---|---|---|---|---|---|---|---| | |
| | 5 languages | [`fr-fair-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/1b) | [`fr-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/2b) | [`fr-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/5b) | [`fr-fair-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/8b) | [`fr-fair-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/12b) | [`fr-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/15b) | [`fr-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/5-languages/30b-final) | | |
| | 419 languages | [`fr-starved-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/1b) | [`fr-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/2b) | [`fr-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/5b) | [`fr-starved-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/8b) | [`fr-starved-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/12b) | [`fr-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/15b) | [`fr-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/fr/419-languages/30b-final) | | |
| #### ar | |
| | tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | 30b-final | | |
| |---|---|---|---|---|---|---|---| | |
| | 5 languages | [`ar-fair-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/1b) | [`ar-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/2b) | [`ar-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/5b) | [`ar-fair-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/8b) | [`ar-fair-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/12b) | [`ar-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/15b) | [`ar-fair`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/5-languages/30b-final) | | |
| | 419 languages | [`ar-starved-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/1b) | [`ar-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/2b) | [`ar-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/5b) | [`ar-starved-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/8b) | [`ar-starved-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/12b) | [`ar-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/15b) | [`ar-starved`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/ar/419-languages/30b-final) | | |
| #### zh | |
| | tokenizer | 1b | 2b | 5b | 8b | 12b | 15b | | |
| |---|---|---|---|---|---|---| | |
| | 5 languages | [`zh-fair-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/5-languages/1b) | [`zh-fair-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/5-languages/2b) | [`zh-fair-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/5-languages/5b) | [`zh-fair-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/5-languages/8b) | [`zh-fair-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/5-languages/12b) | [`zh-fair-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/5-languages/15b) | | |
| | 419 languages | [`zh-starved-1b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/419-languages/1b) | [`zh-starved-2b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/419-languages/2b) | [`zh-starved-5b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/419-languages/5b) | [`zh-starved-8b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/419-languages/8b) | [`zh-starved-12b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/419-languages/12b) | [`zh-starved-15b`](https://huggingface.co/jvonrad/alignment-without-transfer/tree/main/model-checkpoints/monolingual/zh/419-languages/15b) | | |
| <details><summary>Full table (path, exact token count, LR state, original run name)</summary> | |
| | model | path | tokenizer | tokens | LR state | original run | | |
| |---|---|---|---|---|---| | |
| | `en-ar-bi-10b` | `model-checkpoints/bilingual/en-ar/2-languages/10b` | 2 languages (`unigram_bi_ar`) | 9.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step9929_9760M` | | |
| | `en-ar-bi-15b` | `model-checkpoints/bilingual/en-ar/2-languages/15b` | 2 languages (`unigram_bi_ar`) | 14.764B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step15019_14764M` | | |
| | `en-ar-bi-23b` | `model-checkpoints/bilingual/en-ar/2-languages/23b` | 2 languages (`unigram_bi_ar`) | 22.77B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step23163_22770M` | | |
| | `en-ar-bi-2b` | `model-checkpoints/bilingual/en-ar/2-languages/2b` | 2 languages (`unigram_bi_ar`) | 1.754B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step1785_1754M` | | |
| | `en-ar-bi` | `model-checkpoints/bilingual/en-ar/2-languages/30b-final` | 2 languages (`unigram_bi_ar`) | 30B | cooled final (LR 3e-4) | `en-ar__unigram_bi_ar` | | |
| | `en-ar-bi-5b` | `model-checkpoints/bilingual/en-ar/2-languages/5b` | 2 languages (`unigram_bi_ar`) | 4.756B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step4839_4756M` | | |
| | `en-ar-20lang-10b` | `model-checkpoints/bilingual/en-ar/20-languages/10b` | 20 languages (`unigram_20lang`) | 9.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step9929_9760M` | | |
| | `en-ar-20lang-15b` | `model-checkpoints/bilingual/en-ar/20-languages/15b` | 20 languages (`unigram_20lang`) | 14.764B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step15019_14764M` | | |
| | `en-ar-20lang-23b` | `model-checkpoints/bilingual/en-ar/20-languages/23b` | 20 languages (`unigram_20lang`) | 22.766B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step23159_22766M` | | |
| | `en-ar-20lang-2b` | `model-checkpoints/bilingual/en-ar/20-languages/2b` | 20 languages (`unigram_20lang`) | 1.754B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step1785_1754M` | | |
| | `en-ar-20lang` | `model-checkpoints/bilingual/en-ar/20-languages/30b-final` | 20 languages (`unigram_20lang`) | 30B | cooled final (LR 3e-4) | `en-ar__unigram_20lang` | | |
| | `en-ar-20lang-5b` | `model-checkpoints/bilingual/en-ar/20-languages/5b` | 20 languages (`unigram_20lang`) | 4.756B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step4839_4756M` | | |
| | `en-ar-starved-10b` | `model-checkpoints/bilingual/en-ar/419-languages/10b` | 419 languages (`unigram_starved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step10631_9753M` | | |
| | `en-ar-starved-15b` | `model-checkpoints/bilingual/en-ar/419-languages/15b` | 419 languages (`unigram_starved`) | 14.757B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step15793_14757M` | | |
| | `en-ar-starved-23b` | `model-checkpoints/bilingual/en-ar/419-languages/23b` | 419 languages (`unigram_starved`) | 22.758B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step23933_22758M` | | |
| | `en-ar-starved-2b` | `model-checkpoints/bilingual/en-ar/419-languages/2b` | 419 languages (`unigram_starved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step1911_1753M` | | |
| | `en-ar-starved` | `model-checkpoints/bilingual/en-ar/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `en-ar__unigram_starved` | | |
| | `en-ar-starved-5b` | `model-checkpoints/bilingual/en-ar/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step5181_4753M` | | |
| | `en-ar-fair-10b` | `model-checkpoints/bilingual/en-ar/5-languages/10b` | 5 languages (`unigram_destarved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step10631_9753M` | | |
| | `en-ar-fair-15b` | `model-checkpoints/bilingual/en-ar/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.755B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step15937_14755M` | | |
| | `en-ar-fair-23b` | `model-checkpoints/bilingual/en-ar/5-languages/23b` | 5 languages (`unigram_destarved`) | 22.757B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step24077_22757M` | | |
| | `en-ar-fair-2b` | `model-checkpoints/bilingual/en-ar/5-languages/2b` | 5 languages (`unigram_destarved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step1911_1753M` | | |
| | `en-ar-fair` | `model-checkpoints/bilingual/en-ar/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `en-ar__unigram_destarved` | | |
| | `en-ar-fair-5b` | `model-checkpoints/bilingual/en-ar/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step5181_4753M` | | |
| | `en-de-bi-10b` | `model-checkpoints/bilingual/en-de/2-languages/10b` | 2 languages (`unigram_bi_de`) | 9.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step9929_9760M` | | |
| | `en-de-bi-15b` | `model-checkpoints/bilingual/en-de/2-languages/15b` | 2 languages (`unigram_bi_de`) | 14.764B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step15019_14764M` | | |
| | `en-de-bi-23b` | `model-checkpoints/bilingual/en-de/2-languages/23b` | 2 languages (`unigram_bi_de`) | 22.77B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step23163_22770M` | | |
| | `en-de-bi-2b` | `model-checkpoints/bilingual/en-de/2-languages/2b` | 2 languages (`unigram_bi_de`) | 1.754B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step1785_1754M` | | |
| | `en-de-bi` | `model-checkpoints/bilingual/en-de/2-languages/30b-final` | 2 languages (`unigram_bi_de`) | 30B | cooled final (LR 3e-4) | `en-de__unigram_bi_de` | | |
| | `en-de-bi-5b` | `model-checkpoints/bilingual/en-de/2-languages/5b` | 2 languages (`unigram_bi_de`) | 4.756B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step4839_4756M` | | |
| | `en-de-20lang-10b` | `model-checkpoints/bilingual/en-de/20-languages/10b` | 20 languages (`unigram_20lang`) | 9.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step9929_9760M` | | |
| | `en-de-20lang-15b` | `model-checkpoints/bilingual/en-de/20-languages/15b` | 20 languages (`unigram_20lang`) | 14.764B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step15019_14764M` | | |
| | `en-de-20lang-23b` | `model-checkpoints/bilingual/en-de/20-languages/23b` | 20 languages (`unigram_20lang`) | 22.766B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step23159_22766M` | | |
| | `en-de-20lang-2b` | `model-checkpoints/bilingual/en-de/20-languages/2b` | 20 languages (`unigram_20lang`) | 1.754B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step1785_1754M` | | |
| | `en-de-20lang` | `model-checkpoints/bilingual/en-de/20-languages/30b-final` | 20 languages (`unigram_20lang`) | 30B | cooled final (LR 3e-4) | `en-de__unigram_20lang` | | |
| | `en-de-20lang-5b` | `model-checkpoints/bilingual/en-de/20-languages/5b` | 20 languages (`unigram_20lang`) | 4.756B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step4839_4756M` | | |
| | `en-de-starved-10b` | `model-checkpoints/bilingual/en-de/419-languages/10b` | 419 languages (`unigram_starved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step10631_9753M` | | |
| | `en-de-starved-15b` | `model-checkpoints/bilingual/en-de/419-languages/15b` | 419 languages (`unigram_starved`) | 14.756B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step15865_14756M` | | |
| | `en-de-starved-23b` | `model-checkpoints/bilingual/en-de/419-languages/23b` | 419 languages (`unigram_starved`) | 22.758B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step24005_22758M` | | |
| | `en-de-starved-2b` | `model-checkpoints/bilingual/en-de/419-languages/2b` | 419 languages (`unigram_starved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step1911_1753M` | | |
| | `en-de-starved` | `model-checkpoints/bilingual/en-de/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `en-de__unigram_starved` | | |
| | `en-de-starved-5b` | `model-checkpoints/bilingual/en-de/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step5181_4753M` | | |
| | `en-de-fair-10b` | `model-checkpoints/bilingual/en-de/5-languages/10b` | 5 languages (`unigram_destarved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step10631_9753M` | | |
| | `en-de-fair-15b` | `model-checkpoints/bilingual/en-de/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.755B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step15937_14755M` | | |
| | `en-de-fair-23b` | `model-checkpoints/bilingual/en-de/5-languages/23b` | 5 languages (`unigram_destarved`) | 22.757B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step24077_22757M` | | |
| | `en-de-fair-2b` | `model-checkpoints/bilingual/en-de/5-languages/2b` | 5 languages (`unigram_destarved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step1911_1753M` | | |
| | `en-de-fair` | `model-checkpoints/bilingual/en-de/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `en-de__unigram_destarved` | | |
| | `en-de-fair-5b` | `model-checkpoints/bilingual/en-de/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step5181_4753M` | | |
| | `en-fr-bi-10b` | `model-checkpoints/bilingual/en-fr/2-languages/10b` | 2 languages (`unigram_bi_fr`) | 9.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step9929_9760M` | | |
| | `en-fr-bi-15b` | `model-checkpoints/bilingual/en-fr/2-languages/15b` | 2 languages (`unigram_bi_fr`) | 14.764B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step15019_14764M` | | |
| | `en-fr-bi-23b` | `model-checkpoints/bilingual/en-fr/2-languages/23b` | 2 languages (`unigram_bi_fr`) | 22.77B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step23163_22770M` | | |
| | `en-fr-bi-2b` | `model-checkpoints/bilingual/en-fr/2-languages/2b` | 2 languages (`unigram_bi_fr`) | 1.754B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step1785_1754M` | | |
| | `en-fr-bi` | `model-checkpoints/bilingual/en-fr/2-languages/30b-final` | 2 languages (`unigram_bi_fr`) | 30B | cooled final (LR 3e-4) | `en-fr__unigram_bi_fr` | | |
| | `en-fr-bi-5b` | `model-checkpoints/bilingual/en-fr/2-languages/5b` | 2 languages (`unigram_bi_fr`) | 4.756B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step4839_4756M` | | |
| | `en-fr-20lang-10b` | `model-checkpoints/bilingual/en-fr/20-languages/10b` | 20 languages (`unigram_20lang`) | 9.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step9929_9760M` | | |
| | `en-fr-20lang-15b` | `model-checkpoints/bilingual/en-fr/20-languages/15b` | 20 languages (`unigram_20lang`) | 14.764B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step15019_14764M` | | |
| | `en-fr-20lang-23b` | `model-checkpoints/bilingual/en-fr/20-languages/23b` | 20 languages (`unigram_20lang`) | 22.766B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step23159_22766M` | | |
| | `en-fr-20lang-2b` | `model-checkpoints/bilingual/en-fr/20-languages/2b` | 20 languages (`unigram_20lang`) | 1.754B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step1785_1754M` | | |
| | `en-fr-20lang` | `model-checkpoints/bilingual/en-fr/20-languages/30b-final` | 20 languages (`unigram_20lang`) | 30B | cooled final (LR 3e-4) | `en-fr__unigram_20lang` | | |
| | `en-fr-20lang-5b` | `model-checkpoints/bilingual/en-fr/20-languages/5b` | 20 languages (`unigram_20lang`) | 4.756B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step4839_4756M` | | |
| | `en-fr-starved-10b` | `model-checkpoints/bilingual/en-fr/419-languages/10b` | 419 languages (`unigram_starved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step10631_9753M` | | |
| | `en-fr-starved-15b` | `model-checkpoints/bilingual/en-fr/419-languages/15b` | 419 languages (`unigram_starved`) | 14.756B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step15865_14756M` | | |
| | `en-fr-starved-23b` | `model-checkpoints/bilingual/en-fr/419-languages/23b` | 419 languages (`unigram_starved`) | 22.758B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step24005_22758M` | | |
| | `en-fr-starved-2b` | `model-checkpoints/bilingual/en-fr/419-languages/2b` | 419 languages (`unigram_starved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step1911_1753M` | | |
| | `en-fr-starved` | `model-checkpoints/bilingual/en-fr/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `en-fr__unigram_starved` | | |
| | `en-fr-starved-5b` | `model-checkpoints/bilingual/en-fr/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step5181_4753M` | | |
| | `en-fr-fair-10b` | `model-checkpoints/bilingual/en-fr/5-languages/10b` | 5 languages (`unigram_destarved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step10631_9753M` | | |
| | `en-fr-fair-15b` | `model-checkpoints/bilingual/en-fr/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.756B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step15865_14756M` | | |
| | `en-fr-fair-23b` | `model-checkpoints/bilingual/en-fr/5-languages/23b` | 5 languages (`unigram_destarved`) | 22.758B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step24005_22758M` | | |
| | `en-fr-fair-2b` | `model-checkpoints/bilingual/en-fr/5-languages/2b` | 5 languages (`unigram_destarved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step1911_1753M` | | |
| | `en-fr-fair` | `model-checkpoints/bilingual/en-fr/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `en-fr__unigram_destarved` | | |
| | `en-fr-fair-5b` | `model-checkpoints/bilingual/en-fr/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step5181_4753M` | | |
| | `en-zh-bi-10b` | `model-checkpoints/bilingual/en-zh/2-languages/10b` | 2 languages (`unigram_bi_zh`) | 9.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step9929_9760M` | | |
| | `en-zh-bi-15b` | `model-checkpoints/bilingual/en-zh/2-languages/15b` | 2 languages (`unigram_bi_zh`) | 14.764B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step15019_14764M` | | |
| | `en-zh-bi-23b` | `model-checkpoints/bilingual/en-zh/2-languages/23b` | 2 languages (`unigram_bi_zh`) | 22.77B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step23163_22770M` | | |
| | `en-zh-bi-2b` | `model-checkpoints/bilingual/en-zh/2-languages/2b` | 2 languages (`unigram_bi_zh`) | 1.754B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step1785_1754M` | | |
| | `en-zh-bi` | `model-checkpoints/bilingual/en-zh/2-languages/30b-final` | 2 languages (`unigram_bi_zh`) | 30B | cooled final (LR 3e-4) | `en-zh__unigram_bi_zh` | | |
| | `en-zh-bi-5b` | `model-checkpoints/bilingual/en-zh/2-languages/5b` | 2 languages (`unigram_bi_zh`) | 4.756B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step4839_4756M` | | |
| | `en-zh-20lang-10b` | `model-checkpoints/bilingual/en-zh/20-languages/10b` | 20 languages (`unigram_20lang`) | 9.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step9929_9760M` | | |
| | `en-zh-20lang-15b` | `model-checkpoints/bilingual/en-zh/20-languages/15b` | 20 languages (`unigram_20lang`) | 14.764B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step15019_14764M` | | |
| | `en-zh-20lang-23b` | `model-checkpoints/bilingual/en-zh/20-languages/23b` | 20 languages (`unigram_20lang`) | 22.766B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step23159_22766M` | | |
| | `en-zh-20lang-2b` | `model-checkpoints/bilingual/en-zh/20-languages/2b` | 20 languages (`unigram_20lang`) | 1.754B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step1785_1754M` | | |
| | `en-zh-20lang` | `model-checkpoints/bilingual/en-zh/20-languages/30b-final` | 20 languages (`unigram_20lang`) | 30B | cooled final (LR 3e-4) | `en-zh__unigram_20lang` | | |
| | `en-zh-20lang-5b` | `model-checkpoints/bilingual/en-zh/20-languages/5b` | 20 languages (`unigram_20lang`) | 4.756B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step4839_4756M` | | |
| | `en-zh-starved-10b` | `model-checkpoints/bilingual/en-zh/419-languages/10b` | 419 languages (`unigram_starved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step10631_9753M` | | |
| | `en-zh-starved-15b` | `model-checkpoints/bilingual/en-zh/419-languages/15b` | 419 languages (`unigram_starved`) | 14.756B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step15865_14756M` | | |
| | `en-zh-starved-23b` | `model-checkpoints/bilingual/en-zh/419-languages/23b` | 419 languages (`unigram_starved`) | 22.758B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step24005_22758M` | | |
| | `en-zh-starved-2b` | `model-checkpoints/bilingual/en-zh/419-languages/2b` | 419 languages (`unigram_starved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step1911_1753M` | | |
| | `en-zh-starved` | `model-checkpoints/bilingual/en-zh/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `en-zh__unigram_starved` | | |
| | `en-zh-starved-5b` | `model-checkpoints/bilingual/en-zh/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step5181_4753M` | | |
| | `en-zh-fair-10b` | `model-checkpoints/bilingual/en-zh/5-languages/10b` | 5 languages (`unigram_destarved`) | 9.753B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step10631_9753M` | | |
| | `en-zh-fair-15b` | `model-checkpoints/bilingual/en-zh/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.755B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step15937_14755M` | | |
| | `en-zh-fair-23b` | `model-checkpoints/bilingual/en-zh/5-languages/23b` | 5 languages (`unigram_destarved`) | 22.757B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step24077_22757M` | | |
| | `en-zh-fair-2b` | `model-checkpoints/bilingual/en-zh/5-languages/2b` | 5 languages (`unigram_destarved`) | 1.753B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step1911_1753M` | | |
| | `en-zh-fair` | `model-checkpoints/bilingual/en-zh/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `en-zh__unigram_destarved` | | |
| | `en-zh-fair-5b` | `model-checkpoints/bilingual/en-zh/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step5181_4753M` | | |
| | `ar-starved-12b` | `model-checkpoints/monolingual/ar/419-languages/12b` | 419 languages (`unigram_starved`) | 11.754B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step12811_11754M` | | |
| | `ar-starved-15b` | `model-checkpoints/monolingual/ar/419-languages/15b` | 419 languages (`unigram_starved`) | 14.754B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step16081_14754M` | | |
| | `ar-starved-1b` | `model-checkpoints/monolingual/ar/419-languages/1b` | 419 languages (`unigram_starved`) | 1.001B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step1092_1001M` | | |
| | `ar-starved-2b` | `model-checkpoints/monolingual/ar/419-languages/2b` | 419 languages (`unigram_starved`) | 2.253B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step2456_2253M` | | |
| | `ar-starved` | `model-checkpoints/monolingual/ar/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `ar__unigram_starved` | | |
| | `ar-starved-5b` | `model-checkpoints/monolingual/ar/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step5181_4753M` | | |
| | `ar-starved-8b` | `model-checkpoints/monolingual/ar/419-languages/8b` | 419 languages (`unigram_starved`) | 7.753B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step8451_7753M` | | |
| | `ar-fair-12b` | `model-checkpoints/monolingual/ar/5-languages/12b` | 5 languages (`unigram_destarved`) | 11.754B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step12811_11754M` | | |
| | `ar-fair-15b` | `model-checkpoints/monolingual/ar/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.754B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step16081_14754M` | | |
| | `ar-fair-1b` | `model-checkpoints/monolingual/ar/5-languages/1b` | 5 languages (`unigram_destarved`) | 1.001B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step1092_1001M` | | |
| | `ar-fair-2b` | `model-checkpoints/monolingual/ar/5-languages/2b` | 5 languages (`unigram_destarved`) | 2.253B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step2456_2253M` | | |
| | `ar-fair` | `model-checkpoints/monolingual/ar/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `ar__unigram_destarved` | | |
| | `ar-fair-5b` | `model-checkpoints/monolingual/ar/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step5181_4753M` | | |
| | `ar-fair-8b` | `model-checkpoints/monolingual/ar/5-languages/8b` | 5 languages (`unigram_destarved`) | 7.753B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step8451_7753M` | | |
| | `de-starved-12b` | `model-checkpoints/monolingual/de/419-languages/12b` | 419 languages (`unigram_starved`) | 11.754B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step12811_11754M` | | |
| | `de-starved-15b` | `model-checkpoints/monolingual/de/419-languages/15b` | 419 languages (`unigram_starved`) | 14.754B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step16081_14754M` | | |
| | `de-starved-16b` | `model-checkpoints/monolingual/de/419-languages/16b` | 419 languages (`unigram_starved`) | 16.1B | mid-stable (peak LR 3e-3) | `de__unigram_starved` | | |
| | `de-starved-1b` | `model-checkpoints/monolingual/de/419-languages/1b` | 419 languages (`unigram_starved`) | 1.001B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step1092_1001M` | | |
| | `de-starved-2b` | `model-checkpoints/monolingual/de/419-languages/2b` | 419 languages (`unigram_starved`) | 2.253B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step2456_2253M` | | |
| | `de-starved-5b` | `model-checkpoints/monolingual/de/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step5181_4753M` | | |
| | `de-starved-8b` | `model-checkpoints/monolingual/de/419-languages/8b` | 419 languages (`unigram_starved`) | 7.753B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step8451_7753M` | | |
| | `de-fair-12b` | `model-checkpoints/monolingual/de/5-languages/12b` | 5 languages (`unigram_destarved`) | 11.754B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step12811_11754M` | | |
| | `de-fair-15b` | `model-checkpoints/monolingual/de/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.755B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step16009_14755M` | | |
| | `de-fair-1b` | `model-checkpoints/monolingual/de/5-languages/1b` | 5 languages (`unigram_destarved`) | 1.001B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step1092_1001M` | | |
| | `de-fair-2b` | `model-checkpoints/monolingual/de/5-languages/2b` | 5 languages (`unigram_destarved`) | 2.253B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step2456_2253M` | | |
| | `de-fair` | `model-checkpoints/monolingual/de/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `de__unigram_destarved` | | |
| | `de-fair-5b` | `model-checkpoints/monolingual/de/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step5181_4753M` | | |
| | `de-fair-8b` | `model-checkpoints/monolingual/de/5-languages/8b` | 5 languages (`unigram_destarved`) | 7.753B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step8451_7753M` | | |
| | `en-starved-12b` | `model-checkpoints/monolingual/en/419-languages/12b` | 419 languages (`unigram_starved`) | 11.754B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step12739_11754M` | | |
| | `en-starved-15b` | `model-checkpoints/monolingual/en/419-languages/15b` | 419 languages (`unigram_starved`) | 14.757B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step15793_14757M` | | |
| | `en-starved-1b` | `model-checkpoints/monolingual/en/419-languages/1b` | 419 languages (`unigram_starved`) | 1.001B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step1092_1001M` | | |
| | `en-starved-2b` | `model-checkpoints/monolingual/en/419-languages/2b` | 419 languages (`unigram_starved`) | 2.253B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step2456_2253M` | | |
| | `en-starved` | `model-checkpoints/monolingual/en/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `en__unigram_starved` | | |
| | `en-starved-5b` | `model-checkpoints/monolingual/en/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step5181_4753M` | | |
| | `en-starved-8b` | `model-checkpoints/monolingual/en/419-languages/8b` | 419 languages (`unigram_starved`) | 7.753B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step8451_7753M` | | |
| | `en-fair-12b` | `model-checkpoints/monolingual/en/5-languages/12b` | 5 languages (`unigram_destarved`) | 11.754B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step12811_11754M` | | |
| | `en-fair-15b` | `model-checkpoints/monolingual/en/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.755B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step15937_14755M` | | |
| | `en-fair-1b` | `model-checkpoints/monolingual/en/5-languages/1b` | 5 languages (`unigram_destarved`) | 1.001B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step1092_1001M` | | |
| | `en-fair-2b` | `model-checkpoints/monolingual/en/5-languages/2b` | 5 languages (`unigram_destarved`) | 2.253B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step2456_2253M` | | |
| | `en-fair` | `model-checkpoints/monolingual/en/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `en__unigram_destarved` | | |
| | `en-fair-5b` | `model-checkpoints/monolingual/en/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step5181_4753M` | | |
| | `en-fair-8b` | `model-checkpoints/monolingual/en/5-languages/8b` | 5 languages (`unigram_destarved`) | 7.753B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step8451_7753M` | | |
| | `fr-starved-12b` | `model-checkpoints/monolingual/fr/419-languages/12b` | 419 languages (`unigram_starved`) | 11.754B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step12811_11754M` | | |
| | `fr-starved-15b` | `model-checkpoints/monolingual/fr/419-languages/15b` | 419 languages (`unigram_starved`) | 14.755B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step16009_14755M` | | |
| | `fr-starved-1b` | `model-checkpoints/monolingual/fr/419-languages/1b` | 419 languages (`unigram_starved`) | 1.001B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step1092_1001M` | | |
| | `fr-starved-2b` | `model-checkpoints/monolingual/fr/419-languages/2b` | 419 languages (`unigram_starved`) | 2.253B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step2456_2253M` | | |
| | `fr-starved` | `model-checkpoints/monolingual/fr/419-languages/30b-final` | 419 languages (`unigram_starved`) | 30B | cooled final (LR 3e-4) | `fr__unigram_starved` | | |
| | `fr-starved-5b` | `model-checkpoints/monolingual/fr/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step5181_4753M` | | |
| | `fr-starved-8b` | `model-checkpoints/monolingual/fr/419-languages/8b` | 419 languages (`unigram_starved`) | 7.753B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step8451_7753M` | | |
| | `fr-fair-12b` | `model-checkpoints/monolingual/fr/5-languages/12b` | 5 languages (`unigram_destarved`) | 11.754B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step12811_11754M` | | |
| | `fr-fair-15b` | `model-checkpoints/monolingual/fr/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.755B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step16009_14755M` | | |
| | `fr-fair-1b` | `model-checkpoints/monolingual/fr/5-languages/1b` | 5 languages (`unigram_destarved`) | 1.001B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step1092_1001M` | | |
| | `fr-fair-2b` | `model-checkpoints/monolingual/fr/5-languages/2b` | 5 languages (`unigram_destarved`) | 2.253B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step2456_2253M` | | |
| | `fr-fair` | `model-checkpoints/monolingual/fr/5-languages/30b-final` | 5 languages (`unigram_destarved`) | 30B | cooled final (LR 3e-4) | `fr__unigram_destarved` | | |
| | `fr-fair-5b` | `model-checkpoints/monolingual/fr/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step5181_4753M` | | |
| | `fr-fair-8b` | `model-checkpoints/monolingual/fr/5-languages/8b` | 5 languages (`unigram_destarved`) | 7.753B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step8451_7753M` | | |
| | `zh-starved-12b` | `model-checkpoints/monolingual/zh/419-languages/12b` | 419 languages (`unigram_starved`) | 11.754B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step12811_11754M` | | |
| | `zh-starved-15b` | `model-checkpoints/monolingual/zh/419-languages/15b` | 419 languages (`unigram_starved`) | 14.756B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step15865_14756M` | | |
| | `zh-starved-1b` | `model-checkpoints/monolingual/zh/419-languages/1b` | 419 languages (`unigram_starved`) | 1.001B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step1092_1001M` | | |
| | `zh-starved-2b` | `model-checkpoints/monolingual/zh/419-languages/2b` | 419 languages (`unigram_starved`) | 2.253B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step2456_2253M` | | |
| | `zh-starved-5b` | `model-checkpoints/monolingual/zh/419-languages/5b` | 419 languages (`unigram_starved`) | 4.753B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step5181_4753M` | | |
| | `zh-starved-8b` | `model-checkpoints/monolingual/zh/419-languages/8b` | 419 languages (`unigram_starved`) | 7.753B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step8451_7753M` | | |
| | `zh-fair-12b` | `model-checkpoints/monolingual/zh/5-languages/12b` | 5 languages (`unigram_destarved`) | 11.754B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step12811_11754M` | | |
| | `zh-fair-15b` | `model-checkpoints/monolingual/zh/5-languages/15b` | 5 languages (`unigram_destarved`) | 14.756B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step15865_14756M` | | |
| | `zh-fair-1b` | `model-checkpoints/monolingual/zh/5-languages/1b` | 5 languages (`unigram_destarved`) | 1.001B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step1092_1001M` | | |
| | `zh-fair-2b` | `model-checkpoints/monolingual/zh/5-languages/2b` | 5 languages (`unigram_destarved`) | 2.253B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step2456_2253M` | | |
| | `zh-fair-5b` | `model-checkpoints/monolingual/zh/5-languages/5b` | 5 languages (`unigram_destarved`) | 4.753B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step5181_4753M` | | |
| | `zh-fair-8b` | `model-checkpoints/monolingual/zh/5-languages/8b` | 5 languages (`unigram_destarved`) | 7.753B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step8451_7753M` | | |
| </details> | |
| ## Training states | |
| `training-states/` holds the full training-state archives (step checkpoints, | |
| `last.pt`, per-rank optimizer shards) of the 20- and 2-language bilingual runs | |
| (`en-X__unigram_20lang`, `en-X__unigram_bi_X`) and of the abandoned 100B-token | |
| extensions (`en-{de,ar}__unigram_{destarved,starved}__100b`, | |
| `en-ar__*__100b_scratch`). They are only needed to resume or re-analyse | |
| training, not for evaluation. | |
| ## Older material | |
| `src/xscript/`, `run_benchmarks.py` and `requirements.txt` at the top level are | |
| an early export of the code and predate the split-checkpoint layout; use the | |
| GitHub repo instead. | |
| <details><summary>Original runbook (outdated)</summary> | |
| The README this repository shipped with is kept below; its heading levels are | |
| shifted down one, nothing else is changed. It describes the first 15-checkpoint | |
| export (whole `final.pt` files, three benchmarks) and is out of date on the | |
| layout and the model count. | |
| ## External benchmark evaluation | |
| Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained | |
| checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer | |
| (`src/xscript/model.py`) + SentencePiece tokenizer — pure PyTorch, using | |
| `F.scaled_dot_product_attention`, no flash-attn / triton / custom kernels — so | |
| they run on any stock GPU (or CPU, slowly). Each model is ~1B params (fits any | |
| 16GB GPU). | |
| The benchmark harness (`src/xscript/eval/bench.py`) wraps our model into | |
| lm-evaluation-harness and scores Global-MMLU, Belebele, and XNLI on each run's | |
| training languages. It is the *same* harness we would have run on-cluster, so | |
| numbers are directly comparable. | |
| ### 1. Export from Isambard (already done by `upload_to_hf.py`) | |
| The private HF repo mirrors the on-cluster layout: | |
| ``` | |
| src/xscript/** # bundled model + harness code | |
| tokenizers/unigram_{starved,destarved}/{sp.model,meta.json} | |
| runs/<name>/checkpoints/final.pt # 15 checkpoints, fp32, ~4GB each | |
| models.json # friendly name -> tokenizer + langs + orig run | |
| run_benchmarks.py requirements.txt README.md | |
| ``` | |
| Models use friendly names `<mixture>-<starved|fair>` (e.g. `en-fair`, | |
| `en-ar-starved`). `models.json` maps each to its real tokenizer. | |
| ### 2. Run on your GPU | |
| ```bash | |
| ## clone just the runner (or download run_benchmarks.py + requirements.txt from the repo) | |
| pip install torch --index-url https://download.pytorch.org/whl/cu121 # match your CUDA | |
| pip install -r requirements.txt | |
| export HF_TOKEN=hf_... # while the repo is private | |
| ## quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST | |
| python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200 | |
| ## full suite once the quick pass looks sane | |
| python run_benchmarks.py --repo jvonrad/xscript-eval | |
| ``` | |
| The runner downloads one checkpoint at a time and deletes it after eval | |
| (`--keep-checkpoints` to retain), so peak disk is ~5GB. Results: | |
| ``` | |
| xscript_bench/results/bench/<run>_final.json # per-run task accuracies | |
| xscript_bench/results/summary.json # everything combined | |
| ``` | |
| Send those JSONs back for analysis. | |
| ### Notes | |
| - `--runs en-starved en-fair` limits to a subset (friendly names). | |
| - `--tasks xnli_en xnli_de` overrides the task list (default = the run's langs). | |
| - Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs). | |
| - Scores are ordinary accuracy (`acc,none`); raw harness output is preserved in | |
| each per-run JSON for length-normalized variants. | |
| </details> | |