Model card: naming scheme, LR state, loading, checkpoint table

#1
by jvonrad - opened
Files changed (1) hide show
  1. README.md +327 -7
README.md CHANGED
@@ -1,4 +1,324 @@
1
- # External benchmark evaluation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
  Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained
4
  checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer
@@ -12,7 +332,7 @@ lm-evaluation-harness and scores Global-MMLU, Belebele, and XNLI on each run's
12
  training languages. It is the *same* harness we would have run on-cluster, so
13
  numbers are directly comparable.
14
 
15
- ## 1. Export from Isambard (already done by `upload_to_hf.py`)
16
 
17
  The private HF repo mirrors the on-cluster layout:
18
 
@@ -27,18 +347,18 @@ run_benchmarks.py requirements.txt README.md
27
  Models use friendly names `<mixture>-<starved|fair>` (e.g. `en-fair`,
28
  `en-ar-starved`). `models.json` maps each to its real tokenizer.
29
 
30
- ## 2. Run on your GPU
31
 
32
  ```bash
33
- # clone just the runner (or download run_benchmarks.py + requirements.txt from the repo)
34
  pip install torch --index-url https://download.pytorch.org/whl/cu121 # match your CUDA
35
  pip install -r requirements.txt
36
  export HF_TOKEN=hf_... # while the repo is private
37
 
38
- # quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST
39
  python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200
40
 
41
- # full suite once the quick pass looks sane
42
  python run_benchmarks.py --repo jvonrad/xscript-eval
43
  ```
44
 
@@ -52,7 +372,7 @@ xscript_bench/results/summary.json # everything combined
52
 
53
  Send those JSONs back for analysis.
54
 
55
- ## Notes
56
  - `--runs en-starved en-fair` limits to a subset (friendly names).
57
  - `--tasks xnli_en xnli_de` overrides the task list (default = the run's langs).
58
  - Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs).
 
1
+ ---
2
+ language:
3
+ - en
4
+ - de
5
+ - fr
6
+ - ar
7
+ - zh
8
+ tags:
9
+ - multilingual
10
+ - pretraining
11
+ - tokenization
12
+ - cross-lingual-transfer
13
+ - xscript-pretraining
14
+ ---
15
+
16
+ # XScript-Pretraining checkpoints
17
+
18
+ Checkpoints of ~1B-parameter Llama-style decoders (16 layers, d=2048, SwiGLU,
19
+ RoPE, untied embeddings, vocab 65,536) trained from scratch for the
20
+ XScript-Pretraining project: **does the cross-script pretraining penalty
21
+ (EN-AR/ZH transferring worse than EN-DE/FR) survive a fair tokenizer?** Each
22
+ model is trained on one or two of EN/DE/FR/AR/ZH (FineWeb-HQ / FineWeb2-HQ)
23
+ under one of several tokenizers that differ only in how many languages compete
24
+ for the 64k vocabulary.
25
+
26
+ * Code, experiment design and every finding (including the numbers that did
27
+ **not** survive scrutiny): [https://github.com/jvonrad/XScript-Pretraining](https://github.com/jvonrad/XScript-Pretraining) — see `README.md`, the scientific log
28
+ `CLAUDE.md` and `NEURON.md` (running on AWS Trainium).
29
+ * Cached per-layer alignment embeddings for most of these checkpoints:
30
+ [`jvonrad/xscript-embeddings`](https://huggingface.co/datasets/jvonrad/xscript-embeddings).
31
+
32
+ `models.json` currently lists **164 checkpoints**: 68 monolingual,
33
+ 48 fair/starved bilingual and 48 20lang/bi bilingual.
34
+
35
+ ## Naming scheme
36
+
37
+ | pattern | meaning |
38
+ |---|---|
39
+ | `{lang}-{fair\|starved}` | monolingual model, `lang` in en/de/fr/ar/zh (zh only as `-Nb`) |
40
+ | `en-{X}-{fair\|starved}` | EN-anchored bilingual, X in de/fr/ar/zh, token-level 50/50 mixture |
41
+ | `en-{X}-20lang`, `en-{X}-bi` | the same bilinguals under the 20-language and the 2-language (en+X) tokenizer (trained later, on AWS Trainium) |
42
+ | `...-Nb` suffix | an intermediate checkpoint at roughly N billion **total** training tokens (exact count in the table / `orig_run`) |
43
+
44
+ Tokenizers (all SentencePiece Unigram, 65,536 pieces, byte fallback; ids 0-3 are
45
+ `<unk>/<bos>/<eos>/<pad>`):
46
+
47
+ | short name | tokenizer dir | trained on |
48
+ |---|---|---|
49
+ | **fair** | `unigram_destarved` | the 5 study languages (en/de/fr/ar/zh), byte-premium adjusted |
50
+ | **starved** | `unigram_starved` | ~419 FineWeb/FineWeb2 languages, T=100 (ATLAS-scale vocabulary competition) |
51
+ | **20lang** | `unigram_20lang` | 20 languages: the 5 study languages + ces, fas, hun, ind, ita, jpn, kor, nld, pol, por, ron, rus, swe, ukr, vie |
52
+ | **bi** | `unigram_bi_{de,fr,ar,zh}` | English + the one partner language |
53
+
54
+ ### ⚠️ LR state: `-Nb` checkpoints and finals are not comparable
55
+
56
+ Every run uses the same WSD schedule (`configs/base_main.yaml`): warmup 1B,
57
+ stable at peak LR 3.0e-3 until **24B**, then decay to 3.0e-4 by 30B.
58
+
59
+ * An **unsuffixed** name is the **cooled 30B final**.
60
+ * Every **`-Nb`** checkpoint (1b ... 23b) is a **mid-stable** snapshot at peak LR.
61
+ * **Exception: `de-starved-16b`** is the largest `de-starved` checkpoint and is
62
+ **not** cooled: that run (a re-seeded retrain) stops at 16.10B mid-stable.
63
+ There is deliberately no bare `de-starved`.
64
+ * There is no 30B Chinese monolingual: `zh-{fair,starved}` exist only up to `-15b`.
65
+
66
+ Pairing a mid-stable checkpoint with a cooled final hands the cooled model the
67
+ whole decay phase for free; this confound is documented in the project's
68
+ CLAUDE.md §6 (and §6h for `de-starved`). Compare like with like.
69
+
70
+ ## Repository layout
71
+
72
+ ```
73
+ models.json friendly name -> {orig_run, tok, langs}
74
+ runs/<model>/checkpoints/final.pt.part000..004 the checkpoint, split into 5 parts (~900 MB each)
75
+ runs/<model>/checkpoints/n_parts.txt number of parts (5)
76
+ tokenizers/<tok>/{sp.model, meta.json} the tokenizers named in models.json
77
+ src/xscript/** bundled (older) copy of the model + harness code
78
+ run_benchmarks.py, requirements.txt standalone benchmark runner (older copy, see below)
79
+ ```
80
+
81
+ Do not rename files or edit `models.json`: the project's loaders resolve
82
+ checkpoints by these exact paths.
83
+
84
+ `runs/` also holds 14 directories that are **not** in `models.json`: full
85
+ training-state archives (all step checkpoints, `last.pt` and per-rank optimizer
86
+ shards) of the native-Trainium runs (`en-X__unigram_20lang`,
87
+ `en-X__unigram_bi_X`) and of the 100B-extension trunks
88
+ (`en-{de,ar}__unigram_{destarved,starved}__100b`, `en-ar__*__100b_scratch`).
89
+ They exist so those runs can be resumed or re-analysed, not for evaluation.
90
+
91
+ ## Loading a checkpoint
92
+
93
+ The parts are a plain byte split of one `torch.save` file: concatenate them in
94
+ order and load the result. The project's
95
+ [`scripts/external_bench/run_benchmarks.py`](https://github.com/jvonrad/XScript-Pretraining/blob/main/scripts/external_bench/run_benchmarks.py)
96
+ does this in `fetch_checkpoint()` (checks the count against `n_parts.txt`,
97
+ reassembles into `<workdir>/_assembled/runs/<model>/checkpoints/final.pt`), and
98
+ every eval runner in the repo reuses it. Minimal stand-alone version:
99
+
100
+ ```python
101
+ import json, shutil
102
+ from huggingface_hub import hf_hub_download
103
+
104
+ repo, name = "jvonrad/xscript-eval", "en-ar-fair"
105
+ spec = json.load(open(hf_hub_download(repo, "models.json")))[name]
106
+ d = f"runs/{name}/checkpoints"
107
+ n = int(open(hf_hub_download(repo, f"{d}/n_parts.txt")).read())
108
+ with open("final.pt", "wb") as out:
109
+ for i in range(n):
110
+ with open(hf_hub_download(repo, f"{d}/final.pt.part{i:03d}"), "rb") as part:
111
+ shutil.copyfileobj(part, out)
112
+ sp_model = hf_hub_download(repo, f"tokenizers/{spec['tok']}/sp.model")
113
+
114
+ # with the GitHub repo's src/ on sys.path:
115
+ import torch
116
+ from xscript.model import ModelConfig, Transformer
117
+ ck = torch.load("final.pt", map_location="cpu", weights_only=False)
118
+ model = Transformer(ModelConfig(**ck["cfg"]["model"]))
119
+ model.load_state_dict(ck["model"])
120
+ ```
121
+
122
+ (the same three lines as `src/xscript/eval/bench.py` and
123
+ `src/xscript/eval/logitlens.py::load_model`).
124
+
125
+ ⚠️ The `run_benchmarks.py` and `src/xscript/` copies **in this repo predate the
126
+ split-checkpoint layout**: this `run_benchmarks.py` has no `fetch_checkpoint()`
127
+ and looks for a whole `final.pt`, which no longer exists here. Use the GitHub
128
+ version.
129
+
130
+ ## All checkpoints (generated from `models.json`)
131
+
132
+ "tokens" is the total training-token count of the checkpoint (for bilinguals,
133
+ half of it is each language).
134
+
135
+ ### Monolingual
136
+
137
+ | model | languages | tokenizer | tokens | LR state | original run |
138
+ |---|---|---|---|---|---|
139
+ | `ar-fair-1b` | ar | `unigram_destarved` (fair) | 1.00B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step1092_1001M` |
140
+ | `ar-fair-2b` | ar | `unigram_destarved` (fair) | 2.25B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step2456_2253M` |
141
+ | `ar-fair-5b` | ar | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step5181_4753M` |
142
+ | `ar-fair-8b` | ar | `unigram_destarved` (fair) | 7.75B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step8451_7753M` |
143
+ | `ar-fair-12b` | ar | `unigram_destarved` (fair) | 11.75B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step12811_11754M` |
144
+ | `ar-fair-15b` | ar | `unigram_destarved` (fair) | 14.75B | mid-stable (peak LR 3e-3) | `ar__unigram_destarved__step16081_14754M` |
145
+ | `ar-fair` | ar | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `ar__unigram_destarved` |
146
+ | `ar-starved-1b` | ar | `unigram_starved` (starved) | 1.00B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step1092_1001M` |
147
+ | `ar-starved-2b` | ar | `unigram_starved` (starved) | 2.25B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step2456_2253M` |
148
+ | `ar-starved-5b` | ar | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step5181_4753M` |
149
+ | `ar-starved-8b` | ar | `unigram_starved` (starved) | 7.75B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step8451_7753M` |
150
+ | `ar-starved-12b` | ar | `unigram_starved` (starved) | 11.75B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step12811_11754M` |
151
+ | `ar-starved-15b` | ar | `unigram_starved` (starved) | 14.75B | mid-stable (peak LR 3e-3) | `ar__unigram_starved__step16081_14754M` |
152
+ | `ar-starved` | ar | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `ar__unigram_starved` |
153
+ | `de-fair-1b` | de | `unigram_destarved` (fair) | 1.00B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step1092_1001M` |
154
+ | `de-fair-2b` | de | `unigram_destarved` (fair) | 2.25B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step2456_2253M` |
155
+ | `de-fair-5b` | de | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step5181_4753M` |
156
+ | `de-fair-8b` | de | `unigram_destarved` (fair) | 7.75B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step8451_7753M` |
157
+ | `de-fair-12b` | de | `unigram_destarved` (fair) | 11.75B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step12811_11754M` |
158
+ | `de-fair-15b` | de | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `de__unigram_destarved__step16009_14755M` |
159
+ | `de-fair` | de | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `de__unigram_destarved` |
160
+ | `de-starved-1b` | de | `unigram_starved` (starved) | 1.00B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step1092_1001M` |
161
+ | `de-starved-2b` | de | `unigram_starved` (starved) | 2.25B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step2456_2253M` |
162
+ | `de-starved-5b` | de | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step5181_4753M` |
163
+ | `de-starved-8b` | de | `unigram_starved` (starved) | 7.75B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step8451_7753M` |
164
+ | `de-starved-12b` | de | `unigram_starved` (starved) | 11.75B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step12811_11754M` |
165
+ | `de-starved-15b` | de | `unigram_starved` (starved) | 14.75B | mid-stable (peak LR 3e-3) | `de__unigram_starved__step16081_14754M` |
166
+ | `de-starved-16b` | de | `unigram_starved` (starved) | 16.10B | mid-stable (peak LR 3e-3), stopped early (§6h) | `de__unigram_starved` |
167
+ | `en-fair-1b` | en | `unigram_destarved` (fair) | 1.00B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step1092_1001M` |
168
+ | `en-fair-2b` | en | `unigram_destarved` (fair) | 2.25B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step2456_2253M` |
169
+ | `en-fair-5b` | en | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step5181_4753M` |
170
+ | `en-fair-8b` | en | `unigram_destarved` (fair) | 7.75B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step8451_7753M` |
171
+ | `en-fair-12b` | en | `unigram_destarved` (fair) | 11.75B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step12811_11754M` |
172
+ | `en-fair-15b` | en | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `en__unigram_destarved__step15937_14755M` |
173
+ | `en-fair` | en | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `en__unigram_destarved` |
174
+ | `en-starved-1b` | en | `unigram_starved` (starved) | 1.00B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step1092_1001M` |
175
+ | `en-starved-2b` | en | `unigram_starved` (starved) | 2.25B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step2456_2253M` |
176
+ | `en-starved-5b` | en | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step5181_4753M` |
177
+ | `en-starved-8b` | en | `unigram_starved` (starved) | 7.75B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step8451_7753M` |
178
+ | `en-starved-12b` | en | `unigram_starved` (starved) | 11.75B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step12739_11754M` |
179
+ | `en-starved-15b` | en | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `en__unigram_starved__step15793_14757M` |
180
+ | `en-starved` | en | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `en__unigram_starved` |
181
+ | `fr-fair-1b` | fr | `unigram_destarved` (fair) | 1.00B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step1092_1001M` |
182
+ | `fr-fair-2b` | fr | `unigram_destarved` (fair) | 2.25B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step2456_2253M` |
183
+ | `fr-fair-5b` | fr | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step5181_4753M` |
184
+ | `fr-fair-8b` | fr | `unigram_destarved` (fair) | 7.75B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step8451_7753M` |
185
+ | `fr-fair-12b` | fr | `unigram_destarved` (fair) | 11.75B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step12811_11754M` |
186
+ | `fr-fair-15b` | fr | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `fr__unigram_destarved__step16009_14755M` |
187
+ | `fr-fair` | fr | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `fr__unigram_destarved` |
188
+ | `fr-starved-1b` | fr | `unigram_starved` (starved) | 1.00B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step1092_1001M` |
189
+ | `fr-starved-2b` | fr | `unigram_starved` (starved) | 2.25B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step2456_2253M` |
190
+ | `fr-starved-5b` | fr | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step5181_4753M` |
191
+ | `fr-starved-8b` | fr | `unigram_starved` (starved) | 7.75B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step8451_7753M` |
192
+ | `fr-starved-12b` | fr | `unigram_starved` (starved) | 11.75B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step12811_11754M` |
193
+ | `fr-starved-15b` | fr | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `fr__unigram_starved__step16009_14755M` |
194
+ | `fr-starved` | fr | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `fr__unigram_starved` |
195
+ | `zh-fair-1b` | zh | `unigram_destarved` (fair) | 1.00B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step1092_1001M` |
196
+ | `zh-fair-2b` | zh | `unigram_destarved` (fair) | 2.25B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step2456_2253M` |
197
+ | `zh-fair-5b` | zh | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step5181_4753M` |
198
+ | `zh-fair-8b` | zh | `unigram_destarved` (fair) | 7.75B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step8451_7753M` |
199
+ | `zh-fair-12b` | zh | `unigram_destarved` (fair) | 11.75B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step12811_11754M` |
200
+ | `zh-fair-15b` | zh | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `zh__unigram_destarved__step15865_14756M` |
201
+ | `zh-starved-1b` | zh | `unigram_starved` (starved) | 1.00B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step1092_1001M` |
202
+ | `zh-starved-2b` | zh | `unigram_starved` (starved) | 2.25B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step2456_2253M` |
203
+ | `zh-starved-5b` | zh | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step5181_4753M` |
204
+ | `zh-starved-8b` | zh | `unigram_starved` (starved) | 7.75B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step8451_7753M` |
205
+ | `zh-starved-12b` | zh | `unigram_starved` (starved) | 11.75B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step12811_11754M` |
206
+ | `zh-starved-15b` | zh | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `zh__unigram_starved__step15865_14756M` |
207
+
208
+ ### Bilingual, fair vs starved
209
+
210
+ | model | languages | tokenizer | tokens | LR state | original run |
211
+ |---|---|---|---|---|---|
212
+ | `en-ar-fair-2b` | en+ar | `unigram_destarved` (fair) | 1.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step1911_1753M` |
213
+ | `en-ar-fair-5b` | en+ar | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step5181_4753M` |
214
+ | `en-ar-fair-10b` | en+ar | `unigram_destarved` (fair) | 9.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step10631_9753M` |
215
+ | `en-ar-fair-15b` | en+ar | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step15937_14755M` |
216
+ | `en-ar-fair-23b` | en+ar | `unigram_destarved` (fair) | 22.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_destarved__step24077_22757M` |
217
+ | `en-ar-fair` | en+ar | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `en-ar__unigram_destarved` |
218
+ | `en-ar-starved-2b` | en+ar | `unigram_starved` (starved) | 1.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step1911_1753M` |
219
+ | `en-ar-starved-5b` | en+ar | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step5181_4753M` |
220
+ | `en-ar-starved-10b` | en+ar | `unigram_starved` (starved) | 9.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step10631_9753M` |
221
+ | `en-ar-starved-15b` | en+ar | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step15793_14757M` |
222
+ | `en-ar-starved-23b` | en+ar | `unigram_starved` (starved) | 22.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_starved__step23933_22758M` |
223
+ | `en-ar-starved` | en+ar | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `en-ar__unigram_starved` |
224
+ | `en-de-fair-2b` | en+de | `unigram_destarved` (fair) | 1.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step1911_1753M` |
225
+ | `en-de-fair-5b` | en+de | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step5181_4753M` |
226
+ | `en-de-fair-10b` | en+de | `unigram_destarved` (fair) | 9.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step10631_9753M` |
227
+ | `en-de-fair-15b` | en+de | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step15937_14755M` |
228
+ | `en-de-fair-23b` | en+de | `unigram_destarved` (fair) | 22.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_destarved__step24077_22757M` |
229
+ | `en-de-fair` | en+de | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `en-de__unigram_destarved` |
230
+ | `en-de-starved-2b` | en+de | `unigram_starved` (starved) | 1.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step1911_1753M` |
231
+ | `en-de-starved-5b` | en+de | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step5181_4753M` |
232
+ | `en-de-starved-10b` | en+de | `unigram_starved` (starved) | 9.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step10631_9753M` |
233
+ | `en-de-starved-15b` | en+de | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step15865_14756M` |
234
+ | `en-de-starved-23b` | en+de | `unigram_starved` (starved) | 22.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_starved__step24005_22758M` |
235
+ | `en-de-starved` | en+de | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `en-de__unigram_starved` |
236
+ | `en-fr-fair-2b` | en+fr | `unigram_destarved` (fair) | 1.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step1911_1753M` |
237
+ | `en-fr-fair-5b` | en+fr | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step5181_4753M` |
238
+ | `en-fr-fair-10b` | en+fr | `unigram_destarved` (fair) | 9.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step10631_9753M` |
239
+ | `en-fr-fair-15b` | en+fr | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step15865_14756M` |
240
+ | `en-fr-fair-23b` | en+fr | `unigram_destarved` (fair) | 22.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_destarved__step24005_22758M` |
241
+ | `en-fr-fair` | en+fr | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `en-fr__unigram_destarved` |
242
+ | `en-fr-starved-2b` | en+fr | `unigram_starved` (starved) | 1.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step1911_1753M` |
243
+ | `en-fr-starved-5b` | en+fr | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step5181_4753M` |
244
+ | `en-fr-starved-10b` | en+fr | `unigram_starved` (starved) | 9.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step10631_9753M` |
245
+ | `en-fr-starved-15b` | en+fr | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step15865_14756M` |
246
+ | `en-fr-starved-23b` | en+fr | `unigram_starved` (starved) | 22.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_starved__step24005_22758M` |
247
+ | `en-fr-starved` | en+fr | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `en-fr__unigram_starved` |
248
+ | `en-zh-fair-2b` | en+zh | `unigram_destarved` (fair) | 1.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step1911_1753M` |
249
+ | `en-zh-fair-5b` | en+zh | `unigram_destarved` (fair) | 4.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step5181_4753M` |
250
+ | `en-zh-fair-10b` | en+zh | `unigram_destarved` (fair) | 9.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step10631_9753M` |
251
+ | `en-zh-fair-15b` | en+zh | `unigram_destarved` (fair) | 14.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step15937_14755M` |
252
+ | `en-zh-fair-23b` | en+zh | `unigram_destarved` (fair) | 22.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_destarved__step24077_22757M` |
253
+ | `en-zh-fair` | en+zh | `unigram_destarved` (fair) | 30B | **cooled final** (LR 3e-4) | `en-zh__unigram_destarved` |
254
+ | `en-zh-starved-2b` | en+zh | `unigram_starved` (starved) | 1.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step1911_1753M` |
255
+ | `en-zh-starved-5b` | en+zh | `unigram_starved` (starved) | 4.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step5181_4753M` |
256
+ | `en-zh-starved-10b` | en+zh | `unigram_starved` (starved) | 9.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step10631_9753M` |
257
+ | `en-zh-starved-15b` | en+zh | `unigram_starved` (starved) | 14.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step15865_14756M` |
258
+ | `en-zh-starved-23b` | en+zh | `unigram_starved` (starved) | 22.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_starved__step24005_22758M` |
259
+ | `en-zh-starved` | en+zh | `unigram_starved` (starved) | 30B | **cooled final** (LR 3e-4) | `en-zh__unigram_starved` |
260
+
261
+ ### Bilingual, 20lang and bi tokenizers
262
+
263
+ | model | languages | tokenizer | tokens | LR state | original run |
264
+ |---|---|---|---|---|---|
265
+ | `en-ar-20lang-2b` | en+ar | `unigram_20lang` (20lang) | 1.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step1785_1754M` |
266
+ | `en-ar-20lang-5b` | en+ar | `unigram_20lang` (20lang) | 4.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step4839_4756M` |
267
+ | `en-ar-20lang-10b` | en+ar | `unigram_20lang` (20lang) | 9.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step9929_9760M` |
268
+ | `en-ar-20lang-15b` | en+ar | `unigram_20lang` (20lang) | 14.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step15019_14764M` |
269
+ | `en-ar-20lang-23b` | en+ar | `unigram_20lang` (20lang) | 22.77B | mid-stable (peak LR 3e-3) | `en-ar__unigram_20lang__step23159_22766M` |
270
+ | `en-ar-20lang` | en+ar | `unigram_20lang` (20lang) | 30B | **cooled final** (LR 3e-4) | `en-ar__unigram_20lang` |
271
+ | `en-ar-bi-2b` | en+ar | `unigram_bi_ar` (bi (en+ar)) | 1.75B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step1785_1754M` |
272
+ | `en-ar-bi-5b` | en+ar | `unigram_bi_ar` (bi (en+ar)) | 4.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step4839_4756M` |
273
+ | `en-ar-bi-10b` | en+ar | `unigram_bi_ar` (bi (en+ar)) | 9.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step9929_9760M` |
274
+ | `en-ar-bi-15b` | en+ar | `unigram_bi_ar` (bi (en+ar)) | 14.76B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step15019_14764M` |
275
+ | `en-ar-bi-23b` | en+ar | `unigram_bi_ar` (bi (en+ar)) | 22.77B | mid-stable (peak LR 3e-3) | `en-ar__unigram_bi_ar__step23163_22770M` |
276
+ | `en-ar-bi` | en+ar | `unigram_bi_ar` (bi (en+ar)) | 30B | **cooled final** (LR 3e-4) | `en-ar__unigram_bi_ar` |
277
+ | `en-de-20lang-2b` | en+de | `unigram_20lang` (20lang) | 1.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step1785_1754M` |
278
+ | `en-de-20lang-5b` | en+de | `unigram_20lang` (20lang) | 4.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step4839_4756M` |
279
+ | `en-de-20lang-10b` | en+de | `unigram_20lang` (20lang) | 9.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step9929_9760M` |
280
+ | `en-de-20lang-15b` | en+de | `unigram_20lang` (20lang) | 14.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step15019_14764M` |
281
+ | `en-de-20lang-23b` | en+de | `unigram_20lang` (20lang) | 22.77B | mid-stable (peak LR 3e-3) | `en-de__unigram_20lang__step23159_22766M` |
282
+ | `en-de-20lang` | en+de | `unigram_20lang` (20lang) | 30B | **cooled final** (LR 3e-4) | `en-de__unigram_20lang` |
283
+ | `en-de-bi-2b` | en+de | `unigram_bi_de` (bi (en+de)) | 1.75B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step1785_1754M` |
284
+ | `en-de-bi-5b` | en+de | `unigram_bi_de` (bi (en+de)) | 4.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step4839_4756M` |
285
+ | `en-de-bi-10b` | en+de | `unigram_bi_de` (bi (en+de)) | 9.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step9929_9760M` |
286
+ | `en-de-bi-15b` | en+de | `unigram_bi_de` (bi (en+de)) | 14.76B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step15019_14764M` |
287
+ | `en-de-bi-23b` | en+de | `unigram_bi_de` (bi (en+de)) | 22.77B | mid-stable (peak LR 3e-3) | `en-de__unigram_bi_de__step23163_22770M` |
288
+ | `en-de-bi` | en+de | `unigram_bi_de` (bi (en+de)) | 30B | **cooled final** (LR 3e-4) | `en-de__unigram_bi_de` |
289
+ | `en-fr-20lang-2b` | en+fr | `unigram_20lang` (20lang) | 1.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step1785_1754M` |
290
+ | `en-fr-20lang-5b` | en+fr | `unigram_20lang` (20lang) | 4.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step4839_4756M` |
291
+ | `en-fr-20lang-10b` | en+fr | `unigram_20lang` (20lang) | 9.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step9929_9760M` |
292
+ | `en-fr-20lang-15b` | en+fr | `unigram_20lang` (20lang) | 14.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step15019_14764M` |
293
+ | `en-fr-20lang-23b` | en+fr | `unigram_20lang` (20lang) | 22.77B | mid-stable (peak LR 3e-3) | `en-fr__unigram_20lang__step23159_22766M` |
294
+ | `en-fr-20lang` | en+fr | `unigram_20lang` (20lang) | 30B | **cooled final** (LR 3e-4) | `en-fr__unigram_20lang` |
295
+ | `en-fr-bi-2b` | en+fr | `unigram_bi_fr` (bi (en+fr)) | 1.75B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step1785_1754M` |
296
+ | `en-fr-bi-5b` | en+fr | `unigram_bi_fr` (bi (en+fr)) | 4.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step4839_4756M` |
297
+ | `en-fr-bi-10b` | en+fr | `unigram_bi_fr` (bi (en+fr)) | 9.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step9929_9760M` |
298
+ | `en-fr-bi-15b` | en+fr | `unigram_bi_fr` (bi (en+fr)) | 14.76B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step15019_14764M` |
299
+ | `en-fr-bi-23b` | en+fr | `unigram_bi_fr` (bi (en+fr)) | 22.77B | mid-stable (peak LR 3e-3) | `en-fr__unigram_bi_fr__step23163_22770M` |
300
+ | `en-fr-bi` | en+fr | `unigram_bi_fr` (bi (en+fr)) | 30B | **cooled final** (LR 3e-4) | `en-fr__unigram_bi_fr` |
301
+ | `en-zh-20lang-2b` | en+zh | `unigram_20lang` (20lang) | 1.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step1785_1754M` |
302
+ | `en-zh-20lang-5b` | en+zh | `unigram_20lang` (20lang) | 4.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step4839_4756M` |
303
+ | `en-zh-20lang-10b` | en+zh | `unigram_20lang` (20lang) | 9.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step9929_9760M` |
304
+ | `en-zh-20lang-15b` | en+zh | `unigram_20lang` (20lang) | 14.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step15019_14764M` |
305
+ | `en-zh-20lang-23b` | en+zh | `unigram_20lang` (20lang) | 22.77B | mid-stable (peak LR 3e-3) | `en-zh__unigram_20lang__step23159_22766M` |
306
+ | `en-zh-20lang` | en+zh | `unigram_20lang` (20lang) | 30B | **cooled final** (LR 3e-4) | `en-zh__unigram_20lang` |
307
+ | `en-zh-bi-2b` | en+zh | `unigram_bi_zh` (bi (en+zh)) | 1.75B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step1785_1754M` |
308
+ | `en-zh-bi-5b` | en+zh | `unigram_bi_zh` (bi (en+zh)) | 4.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step4839_4756M` |
309
+ | `en-zh-bi-10b` | en+zh | `unigram_bi_zh` (bi (en+zh)) | 9.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step9929_9760M` |
310
+ | `en-zh-bi-15b` | en+zh | `unigram_bi_zh` (bi (en+zh)) | 14.76B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step15019_14764M` |
311
+ | `en-zh-bi-23b` | en+zh | `unigram_bi_zh` (bi (en+zh)) | 22.77B | mid-stable (peak LR 3e-3) | `en-zh__unigram_bi_zh__step23163_22770M` |
312
+ | `en-zh-bi` | en+zh | `unigram_bi_zh` (bi (en+zh)) | 30B | **cooled final** (LR 3e-4) | `en-zh__unigram_bi_zh` |
313
+
314
+ ## Original runbook
315
+
316
+ The README this repository shipped with is kept below; its heading levels are
317
+ shifted down one, nothing else is changed. It describes the first 15-checkpoint
318
+ export (whole `final.pt` files, three benchmarks) and is out of date on the
319
+ layout and the model count.
320
+
321
+ ## External benchmark evaluation
322
 
323
  Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained
324
  checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer
 
332
  training languages. It is the *same* harness we would have run on-cluster, so
333
  numbers are directly comparable.
334
 
335
+ ### 1. Export from Isambard (already done by `upload_to_hf.py`)
336
 
337
  The private HF repo mirrors the on-cluster layout:
338
 
 
347
  Models use friendly names `<mixture>-<starved|fair>` (e.g. `en-fair`,
348
  `en-ar-starved`). `models.json` maps each to its real tokenizer.
349
 
350
+ ### 2. Run on your GPU
351
 
352
  ```bash
353
+ ## clone just the runner (or download run_benchmarks.py + requirements.txt from the repo)
354
  pip install torch --index-url https://download.pytorch.org/whl/cu121 # match your CUDA
355
  pip install -r requirements.txt
356
  export HF_TOKEN=hf_... # while the repo is private
357
 
358
+ ## quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST
359
  python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200
360
 
361
+ ## full suite once the quick pass looks sane
362
  python run_benchmarks.py --repo jvonrad/xscript-eval
363
  ```
364
 
 
372
 
373
  Send those JSONs back for analysis.
374
 
375
+ ### Notes
376
  - `--runs en-starved en-fair` limits to a subset (friendly names).
377
  - `--tasks xnli_en xnli_de` overrides the task list (default = the run's langs).
378
  - Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs).