Card: finished runs (R1-R3, K22) with results; resume-test folder
Browse files
README.md
CHANGED
|
@@ -7,13 +7,29 @@ tags:
|
|
| 7 |
- training-history
|
| 8 |
---
|
| 9 |
|
| 10 |
-
# Ablation checkpoints (~180M parameters, 2.5B-token budget)
|
| 11 |
|
| 12 |
-
Checkpoints of
|
|
|
|
| 13 |
(16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
|
| 14 |
Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
|
| 15 |
planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B).
|
| 16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
| run | change from the reference | stopped at step | checkpoints here |
|
| 18 |
|---|---|---:|---|
|
| 19 |
| K12 | 12 layers × d_model 1024 × 16 heads (shallower, wider) | 3060 | 500, 1000, 1500, 2000, 2500, 3000 |
|
|
@@ -26,12 +42,22 @@ no final evaluation exists for them. The checkpoints are published so that the r
|
|
| 26 |
Each file holds weights and optimizer state (`torch.save` dictionary of `train_gpt_ref_r23.py`: `model`, `opt`, `step`,
|
| 27 |
`config`); load with `torch.load(path, map_location="cpu", weights_only=True)`.
|
| 28 |
|
| 29 |
-
`
|
| 30 |
|
| 31 |
## SHA-256
|
| 32 |
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
```
|
| 37 |
846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 K12/ckpt_500.pt
|
|
|
|
| 7 |
- training-history
|
| 8 |
---
|
| 9 |
|
| 10 |
+
# Ablation checkpoints (~180M parameters, 2.5B-token budget)
|
| 11 |
|
| 12 |
+
Checkpoints of single-factor ablation runs: four finished runs (final checkpoints) and four runs interrupted mid-training
|
| 13 |
+
(intermediate checkpoints). Each run changes **one** factor of a common reference recipe
|
| 14 |
(16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
|
| 15 |
Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
|
| 16 |
planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B).
|
| 17 |
|
| 18 |
+
## Finished runs (final checkpoint, step 4768)
|
| 19 |
+
|
| 20 |
+
| run | change from the reference | final checkpoint | val bpb PL | val bpb EN | weighted bpb (0.4 PL + 0.6 EN) |
|
| 21 |
+
|---|---|---|---:|---:|---:|
|
| 22 |
+
| R1 | none (reference), seed 1 | `R1/ckpt_4768.pt` | 1.086412 | 0.999300 | 1.034145 |
|
| 23 |
+
| R2 | none (reference), seed 2 | `R2/ckpt_4768.pt` | 1.088138 | 1.000682 | 1.035664 |
|
| 24 |
+
| R3 | none (reference), seed 3 | `R3/ckpt_4768.pt` | 1.086788 | 1.000410 | 1.034961 |
|
| 25 |
+
| K22 | 22 layers × d_model 768 × 12 heads (deeper, narrower) | `K22/ckpt_4768.pt` | 1.091207 | 1.003861 | 1.038799 |
|
| 26 |
+
|
| 27 |
+
Bits per byte on held-out validation text (Polish and English), measured once after the last step. The three reference
|
| 28 |
+
seeds give mean 1.034923; K22 is 0.003876 higher (worse), beyond the decision threshold of 0.00215 fixed before the
|
| 29 |
+
runs.
|
| 30 |
+
|
| 31 |
+
## Interrupted runs
|
| 32 |
+
|
| 33 |
| run | change from the reference | stopped at step | checkpoints here |
|
| 34 |
|---|---|---:|---|
|
| 35 |
| K12 | 12 layers × d_model 1024 × 16 heads (shallower, wider) | 3060 | 500, 1000, 1500, 2000, 2500, 3000 |
|
|
|
|
| 42 |
Each file holds weights and optimizer state (`torch.save` dictionary of `train_gpt_ref_r23.py`: `model`, `opt`, `step`,
|
| 43 |
`config`); load with `torch.load(path, map_location="cpu", weights_only=True)`.
|
| 44 |
|
| 45 |
+
`test_wznowienia/`: checkpoints of a 40-step resume test of the trainer (continuous run vs. resumed run), not a model.
|
| 46 |
|
| 47 |
## SHA-256
|
| 48 |
|
| 49 |
+
Finished runs — recorded off the machine before the copy (in the result file of each run); files on this repository
|
| 50 |
+
are downloaded back and checked against them:
|
| 51 |
+
|
| 52 |
+
```
|
| 53 |
+
c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1 R1/ckpt_4768.pt
|
| 54 |
+
f4a06ade988a35bfa2d47f686253907400eff9eacd62c91818457fd863d48394 R2/ckpt_4768.pt
|
| 55 |
+
ea6748f1bcd72a38d3707bc33ef319125fe9c5e9d273e2d424d733dbe32d4564 R3/ckpt_4768.pt
|
| 56 |
+
0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189 K22/ckpt_4768.pt
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
Interrupted runs — recorded by the checkpoint copier on the training machine (files dated before the interruption). These sums were stored
|
| 60 |
+
on the same machine; no independent copy of them exists. Every file listed was downloaded back and matches them.
|
| 61 |
|
| 62 |
```
|
| 63 |
846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 K12/ckpt_500.pt
|