Maggio33 commited on
Commit
0729450
·
verified ·
1 Parent(s): ab1b0f4

Card: finished runs (R1-R3, K22) with results; resume-test folder

Browse files
Files changed (1) hide show
  1. README.md +31 -5
README.md CHANGED
@@ -7,13 +7,29 @@ tags:
7
  - training-history
8
  ---
9
 
10
- # Ablation checkpoints (~180M parameters, 2.5B-token budget) — interrupted runs
11
 
12
- Checkpoints of four single-factor ablation runs. Each run changes **one** factor of a common reference recipe
 
13
  (16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
14
  Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
15
  planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B).
16
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  | run | change from the reference | stopped at step | checkpoints here |
18
  |---|---|---:|---|
19
  | K12 | 12 layers × d_model 1024 × 16 heads (shallower, wider) | 3060 | 500, 1000, 1500, 2000, 2500, 3000 |
@@ -26,12 +42,22 @@ no final evaluation exists for them. The checkpoints are published so that the r
26
  Each file holds weights and optimizer state (`torch.save` dictionary of `train_gpt_ref_r23.py`: `model`, `opt`, `step`,
27
  `config`); load with `torch.load(path, map_location="cpu", weights_only=True)`.
28
 
29
- `mini_test/` (if present): artifacts of a 40-step resume test of the trainer, not a model.
30
 
31
  ## SHA-256
32
 
33
- Recorded by the checkpoint copier on the training machine (files dated before the interruption). These sums were stored
34
- on the same machine; no independent copy of them exists. Every file here was downloaded back and matches them.
 
 
 
 
 
 
 
 
 
 
35
 
36
  ```
37
  846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 K12/ckpt_500.pt
 
7
  - training-history
8
  ---
9
 
10
+ # Ablation checkpoints (~180M parameters, 2.5B-token budget)
11
 
12
+ Checkpoints of single-factor ablation runs: four finished runs (final checkpoints) and four runs interrupted mid-training
13
+ (intermediate checkpoints). Each run changes **one** factor of a common reference recipe
14
  (16 layers × d_model 896 × 14 heads, RMSNorm, QK-norm, RoPE θ 500 000, value residual, SwiGLU, no biases,
15
  Muon lr 0.02 + AdamW lr 6e-4, warmup 500, constant lr, decay over the last 20 % to lr / 10; batch 128 × 4096 tokens;
16
  planned 4768 steps = 2.5B tokens; tokenizer: the 32 768-token tokenizer of Slayer1B).
17
 
18
+ ## Finished runs (final checkpoint, step 4768)
19
+
20
+ | run | change from the reference | final checkpoint | val bpb PL | val bpb EN | weighted bpb (0.4 PL + 0.6 EN) |
21
+ |---|---|---|---:|---:|---:|
22
+ | R1 | none (reference), seed 1 | `R1/ckpt_4768.pt` | 1.086412 | 0.999300 | 1.034145 |
23
+ | R2 | none (reference), seed 2 | `R2/ckpt_4768.pt` | 1.088138 | 1.000682 | 1.035664 |
24
+ | R3 | none (reference), seed 3 | `R3/ckpt_4768.pt` | 1.086788 | 1.000410 | 1.034961 |
25
+ | K22 | 22 layers × d_model 768 × 12 heads (deeper, narrower) | `K22/ckpt_4768.pt` | 1.091207 | 1.003861 | 1.038799 |
26
+
27
+ Bits per byte on held-out validation text (Polish and English), measured once after the last step. The three reference
28
+ seeds give mean 1.034923; K22 is 0.003876 higher (worse), beyond the decision threshold of 0.00215 fixed before the
29
+ runs.
30
+
31
+ ## Interrupted runs
32
+
33
  | run | change from the reference | stopped at step | checkpoints here |
34
  |---|---|---:|---|
35
  | K12 | 12 layers × d_model 1024 × 16 heads (shallower, wider) | 3060 | 500, 1000, 1500, 2000, 2500, 3000 |
 
42
  Each file holds weights and optimizer state (`torch.save` dictionary of `train_gpt_ref_r23.py`: `model`, `opt`, `step`,
43
  `config`); load with `torch.load(path, map_location="cpu", weights_only=True)`.
44
 
45
+ `test_wznowienia/`: checkpoints of a 40-step resume test of the trainer (continuous run vs. resumed run), not a model.
46
 
47
  ## SHA-256
48
 
49
+ Finished runs — recorded off the machine before the copy (in the result file of each run); files on this repository
50
+ are downloaded back and checked against them:
51
+
52
+ ```
53
+ c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1 R1/ckpt_4768.pt
54
+ f4a06ade988a35bfa2d47f686253907400eff9eacd62c91818457fd863d48394 R2/ckpt_4768.pt
55
+ ea6748f1bcd72a38d3707bc33ef319125fe9c5e9d273e2d424d733dbe32d4564 R3/ckpt_4768.pt
56
+ 0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189 K22/ckpt_4768.pt
57
+ ```
58
+
59
+ Interrupted runs — recorded by the checkpoint copier on the training machine (files dated before the interruption). These sums were stored
60
+ on the same machine; no independent copy of them exists. Every file listed was downloaded back and matches them.
61
 
62
  ```
63
  846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 K12/ckpt_500.pt