Card: Track links on the owner's account
Browse files
README.md
CHANGED
|
@@ -27,11 +27,11 @@ the weights is pending the owners' decision. Everything up to `ckpt_6750.pt` is
|
|
| 27 |
| Hardware | steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from `ckpt_6750.pt` (sha256 `970518a1…`) with the same trainer, recipe, data order and global batch. The data order of the resume was verified against the original run (batch hash of zero-based iterations 6751 and 7149, i.e. native steps 6752 and 7150, identical). DDP; global batch 256 × 4096 tokens as in 1.00–1.02 (effective: 4 ranks × micro-batch 2 × gradient accumulation 32) |
|
| 28 |
| Checkpoints | every 150 steps from 5850 and the final step 7150; all are constant-lr states |
|
| 29 |
| Trainer | `train_gpt_ref_r24_1.py`: the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming (needed on 24 GB cards); the model definition is unchanged |
|
| 30 |
-
| Training curve | steps up to
|
| 31 |
-
| Benchmarks 1.00–1.03 (lm-eval 0.4.13, 0-shot) | https://track.fabryka.ai/run/
|
| 32 |
|
| 33 |
Training continues from `ckpt_7150.pt` on the rest of the prepared corpus (track:
|
| 34 |
-
https://track.fabryka.ai/run/
|
| 35 |
|
| 36 |
## Benchmark caveat
|
| 37 |
|
|
|
|
| 27 |
| Hardware | steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from `ckpt_6750.pt` (sha256 `970518a1…`) with the same trainer, recipe, data order and global batch. The data order of the resume was verified against the original run (batch hash of zero-based iterations 6751 and 7149, i.e. native steps 6752 and 7150, identical). DDP; global batch 256 × 4096 tokens as in 1.00–1.02 (effective: 4 ranks × micro-batch 2 × gradient accumulation 32) |
|
| 28 |
| Checkpoints | every 150 steps from 5850 and the final step 7150; all are constant-lr states |
|
| 29 |
| Trainer | `train_gpt_ref_r24_1.py`: the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming (needed on 24 GB cards); the model definition is unchanged |
|
| 30 |
+
| Training curve | steps up to 7150 (6860–7150 from the L40S resume): https://track.fabryka.ai/run/7ca4f84f-3b07-4525-8cde-f3a936dc3425 |
|
| 31 |
+
| Benchmarks 1.00–1.03 (lm-eval 0.4.13, 0-shot) | https://track.fabryka.ai/run/1426c0a8-2f07-5a03-80fc-92f7e922aa4b |
|
| 32 |
|
| 33 |
Training continues from `ckpt_7150.pt` on the rest of the prepared corpus (track:
|
| 34 |
+
https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5); those checkpoints belong to the next versions.
|
| 35 |
|
| 36 |
## Benchmark caveat
|
| 37 |
|