Slayer1B 1.09: update cards and training files
Browse files- DATA_CARD.md +1 -1
- README.md +1 -1
- TRAINING_CONFIGURATION.md +4 -2
DATA_CARD.md
CHANGED
|
@@ -20,7 +20,7 @@ tags:
|
|
| 20 |
|
| 21 |
The request is to exceed **100B unique tokens after continuation**, with PL/EN recomputed and allowed to differ from the earlier 21.37% target. Report inherited unique-data contribution, retained new-data capacity and actually processed positions separately. Use a declared common tokenizer/unit, including EOS where counted; replay, epochs, old exposure totals, proxy tokenizers and source-language labels cannot establish the unique-history total.
|
| 22 |
|
| 23 |
-
The selected new-source allocation is **20% PL / 80% EN** for a planned 80,000 updates, or **83,886,080,000 new positions**. These are recipe targets, not retained or consumed counts. The
|
| 24 |
|
| 25 |
The abandoned schedule was 95,000 updates × 1,048,576 positions = **99,614,720,000**, not exactly 100B. It is not a newly approved CPT length or a count of distinct data already seen by the starting model.
|
| 26 |
|
|
|
|
| 20 |
|
| 21 |
The request is to exceed **100B unique tokens after continuation**, with PL/EN recomputed and allowed to differ from the earlier 21.37% target. Report inherited unique-data contribution, retained new-data capacity and actually processed positions separately. Use a declared common tokenizer/unit, including EOS where counted; replay, epochs, old exposure totals, proxy tokenizers and source-language labels cannot establish the unique-history total.
|
| 22 |
|
| 23 |
+
The selected new-source allocation is **20% PL / 80% EN** for a planned 80,000 updates, or **83,886,080,000 new positions**. These are recipe targets, not retained or consumed counts. The independently reviewed lineage count gives **18,863,257,682 inherited source/file-disjoint positions**, with an upper estimate of about 23B. With a complete pass over new deduplicated positions, the planned source-position total is at least **102,749,337,682**, approximately **102.7–107B**. Historical content-level near-duplicates have not been deducted, so this is not certification of a globally content-unique total or an achieved result. The inherited language split remains unresolved; 20/80 is not the cumulative model-language ratio.
|
| 24 |
|
| 25 |
The abandoned schedule was 95,000 updates × 1,048,576 positions = **99,614,720,000**, not exactly 100B. It is not a newly approved CPT length or a count of distinct data already seen by the starting model.
|
| 26 |
|
README.md
CHANGED
|
@@ -40,7 +40,7 @@ The [training configuration](TRAINING_CONFIGURATION.md) preserves exact evidence
|
|
| 40 |
| Checkpoints | Full state every 200 updates; slot-end weight milestones; local workstation copies |
|
| 41 |
| Code | Launcher v5 SHA256 `796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656`; exact command and converter pins in the configuration document |
|
| 42 |
|
| 43 |
-
The
|
| 44 |
|
| 45 |
## Archived from-scratch attempt — not the CPT recipe
|
| 46 |
|
|
|
|
| 40 |
| Checkpoints | Full state every 200 updates; slot-end weight milestones; local workstation copies |
|
| 41 |
| Code | Launcher v5 SHA256 `796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656`; exact command and converter pins in the configuration document |
|
| 42 |
|
| 43 |
+
The lineage-based planning count is **at least about 18.86B inherited positions disjoint by source files**, with an upper estimate of about 23B. The lower count, **18,863,257,682**, was recomputed from deterministic sampler coverage and step logs and independently repeated in review; it does not multiply a source share by an epoch fraction. If all 80,000 updates consume new deduplicated positions once, the corresponding planned source-position total is at least **102,749,337,682**, approximately **102.7–107B**. **Historical content-level near-duplicates across sources have not been deducted**; this source-position calculation alone does not certify the requested globally content-unique total or actual completion.
|
| 44 |
|
| 45 |
## Archived from-scratch attempt — not the CPT recipe
|
| 46 |
|
TRAINING_CONFIGURATION.md
CHANGED
|
@@ -31,9 +31,11 @@ bash /workspace/kod/uruchom_109.sh
|
|
| 31 |
|
| 32 |
**Set all three overrides:** `KROKI=80000`, `S=5000`, `CKPT_EVERY=200`; v5 retains different historical defaults. `START=108` requires `--init-from` and a 2,000-update warmup. The converter passes weights/config only, not optimizer state; `--resume` recovers only this new experiment. A new output directory plus step-zero `INIT-FROM` preflight are mandatory before calling this actual CPT.
|
| 33 |
|
| 34 |
-
Shared architecture
|
| 35 |
|
| 36 |
-
80,000 updates correspond
|
|
|
|
|
|
|
| 37 |
|
| 38 |
## Historical code identities
|
| 39 |
|
|
|
|
| 31 |
|
| 32 |
**Set all three overrides:** `KROKI=80000`, `S=5000`, `CKPT_EVERY=200`; v5 retains different historical defaults. `START=108` requires `--init-from` and a 2,000-update warmup. The converter passes weights/config only, not optimizer state; `--resume` recovers only this new experiment. A new output directory plus step-zero `INIT-FROM` preflight are mandatory before calling this actual CPT.
|
| 33 |
|
| 34 |
+
Shared architecture, batch and optimizer peaks below are unchanged: eight B200 ranks, microbatch 8 × accumulation 4, context 4,096, Muon 0.02 / AdamW 0.0006. The selected 80,000-update plan decays in its last 20% to 0.00006. Target new-source mix is 20% PL / 80% EN, not a measured cumulative language ratio.
|
| 35 |
|
| 36 |
+
80,000 updates correspond to **83,886,080,000 new positions**. The reviewed inherited source/file-disjoint planning count is **18,863,257,682**, with an upper estimate of about 23B. If all planned updates consume new deduplicated positions once, the corresponding total is at least **102,749,337,682**, approximately **102.7–107B**. Historical content-level near-duplicates are not deducted; this is not certification of globally content-unique exposure, completed training or a measured final language split.
|
| 37 |
+
|
| 38 |
+
Counting footprint: counter SHA256 `abb39b8ce019c1a18a19ccf0ef4c32977c0c5a8f91496d4560125838789150f5`, using EpochSampler from trainer SHA `46f83b51…` with seed 20261109, 2,800,806,882 positions, block 4,096, batch 256 and 1,709 updates. The sampled target intersections with `[0, 1,925,806,163)` contribute 1,232,346,307 positions; the other terms are 13,848 × 1,048,576 = 14,520,680,448 and a conservative 3,110,230,927 Polish positions from 1.07. Review repeated offset 2,959, 683,790 windows/epoch and 437,504 used windows. These terms count source-disjoint positions, not a content-deduplicated historical union.
|
| 39 |
|
| 40 |
## Historical code identities
|
| 41 |
|