Maggio33 commited on
Commit
632be8f
·
verified ·
1 Parent(s): 64ae577

Slayer1B 1.09: update cards and training files

Browse files
Files changed (3) hide show
  1. DATA_CARD.md +1 -1
  2. README.md +1 -1
  3. TRAINING_CONFIGURATION.md +4 -2
DATA_CARD.md CHANGED
@@ -20,7 +20,7 @@ tags:
20
 
21
  The request is to exceed **100B unique tokens after continuation**, with PL/EN recomputed and allowed to differ from the earlier 21.37% target. Report inherited unique-data contribution, retained new-data capacity and actually processed positions separately. Use a declared common tokenizer/unit, including EOS where counted; replay, epochs, old exposure totals, proxy tokenizers and source-language labels cannot establish the unique-history total.
22
 
23
- The selected new-source allocation is **20% PL / 80% EN** for a planned 80,000 updates, or **83,886,080,000 new positions**. These are recipe targets, not retained or consumed counts. The inherited language split and verified unique-history contribution remain separate unresolved quantities; an upper estimate of approximately 23B cannot prove the over-100B target.
24
 
25
  The abandoned schedule was 95,000 updates × 1,048,576 positions = **99,614,720,000**, not exactly 100B. It is not a newly approved CPT length or a count of distinct data already seen by the starting model.
26
 
 
20
 
21
  The request is to exceed **100B unique tokens after continuation**, with PL/EN recomputed and allowed to differ from the earlier 21.37% target. Report inherited unique-data contribution, retained new-data capacity and actually processed positions separately. Use a declared common tokenizer/unit, including EOS where counted; replay, epochs, old exposure totals, proxy tokenizers and source-language labels cannot establish the unique-history total.
22
 
23
+ The selected new-source allocation is **20% PL / 80% EN** for a planned 80,000 updates, or **83,886,080,000 new positions**. These are recipe targets, not retained or consumed counts. The independently reviewed lineage count gives **18,863,257,682 inherited source/file-disjoint positions**, with an upper estimate of about 23B. With a complete pass over new deduplicated positions, the planned source-position total is at least **102,749,337,682**, approximately **102.7–107B**. Historical content-level near-duplicates have not been deducted, so this is not certification of a globally content-unique total or an achieved result. The inherited language split remains unresolved; 20/80 is not the cumulative model-language ratio.
24
 
25
  The abandoned schedule was 95,000 updates × 1,048,576 positions = **99,614,720,000**, not exactly 100B. It is not a newly approved CPT length or a count of distinct data already seen by the starting model.
26
 
README.md CHANGED
@@ -40,7 +40,7 @@ The [training configuration](TRAINING_CONFIGURATION.md) preserves exact evidence
40
  | Checkpoints | Full state every 200 updates; slot-end weight milestones; local workstation copies |
41
  | Code | Launcher v5 SHA256 `796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656`; exact command and converter pins in the configuration document |
42
 
43
- The reported inherited approximately 23B is an **upper estimate**, useful for conservative exclusion of previously seen data, not a verified lower count. Even if all planned new positions are unique and processed once, exceeding 100B requires supported inherited exposure **above 16,113,920,000 unique positions**. Approximately 107B is therefore a conditional estimate, not an established result.
44
 
45
  ## Archived from-scratch attempt — not the CPT recipe
46
 
 
40
  | Checkpoints | Full state every 200 updates; slot-end weight milestones; local workstation copies |
41
  | Code | Launcher v5 SHA256 `796c71016c1e167bfd3ca98b78518f91c3502a96eac233dd7186ef6639fcc656`; exact command and converter pins in the configuration document |
42
 
43
+ The lineage-based planning count is **at least about 18.86B inherited positions disjoint by source files**, with an upper estimate of about 23B. The lower count, **18,863,257,682**, was recomputed from deterministic sampler coverage and step logs and independently repeated in review; it does not multiply a source share by an epoch fraction. If all 80,000 updates consume new deduplicated positions once, the corresponding planned source-position total is at least **102,749,337,682**, approximately **102.7–107B**. **Historical content-level near-duplicates across sources have not been deducted**; this source-position calculation alone does not certify the requested globally content-unique total or actual completion.
44
 
45
  ## Archived from-scratch attempt — not the CPT recipe
46
 
TRAINING_CONFIGURATION.md CHANGED
@@ -31,9 +31,11 @@ bash /workspace/kod/uruchom_109.sh
31
 
32
  **Set all three overrides:** `KROKI=80000`, `S=5000`, `CKPT_EVERY=200`; v5 retains different historical defaults. `START=108` requires `--init-from` and a 2,000-update warmup. The converter passes weights/config only, not optimizer state; `--resume` recovers only this new experiment. A new output directory plus step-zero `INIT-FROM` preflight are mandatory before calling this actual CPT.
33
 
34
- Shared architecture/batch/optimizer peaks below are unchanged: eight B200 ranks, microbatch8 × accumulation4, context4096, Muon0.02/AdamW0.0006. The selected 80,000-update plan decays in its last20% to0.00006. Target new-source mix is20%PL/80%EN, not a measured cumulative language ratio.
35
 
36
- 80,000 updates correspond to83,886,080,000 new positions. The inherited approximately23B is an upper estimate, not proof of the lower unique-history count needed for the over100B goal; final counts and exposure must verify that separately. No old95,000 default or repeated source material supplies that proof.
 
 
37
 
38
  ## Historical code identities
39
 
 
31
 
32
  **Set all three overrides:** `KROKI=80000`, `S=5000`, `CKPT_EVERY=200`; v5 retains different historical defaults. `START=108` requires `--init-from` and a 2,000-update warmup. The converter passes weights/config only, not optimizer state; `--resume` recovers only this new experiment. A new output directory plus step-zero `INIT-FROM` preflight are mandatory before calling this actual CPT.
33
 
34
+ Shared architecture, batch and optimizer peaks below are unchanged: eight B200 ranks, microbatch 8 × accumulation 4, context 4,096, Muon 0.02 / AdamW 0.0006. The selected 80,000-update plan decays in its last 20% to 0.00006. Target new-source mix is 20% PL / 80% EN, not a measured cumulative language ratio.
35
 
36
+ 80,000 updates correspond to **83,886,080,000 new positions**. The reviewed inherited source/file-disjoint planning count is **18,863,257,682**, with an upper estimate of about 23B. If all planned updates consume new deduplicated positions once, the corresponding total is at least **102,749,337,682**, approximately **102.7–107B**. Historical content-level near-duplicates are not deducted; this is not certification of globally content-unique exposure, completed training or a measured final language split.
37
+
38
+ Counting footprint: counter SHA256 `abb39b8ce019c1a18a19ccf0ef4c32977c0c5a8f91496d4560125838789150f5`, using EpochSampler from trainer SHA `46f83b51…` with seed 20261109, 2,800,806,882 positions, block 4,096, batch 256 and 1,709 updates. The sampled target intersections with `[0, 1,925,806,163)` contribute 1,232,346,307 positions; the other terms are 13,848 × 1,048,576 = 14,520,680,448 and a conservative 3,110,230,927 Polish positions from 1.07. Review repeated offset 2,959, 683,790 windows/epoch and 437,504 used windows. These terms count source-disjoint positions, not a content-deduplicated historical union.
39
 
40
  ## Historical code identities
41