osoblanco commited on
Commit
e88d0bd
·
verified ·
1 Parent(s): 0ef4714

Sync with paper v2: subset manifest, comparable corpus sizes

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -41,9 +41,11 @@ Armenian SFT.
41
  token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).
42
  The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled
43
  at random with balanced stratification across the math and science
44
- pools). A control run trained on the full 373K corpus (~1.3 epochs
45
- instead of 7–9) scores within noise of the released model (paper §5), so
46
- the repetition is costless.
 
 
47
  - **Details**: global batch 512 sequences (~2.1M tokens/step), 4,770 steps,
48
  100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.
49
 
 
41
  token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).
42
  The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled
43
  at random with balanced stratification across the math and science
44
+ pools); its manifest ships with ArmSTEM (`stats/cpt_training_subset_ids.txt`,
45
+ 104,630 items verbatim in the release, 5,255 superseded before release).
46
+ A control run trained on the full 373K corpus (~1.3 epochs instead of
47
+ 7–9) scores within noise of the released model (paper §5), so the
48
+ repetition is costless.
49
  - **Details**: global batch 512 sequences (~2.1M tokens/step), 4,770 steps,
50
  100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.
51