Sync with paper v2: subset manifest, comparable corpus sizes
Browse files
README.md
CHANGED
|
@@ -41,9 +41,11 @@ Armenian SFT.
|
|
| 41 |
token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).
|
| 42 |
The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled
|
| 43 |
at random with balanced stratification across the math and science
|
| 44 |
-
pools)
|
| 45 |
-
|
| 46 |
-
the
|
|
|
|
|
|
|
| 47 |
- **Details**: global batch 512 sequences (~2.1M tokens/step), 4,770 steps,
|
| 48 |
100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.
|
| 49 |
|
|
|
|
| 41 |
token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).
|
| 42 |
The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled
|
| 43 |
at random with balanced stratification across the math and science
|
| 44 |
+
pools); its manifest ships with ArmSTEM (`stats/cpt_training_subset_ids.txt`,
|
| 45 |
+
104,630 items verbatim in the release, 5,255 superseded before release).
|
| 46 |
+
A control run trained on the full 373K corpus (~1.3 epochs instead of
|
| 47 |
+
7–9) scores within noise of the released model (paper §5), so the
|
| 48 |
+
repetition is costless.
|
| 49 |
- **Details**: global batch 512 sequences (~2.1M tokens/step), 4,770 steps,
|
| 50 |
100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.
|
| 51 |
|