Sync PROGRESS.md: Scale to Global Batch Size 128
Browse files- docs/PROGRESS.md +7 -0
docs/PROGRESS.md
CHANGED
|
@@ -433,3 +433,10 @@ Kaggle par manual file upload se version mismatch hota hai (v1 tokenizer wala ca
|
|
| 433 |
- Pretraining now ingests 100% of raw internet data including uncensored dialogues, Hinglish street banter, and slang without filtering.
|
| 434 |
- Added --hf_safety CLI argument to train.py for runtime override.
|
| 435 |
- Updated train_config.yaml (commit e7638d93) and train.py (commit a4854271) on ViuAI/ViuMini-MoE-242M.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 433 |
- Pretraining now ingests 100% of raw internet data including uncensored dialogues, Hinglish street banter, and slang without filtering.
|
| 434 |
- Added --hf_safety CLI argument to train.py for runtime override.
|
| 435 |
- Updated train_config.yaml (commit e7638d93) and train.py (commit a4854271) on ViuAI/ViuMini-MoE-242M.
|
| 436 |
+
|
| 437 |
+
- [x] 2026-09-24 SCALE TO GLOBAL BATCH SIZE 128 (131K TOKENS/STEP, 800K MAX STEPS):
|
| 438 |
+
- Upgraded target effective batch to 64 per GPU (grad_accum: 16, micro: 4) = 128 global sequences across 2x T4 GPUs.
|
| 439 |
+
- Model now processes 131,072 tokens per optimizer step (~131K tokens/step).
|
| 440 |
+
- Scaled total training steps from 6,400,000 to 800,000 steps (~105B tokens / 131,072).
|
| 441 |
+
- Checkpoint sync interval set to 2,000 steps (~6.8 hours / 262M tokens) for Kaggle 12-hour session safety.
|
| 442 |
+
- Verified and pushed live to ViuAI/ViuMini-MoE-242M (commit 39c9a3c2).
|