27 days ago
-
feat: update RTX 5090 notebook runner and optimized config
28 days ago
-
config: set Stage 1 micro 32 accum 4 for peak Tensor Core speed
-
config: set micro 16 accum 8 with grad_ckpt true for safe high-throughput pretraining
-
fix: restore grad_ckpt and safe micro 8 accum 8 for zero-OOM guarantee on T4
-
perf: disable grad_ckpt on T4 and enlarge loss chunk_size to 2048 for 10k+ tok/s
-
config: update Stage 1 batch_size to 16 for micro=16 accum=8 per GPU
-
Update model/configs/train_kaggle_t4.yaml: log_every: 10 and real-time step streaming
-
Remove dead config model/configs/model_config.yaml
-
Remove dead config model/configs/viu_1_5b_moe_config.yaml
-
Remove dead config model/configs/viu1_moe_config.yaml
-
Archive model_config.yaml to deprecated folder
-
Archive viu_1_5b_moe_config.yaml to deprecated folder
-
Archive viu1_moe_config.yaml to deprecated folder
-
Fix model/configs/train_config.yaml: batch math comments and find_unused_parameters=False
-
Audit fix: Update model/configs/viu_1_5b_moe_config.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/viu1_moe_config.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/model_config.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/train_dense_baseline.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/train_h100_h200.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/train_rtx5090.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/train_rtx4090.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/train_rtx3090.yaml for ViuMini-Dense-360M
-
Audit fix: Update model/configs/config.json for ViuMini-Dense-360M
-
Audit fix: Update model/configs/viu1_dense_config.yaml for ViuMini-Dense-360M
-
Update model/configs/train_rtx5090.yaml for ViuMini-Dense-360M
-
Update model/configs/train_rtx4090.yaml for ViuMini-Dense-360M
-
Update model/configs/train_rtx3090.yaml for ViuMini-Dense-360M
-
Update model/configs/train_dense_baseline.yaml for ViuMini-Dense-360M
-
Update model/configs/train_config.yaml for ViuMini-Dense-360M
-
Update model/configs/train_kaggle_t4.yaml for ViuMini-Dense-360M
-
Update config.json for 32 layers, dim 960
-
Set Option A: 32 layers, dim 960 (366.6M params, SmolLM-style)
-
Upgrade architecture to ViuMini-Dense-350M (100% Dense, 360M active)
-
Upgrade architecture to ViuMini-Dense-350M (100% Dense, 360M active)
-
Upgrade architecture to ViuMini-Dense-350M (100% Dense, 360M active)
-
Set checkpoint save, eval, sample, and HF push intervals to 1000 steps
-
Switch to PagedAdamW8bit (lr=2.5e-4, warmup=1000) for 100% stable convergence
29 days ago
-
Set log_every=1 for live step-by-step console feedback
-
Restore full batch: grad_accum=16, eff_batch=128 (65,536 tokens/step) on full 126.5B tokens
-
Set grad_accum=8: 2x step speed, eff_batch=64 (32,768 tokens/step)
-
Restore batch_size=8 and grad_ckpt=True for 100% VRAM safety on Dual T4 (10.5GB)
-
Speed optimizations: batch_size=16, grad_ckpt=False, hf_max_blocks=25k, enforce_mix=False
-
Update warmup_steps to 1000
-
Update warmup_steps to 1000 for fast convergence on Kaggle Dual T4
-
Align dataset streaming config with ViuAI/viu-mini-raw-pretrain
-
Fix hf_repo to ViuMini-MoE-242M
-
Optimize train_kaggle_t4.yaml: micro-batch 8 / accum 16 (8 passes/GPU on Dual T4), log_every 1
-
Optimize train_rtx5090.yaml: micro-batch 32 / grad_accum 4 (~5-7s per step)
30 days ago
-
Optimize train_rtx4090.yaml: micro-batch 32 / grad_accum 4 (~10-12s per step)
-
RTX 3090: Upgrade Stage 1 micro-batch to 16, grad_accum to 32 (2x GEMM saturation, 32 passes/step)
