| # Model — NOTES (ViuMini-Dense-360M) |
| Target: 366.6M params (100% active), dim=960, L=32, H=15, KV=5, vocab=48k, intermediate_size=2624, hybrid SWA+full, QK-Norm, SwiGLU Dense. |
| |
| ## Structure |
| ``` |
| model/ |
| configs/ |
| scripts/ |
| checkpoints/ |
| ``` |
| |
| ## Log |
| - 2026-09-15: folder banaya, model.py baaki hai (tokenizer final hone ke baad config lock hogi) |
| - 2026-09-15: `scripts/model.py` banaya (RMSNorm, RoPE, SwiGLU, GQA, tied) + `configs/model_config.yaml` — test: 47.1M params (emb 24.6M + transformer 22.5M), smoke forward OK, 2 bugs fix (n_kv_heads, tied count) |
| - 2026-09-15: `scripts/viu1_moe.py` + `configs/viu1_moe_config.yaml` (VIU-1 Frontier) — test: total 217.8M / active 147.3M (target 200/135, ~9% upar, quality ke liye OK), smoke OK (loss 12.05, moe_aux 0.21), MoE=8 dense=16 sliding=16 full=8, n_kv bug fix |
| - 2026-09-15: 28 layers LOCK — test: total 241.7M / active 166.2M, smoke OK (loss 12.18), MoE=9 dense=19 sliding=19 full=9 |
| - 2026-09-15: `scripts/train.py` + `configs/train_config.yaml` — smoke OK CPU 3 steps (loss 6.18->6.01 down), ckpt/resume/eval/sample ready, warnings fix |
| - 2026-09-18: Codebase audit & P0/P1 fixes (`viu1_moe.py`, `train.py`) — multi-token target alignment fix (predict t+2, not t+3), RoPE half-precision dtype alignment, causal/sliding mask caching in Attention, export expert usages in aux_info, switch_gate output gating, AdamW weight decay groups (0.1 on 2D weights, 0.0 on RMSNorm/biases), AMP autocast in evaluate, scaler save/restore in checkpoints, temperature+top-k sampling in sample_text, real-time MoE max_share monitoring. Tests pass (241.7M/166.2M verified, smoke OK), HF pushed. |
| - 2026-09-18: Round 2 optimizations (`viu1_moe.py`, `train.py`) — FlashAttention native SDPA in full and sliding attention (fused softmax/dropout/matmul, ~50% VRAM drop), language tags `<|hinglish|>`, `<|hindi|>`, `<|english|>` input injection, probe 2GB AdamW GPU safety buffer, gradient accum aux averaging, slim ~480MB BF16 checkpoint export for HF. Parameter count verified 241,668,224 (241.7M) / active 166.2M (28L LOCK preserved). Smoke pass. |
| - 2026-10-04: ViuMini-Dense-360M Transition — 32 Layers Dense Transformer (Option A, SmolLM-360M blueprint scaled). Dim=960, L=32, H=15, KV=5 (GQA 3:1), intermediate_size=2624 (SwiGLU, mult=2.7, multiple_of=64), vocab=48,000 tied embeddings, Gemma 4 Pure QK-Norm, max_seq_len=2048. Exact parameter count: 366,617,536 (366.6M, 100% active). Optimizer: PagedAdamW8bit (lr: 2.5e-4, warmup: 1000). Checkpoint cadence: 1,000 steps. HF repo: ViuAI/ViuMini-Dense-360M. |
|
|
|
|
