ViuAI commited on
Commit
0aeabb2
·
verified ·
1 Parent(s): c49f48e

Update docs/PROGRESS.md for ViuMini-Dense-360M

Browse files
Files changed (1) hide show
  1. docs/PROGRESS.md +31 -2
docs/PROGRESS.md CHANGED
@@ -379,9 +379,38 @@ Active Sparsity: **16.19%** (Full 1.85B knowledge capacity at mobile-edge ~300M
379
  - DeepSeek-V4.1 introduced dynamic learnable **router bias** (\(\text{logits} = Wx + b_{\text{router}}\)) to achieve automatic load balancing without heavy loss penalties.
380
  4. **Muon Optimizer Integration**:
381
  - DeepSeek-V4 officially adopted the **Muon Optimizer** (5th-order Newton-Schulz polynomial iterations) for 2D weight matrices, matching our training engine.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
382
  * **Verification & Benchmarks**:
383
- - `verify_arch.py`: **100% Validated** — 1,855,545,600 total params, 300,474,624 active params (16.19% active per token), 1,392 experts across 48 layers.
384
- - `train.py --smoke`: Passed with Exit Code 0 (Muon + AdamW hybrid, loss: 5.32).
 
385
 
386
 
387
 
 
379
  - DeepSeek-V4.1 introduced dynamic learnable **router bias** (\(\text{logits} = Wx + b_{\text{router}}\)) to achieve automatic load balancing without heavy loss penalties.
380
  4. **Muon Optimizer Integration**:
381
  - DeepSeek-V4 officially adopted the **Muon Optimizer** (5th-order Newton-Schulz polynomial iterations) for 2D weight matrices, matching our training engine.
382
+
383
+ ## 🚀 18. ViuMini-Dense-360M: Migration to 32-Layer Sovereign Dense Architecture (Locked & Verified)
384
+ * **Date**: October 2026
385
+ * **User Directives Enforced**:
386
+ - *"ap khaana chaite o muon ko hatana hai"* — Drop Muon optimizer due to instability on small/narrow dimensions.
387
+ - *"okay check point 1k per push hoan chaiye save checkpoint"* — Lock checkpoint interval to every 1,000 steps.
388
+ - *"agar hum isko dence ban de to kya fyda hoga... isko hum 350m ka deance banye to... agar Moe nad dance mode ki capabilyt chack kro to kon jyda smart hoga... kro and pura change kro is model ko"* — Transition completely from 242M MoE to ~360M Dense.
389
+ - *"ek questin hai agar hum layer ko 32 kr de to"* — Adopt 32-layer deep hierarchical architecture.
390
+ - *"optin a"* — Select Option A (SmolLM-360M blueprint scaled with 32 layers and Indic 48k vocabulary).
391
+ - *"iske doucment update kro and folder and repo name bhi change kro"* — Rename Hugging Face repo to `ViuMini-Dense-360M` and update all documents and configurations.
392
+ * **Root Cause Diagnosis of MoE Divergence**:
393
+ - On Kaggle Dual T4, Muon with `muon_lr: 0.02` was applied to narrow linear matrices (`dim=640`). Because initial standard deviation was 0.02, Muon rotated weight directions by up to 40% of their total norm in a single update step.
394
+ - Gradient norm spiked to 135.125 at step 1,188, routers collapsed to 2 experts, loss exploded to 40.7, and output degenerate into single-token repetitions (`Court Court Court...`).
395
+ * **Architectural Specifications (`Option A: ViuMini-Dense-360M`)**:
396
+ - **Total & Active Parameters**: **366,617,536 (~366.6M)** (100% active per token vs 166M active on 242M MoE).
397
+ - **Transformer Layers**: **32 Deep Dense Layers** (`n_layers: 32`).
398
+ - **Hidden Dimension**: **960** (`dim: 960`).
399
+ - **Attention Heads**: **15 Query Heads** ($d_{\text{head}}=64$) and **5 KV Heads** (Grouped-Query Attention with 3:1 ratio).
400
+ - **Intermediate MLP Dimension**: **2,560** (SwiGLU activation, multiple of 64).
401
+ - **Vocabulary Size**: **48,000 Tokens** (Custom Indic BPE with tied input/output embeddings).
402
+ - **Stability Mechanism**: **Gemma 4 Pure QK-Norm** (`qk_norm: true`), no softcapping.
403
+ - **Optimizer**: **PagedAdamW8bit** (`lr: 2.5e-4`, `min_lr: 2.5e-5`, `warmup_steps: 1000`, `grad_clip: 1.0`, `weight_decay: 0.1`).
404
+ * **Hugging Face Hub Repository Migration**:
405
+ - Executed atomic repo rename: `ViuAI/ViuMini-MoE-242M` $\rightarrow$ **`ViuAI/ViuMini-Dense-360M`**.
406
+ - Verified live on Hugging Face Hub (`https://huggingface.co/ViuAI/ViuMini-Dense-360M`).
407
+ * **Checkpoint & Runbook Policy**:
408
+ - Checkpoint saving and full HF Hub pushing locked to every **1,000 steps** (`save_every: 1000`, `hf_push_full_every: 1000`).
409
+ - Authored session-proof Kaggle runner in `docs/KAGGLE_RUN.md`.
410
  * **Verification & Benchmarks**:
411
+ - Forward & backward pass test verified: clean initial loss **10.8720** (pure base cross-entropy $\ln(48,000)$), `moe_aux: 0.0`, `mt_loss: 0.0`.
412
+ - Smoke pretraining test passed on CPU with exit code 0.
413
+
414
 
415
 
416