| # ROADMAP — Model Scratch se Deployment tak (ViuMini-MoE-242M status ke saath) |
| > Har step ka status: [x] done / [ ] bacha. Evidence file bracket me. |
| > Update rule: step complete hote hi tick + PROGRESS + TEST_REPORT entry + HF push. |
| |
| ## Step 0 — Goal + design lock [x] DONE |
| - Model type, size, language mix fix (Hinglish 40:30:30, 28L MoE 241.7M/166.2M LOCK) |
| - Evidence: `docs/VIU1_MOE_PLAN.md`, `model/configs/viu1_moe_config.yaml` |
| |
| ## Step 1 — Repo + code skeleton [x] DONE |
| - Folder structure, arch code (`viu1_moe.py`), config, requirements, HF card |
| - Evidence: `README.md`, `model/scripts/viu1_moe.py`, HF https://huggingface.co/ViuAI/ViuMini-MoE-242M |
|
|
| ## Step 2 — Audit + bugfix [x] DONE |
| - Sliding-mask, MoE dispatch, aux-loss, batch, ckpt, paths — 10 critical fix verify |
| - Evidence: `docs/TEST_REPORT.md` (5/5 suite PASS) |
|
|
| ## Step 3 — Toy test (Kaggle) [x] DONE (2026-09-17, 0.05 LOCKED) |
| - 2L tiny MoE: A(0.01)→D(0.01/50, max 0.250), B(0.02)→E(0.02/50, max 0.229), C(0.05/50, max 0.199), F(0.10/50, max 0.180), G(0.05/100, max 0.186 + loss 3.82) |
| - Decision: 0.05 lock (beech-ka-rasta; lambe steps balance khud sudharenge). Configs + code default updated. |
| - Evidence: `docs/TEST_REPORT.md` (Step 3 entries), `model/scripts/toy_router_test.py` |
|
|
| ## Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI) |
| - Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across 26 Exhaustive Pillars (NCERT, History, Law, SMT/PCB/FA, QMS, Tech, Emails, Cinema, Global Cooking, Translation, Stories, Songs, Mobile Tech, Jokes & Desi Roasts, Complete Grammar & Vyakaran) |
| - Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested) |
| - Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB) |
| - Evidence: `data/scripts/kaggle_collector.py`, `docs/KAGGLE_DATA_COLLECTION.md`, HF repo `ViuAI/viu-mini-raw-pretrain` |
|
|
| ## Step 5 — FINAL tokenizer [x] DONE (2026-09-17, v1 ACCEPT) |
| - 500k+ lines pe 48k BPE → vocab 47756, fertility 1.83, unk=0. `tokenizer/outputs/tokenizer.json` FINAL (3.5MB, HF main). |
| - Evidence: `docs/TEST_REPORT.md` (Step 5 entries) |
|
|
| ## Step 6 — Dense baseline (GPU) [x] DONE (2026-09-18, rental 3090 24GB) |
| - GPU: 3090 (Kaggle T4 run stopped → rental shift). train.py auto-GPU (amp auto + OOM-probe micro-batch) isi run me verify. |
| - Configs: `model/configs/viu1_dense_config.yaml` (28L MoE-OFF, moe_every 999) + `train_dense_baseline.yaml` (5k steps, seq 512, eff-batch 16) |
| - Data: clean mix 23:39:38 (383502 lines) + FINAL 48k tokenizer |
| - Result: 5000/5000 in 36:45 (~18.5k tok/s), loss 8.42→1.83, val ppl 26.5→10.2, Hinglish sample OK (repetition loops noted) |
| - Success: loss smooth down + Hinglish sample fluent → TEST_REPORT |
| - Evidence: ckpt + val PPL + sample log (TEST_REPORT) |
| |
| ## Step 7 — Full MoE pretrain (GPU 2-4 hafte) [ ] BACHA |
| - 9 MoE layers on, BF16, ckpt har 5k, early-stop (expert >60% for 2k steps → roko) |
| - Success: dense se +5% better + router balanced |
| |
| ## Step 8 — Eval + inference [ ] BACHA |
| - PPL (3 langs), CoT accuracy, human rating + `generate.py` (top-k/temp) |
| - Evidence: eval numbers + samples PROGRESS me |
| |
| ## Step 9 — SFT / alignment (optional) [ ] BACHA |
| - Instruction + DPO data pe finetune (plan me baad me) |
| |
| ## Step 10 — Release [ ] BACHA |
| - FINAL ckpt + model card + public HF + version tag |
| |
| ## Abhi kaha hai? |
| Step 0, 1, 2, 3, 5, 6 **DONE**. Agla: **Step 4 (Data v2 collection — 4-5B tokens)** → **Step 7 (Full MoE Pretraining on GPU)**. |
| |
