docs/ROADMAP_SCRATCH.md

ROADMAP — Model Scratch se Deployment tak (ViuMini-MoE-242M status ke saath)

Har step ka status: [x] done / [ ] bacha. Evidence file bracket me. Update rule: step complete hote hi tick + PROGRESS + TEST_REPORT entry + HF push.

Step 0 — Goal + design lock [x] DONE

• Model type, size, language mix fix (Hinglish 40:30:30, 28L MoE 241.7M/166.2M LOCK)
• Evidence: docs/VIU1_MOE_PLAN.md, model/configs/viu1_moe_config.yaml

Step 1 — Repo + code skeleton [x] DONE

• Folder structure, arch code (viu1_moe.py), config, requirements, HF card
• Evidence: README.md, model/scripts/viu1_moe.py, HF https://huggingface.co/ViuAI/ViuMini-MoE-242M

Step 2 — Audit + bugfix [x] DONE

• Sliding-mask, MoE dispatch, aux-loss, batch, ckpt, paths — 10 critical fix verify
• Evidence: docs/TEST_REPORT.md (5/5 suite PASS)

Step 3 — Toy test (Kaggle) [x] DONE (2026-09-17, 0.05 LOCKED)

• 2L tiny MoE: A(0.01)→D(0.01/50, max 0.250), B(0.02)→E(0.02/50, max 0.229), C(0.05/50, max 0.199), F(0.10/50, max 0.180), G(0.05/100, max 0.186 + loss 3.82)
• Decision: 0.05 lock (beech-ka-rasta; lambe steps balance khud sudharenge). Configs + code default updated.
• Evidence: docs/TEST_REPORT.md (Step 3 entries), model/scripts/toy_router_test.py

Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI)

• Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across 26 Exhaustive Pillars (NCERT, History, Law, SMT/PCB/FA, QMS, Tech, Emails, Cinema, Global Cooking, Translation, Stories, Songs, Mobile Tech, Jokes & Desi Roasts, Complete Grammar & Vyakaran)
• Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested)
• Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB)
• Evidence: data/scripts/kaggle_collector.py, docs/KAGGLE_DATA_COLLECTION.md, HF repo ViuAI/viu-mini-raw-pretrain

Step 5 — FINAL tokenizer [x] DONE (2026-09-17, v1 ACCEPT)

• 500k+ lines pe 48k BPE → vocab 47756, fertility 1.83, unk=0. tokenizer/outputs/tokenizer.json FINAL (3.5MB, HF main).
• Evidence: docs/TEST_REPORT.md (Step 5 entries)

Step 6 — Dense baseline (GPU) [x] DONE (2026-09-18, rental 3090 24GB)

• GPU: 3090 (Kaggle T4 run stopped → rental shift). train.py auto-GPU (amp auto + OOM-probe micro-batch) isi run me verify.
• Configs: model/configs/viu1_dense_config.yaml (28L MoE-OFF, moe_every 999) + train_dense_baseline.yaml (5k steps, seq 512, eff-batch 16)
• Data: clean mix 23:39:38 (383502 lines) + FINAL 48k tokenizer
• Result: 5000/5000 in 36:45 (~18.5k tok/s), loss 8.42→1.83, val ppl 26.5→10.2, Hinglish sample OK (repetition loops noted)
• Success: loss smooth down + Hinglish sample fluent → TEST_REPORT
• Evidence: ckpt + val PPL + sample log (TEST_REPORT)

Step 7 — Full MoE pretrain (GPU 2-4 hafte) [ ] BACHA

• 9 MoE layers on, BF16, ckpt har 5k, early-stop (expert >60% for 2k steps → roko)
• Success: dense se +5% better + router balanced

Step 8 — Eval + inference [ ] BACHA

• PPL (3 langs), CoT accuracy, human rating + generate.py (top-k/temp)
• Evidence: eval numbers + samples PROGRESS me

Step 9 — SFT / alignment (optional) [ ] BACHA

• Instruction + DPO data pe finetune (plan me baad me)

Step 10 — Release [ ] BACHA

• FINAL ckpt + model card + public HF + version tag

Abhi kaha hai?

Step 0, 1, 2, 3, 5, 6 DONE. Agla: Step 4 (Data v2 collection — 4-5B tokens) → Step 7 (Full MoE Pretraining on GPU).