docs/ROADMAP_SCRATCH.md

# ROADMAP — Model Scratch se Deployment tak (ViuMini-MoE-242M status ke saath)
> Har step ka status: [x] done / [ ] bacha. Evidence file bracket me.
> Update rule: step complete hote hi tick + PROGRESS + TEST_REPORT entry + HF push.

## Step 0 — Goal + design lock [x] DONE
- Model type, size, language mix fix (Hinglish 40:30:30, 28L MoE 241.7M/166.2M LOCK)
- Evidence: `docs/VIU1_MOE_PLAN.md`, `model/configs/viu1_moe_config.yaml`

## Step 1 — Repo + code skeleton [x] DONE
- Folder structure, arch code (`viu1_moe.py`), config, requirements, HF card
- Evidence: `README.md`, `model/scripts/viu1_moe.py`, HF https://huggingface.co/ViuAI/ViuMini-MoE-242M

## Step 2 — Audit + bugfix [x] DONE
- Sliding-mask, MoE dispatch, aux-loss, batch, ckpt, paths — 10 critical fix verify
- Evidence: `docs/TEST_REPORT.md` (5/5 suite PASS)

## Step 3 — Toy test (Kaggle) [x] DONE (2026-09-17, 0.05 LOCKED)
- 2L tiny MoE: A(0.01)→D(0.01/50, max 0.250), B(0.02)→E(0.02/50, max 0.229), C(0.05/50, max 0.199), F(0.10/50, max 0.180), G(0.05/100, max 0.186 + loss 3.82)
- Decision: 0.05 lock (beech-ka-rasta; lambe steps balance khud sudharenge). Configs + code default updated.
- Evidence: `docs/TEST_REPORT.md` (Step 3 entries), `model/scripts/toy_router_test.py`

## Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI)
- Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across 26 Exhaustive Pillars (NCERT, History, Law, SMT/PCB/FA, QMS, Tech, Emails, Cinema, Global Cooking, Translation, Stories, Songs, Mobile Tech, Jokes & Desi Roasts, Complete Grammar & Vyakaran)
- Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested)
- Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB)
- Evidence: `data/scripts/kaggle_collector.py`, `docs/KAGGLE_DATA_COLLECTION.md`, HF repo `ViuAI/viu-mini-raw-pretrain`

## Step 5 — FINAL tokenizer [x] DONE (2026-09-17, v1 ACCEPT)
- 500k+ lines pe 48k BPE → vocab 47756, fertility 1.83, unk=0. `tokenizer/outputs/tokenizer.json` FINAL (3.5MB, HF main).
- Evidence: `docs/TEST_REPORT.md` (Step 5 entries)

## Step 6 — Dense baseline (GPU) [x] DONE (2026-09-18, rental 3090 24GB)
- GPU: 3090 (Kaggle T4 run stopped → rental shift). train.py auto-GPU (amp auto + OOM-probe micro-batch) isi run me verify.
- Configs: `model/configs/viu1_dense_config.yaml` (28L MoE-OFF, moe_every 999) + `train_dense_baseline.yaml` (5k steps, seq 512, eff-batch 16)
- Data: clean mix 23:39:38 (383502 lines) + FINAL 48k tokenizer
- Result: 5000/5000 in 36:45 (~18.5k tok/s), loss 8.42→1.83, val ppl 26.5→10.2, Hinglish sample OK (repetition loops noted)
- Success: loss smooth down + Hinglish sample fluent → TEST_REPORT
- Evidence: ckpt + val PPL + sample log (TEST_REPORT)

## Step 7 — Full MoE pretrain (GPU 2-4 hafte) [ ] BACHA
- 9 MoE layers on, BF16, ckpt har 5k, early-stop (expert >60% for 2k steps → roko)
- Success: dense se +5% better + router balanced

## Step 8 — Eval + inference [ ] BACHA
- PPL (3 langs), CoT accuracy, human rating + `generate.py` (top-k/temp)
- Evidence: eval numbers + samples PROGRESS me

## Step 9 — SFT / alignment (optional) [ ] BACHA
- Instruction + DPO data pe finetune (plan me baad me)

## Step 10 — Release [ ] BACHA
- FINAL ckpt + model card + public HF + version tag

## Abhi kaha hai?
Step 0, 1, 2, 3, 5, 6 **DONE**. Agla: **Step 4 (Data v2 collection — 4-5B tokens)** → **Step 7 (Full MoE Pretraining on GPU)**.