step4: clean PASS+WARNING, hinglish gap

docs/PROGRESS.md

@@ -45,6 +45,7 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
 - [x] 2026-09-17: `docs/DATA_SOURCES.md` FINAL (IDs verified: romanized_hindi + IndicCorpV2-hin_Deva + TinyStories, 500k plan + Kaggle streaming cell), HF push — data pull Kaggle pending
 - [x] 2026-09-17: Kaggle pull PARTIAL (hinglish 199993 OK / hindi 75000 aadha / english 149972 OK; column `Transliterated Hindi` fix + None-guard fix), `data/scripts/clean_dedup.py` banaya (script-check + dedup + eval-split), HF push — Hindi top-up + clean run pending
 - [x] 2026-09-17: Hindi top-up done (+50k +25k = 150000) — PLAN TALLY: hinglish 199993/200000 + hindi 150000/150000 + english 149972/150000 ≈ 500k lines ON-PLAN; policy me "har kaam plan ke according" line jodi, HF push — clean run pending
+- [x] 2026-09-17: CLEAN Kaggle PASS+WARNING (hinglish train sirf 44569 — short 68k + dup 85k; hindi 148561 + english 78192 OK; mix 18:60:32 vs plan 40:30:30), TEST_REPORT entry, HF push — Hinglish top-up approval pending
 - [x] 2026-09-17: `docs/ROADMAP_SCRATCH.md` banaya (Step 0-10, status: 0-2 done, agla Step 3 toy-Kaggle), HF push
 - [ ] TODO: toy router-balance test (Kaggle), FINAL tokenizer (500k lines wait), dense baseline
 

docs/TEST_REPORT.md

@@ -97,6 +97,17 @@ Har test ke baad yaha entry hogi: date, env, command, result, verdict.
 - Changed: `model/configs/viu1_moe_config.yaml` + `model/configs/config.json` + `ViuArgs` default 0.01→0.05
 - Step 3 status: CLOSED. Agla: Step 4 data.
 
+## 2026-09-17 — Step 4 CLEAN (Kaggle, PASS with WARNING)
+- Env: Kaggle notebook, CPU
+- Cmd: `python data/scripts/clean_dedup.py` (seed 42, eval 1000/lang)
+- Result:
+  - hinglish: raw 200228 → short 68678 + script 741 + dup 85240 → **train 44569** eval 1000 avg_words 9
+  - hindi: raw 150868 → short 427 + dup 12 → **train 148561** eval 1000 avg_words 57
+  - english: raw 143742 → short 5325 + dup 16089 → **train 78192** eval 1000 avg_words 33
+- WARNING (plan se bahar): Hinglish sirf 44.5k bacha (target 2L ka ~22%) — source chhoti + duplicate lines bhari hain. Train-mix ab ~18:60:32, plan 40:30:30 nahi raha. Tokenizer ko 500k lines chahiye, abhi ~271k train lines hain.
+- Verdict: PASS (script sahi chala) + data-gap OPEN — Hinglish top-up (subtitles/AI-made casual) approval pending
+- Screenshot verified (user-provided)
+
 ## Template (next test ke liye)
 ```
 ## YYYY-MM-DD — <test naam> (<env>)