| # Data — NOTES | |
| Mix (OPTIONAL rule, default OFF): 40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English. | |
| Training me `model/configs/train_config.yaml` (`enforce_mix: false` default) ya tokenizer me `--mix/--no-mix` se control hota hai. Weights badal sakte ho, force nahi hai. | |
| ## Structure | |
| ``` | |
| data/ | |
| raw/ (hinglish.txt, hindi.txt, english.txt) | |
| processed/ (tokenized bin future) | |
| eval/sample_test.txt (10 lines coverage set) | |
| scripts/ (cleaning/dedup future) | |
| ``` | |
| ## Log | |
| - 2026-09-15: sample raw data banaya (15 Hinglish + 10 Hindi + 10 English lines) — sirf tokenizer test ke liye | |
| - 2026-09-18: `scripts/clean_dedup.py` upgraded to streaming line-by-line generator + 64-bit integer hash dedup (RAM-safe on 4-5B tokens), English Latin-script filtering added. Test pass. | |
| - 2026-09-18: Step 4 upgraded to 30B–50B+ Tokens (Zero Sub-Sampling / Full Ingestion across 20 Knowledge Pillars). Built `data/scripts/collect_domains.py` (20-pillar knowledge generator) and `data/scripts/kaggle_collector.py` (Kaggle cloud worker streaming directly to `ViuAI/viu-mini-raw-pretrain` with chunk-upload-delete loop). Runbook documented in `docs/KAGGLE_DATA_COLLECTION.md`. Dry-run PASS. | |
| - 2026-09-22: HF audit (621 files/157GB): hindi 128 shards missing, english/hindi-big oversized, schema drift, duplicates. Fix: collector shard 150MB + stable blake2b hash + resume fix; new `standardize_to_unified.py`, `filter_by_safety.py` (uncensored/toxic tagged, not deleted - user decision), `repair_backfill_hindi.py`, `repair_reshard_large.py`, `kaggle_repair_all.py`. Full runbook `docs/HF_REPAIR_RUNBOOK.md`. HF upload pending (Kaggle). | |
| - 2026-09-22 (night): DECISION — data repair LOCAL NAHI, 100% KAGGLE par. Local sirf scripts/docs. Execution log `docs/PROGRESS.md` (Kaggle repair live log K1-K6) me har step ke saath update hoga. | |