sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts

991af26 verified 17 days ago

1.83 kB

Data — NOTES

Mix (OPTIONAL rule, default OFF): 40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English. Training me model/configs/train_config.yaml (enforce_mix: false default) ya tokenizer me --mix/--no-mix se control hota hai. Weights badal sakte ho, force nahi hai.

Structure

data/
  raw/ (hinglish.txt, hindi.txt, english.txt)
  processed/ (tokenized bin future)
  eval/sample_test.txt (10 lines coverage set)
  scripts/ (cleaning/dedup future)

Log

2026-09-15: sample raw data banaya (15 Hinglish + 10 Hindi + 10 English lines) — sirf tokenizer test ke liye
2026-09-18: scripts/clean_dedup.py upgraded to streaming line-by-line generator + 64-bit integer hash dedup (RAM-safe on 4-5B tokens), English Latin-script filtering added. Test pass.
2026-09-18: Step 4 upgraded to 30B–50B+ Tokens (Zero Sub-Sampling / Full Ingestion across 20 Knowledge Pillars). Built data/scripts/collect_domains.py (20-pillar knowledge generator) and data/scripts/kaggle_collector.py (Kaggle cloud worker streaming directly to ViuAI/viu-mini-raw-pretrain with chunk-upload-delete loop). Runbook documented in docs/KAGGLE_DATA_COLLECTION.md. Dry-run PASS.
2026-09-22: HF audit (621 files/157GB): hindi 128 shards missing, english/hindi-big oversized, schema drift, duplicates. Fix: collector shard 150MB + stable blake2b hash + resume fix; new standardize_to_unified.py, filter_by_safety.py (uncensored/toxic tagged, not deleted - user decision), repair_backfill_hindi.py, repair_reshard_large.py, kaggle_repair_all.py. Full runbook docs/HF_REPAIR_RUNBOOK.md. HF upload pending (Kaggle).
2026-09-22 (night): DECISION — data repair LOCAL NAHI, 100% KAGGLE par. Local sirf scripts/docs. Execution log docs/PROGRESS.md (Kaggle repair live log K1-K6) me har step ke saath update hoga.