Sync NOTES.md with massive Mythology and Bollywood Cinema corpora expansion

NOTES.md

@@ -0,0 +1,362 @@
+# Viu-1.5B-MoE: Technical Architecture & Development Notes
+
+Comprehensive engineering and strategic reference for the **Viu-1.5B-MoE** sovereign Indian language model.
+
+> [!IMPORTANT]
+> ### 🛡️ Golden Development Protocol (Mandatory Workflow)
+> 1. **Pre-Execution Preparation**: Kisi bhi change ya task ko shuru karne se pehle uski file, plan aur structure ko fully define aur ready karna hai.
+> 2. **Execution & Verification**: Implementation ke baad strict verification (audit / smoke test) execute karna hai.
+> 3. **Immediate Note Update**: Jo bhi kaam hua, uski exact details, numbers aur results ko turant is `NOTES.md` aur `docs/PROGRESS.md` me record aur sync karna hai.
+
+---
+
+## 📌 1. Project Overview & Philosophy
+
+* **Model Name**: `Viu-1.5B-MoE` (Scaled to 48 Layers)
+* **Scale**: **1,855,544,064 Total Parameters (~1.856 Billion with MTP / ~1.819 Billion Base)**
+* **Compute Footprint**: **300,473,088 Active Parameters (~300.5 Million per token)**
+* **Sparsity**: **16.19% active per token** (high-speed on-device inference + 1.85B capacity).
+* **Core Philosophy**: Indigenous frontier transformer incorporating cutting-edge architecture advances:
+  - **48 Ultra-Deep Layers**: Maximum sequential reasoning depth across 4 abstraction tiers (12 layers each).
+  - **Multi-Head Latent Attention (MLA)** (DeepSeek-V3/V4 style low-rank KV compression & decoupled RoPE).
+  - **1 Permanent Shared Expert + 28 Routed Micro-Experts** across **48 Layers** (1,392 Total Experts).
+  - **Logit Soft-Capping** (Gemma-2 style $\tanh$-capping at 50.0 for attention and 30.0 for logits).
+  - **Multi-Token Prediction (MTP)** for forward planning.
+
+---
+
+## 🧠 2. 48-Layer Ultra-Deep Frontier Architecture
+
+```
+[Tokens] -> Embeddings (48k x 768 = 36.86M)
+    ↓
+Tier 1: Layers 1–12  (348 Experts) -> Subwords, Devanagari script, phonetics & transliteration
+    ↓
+Tier 2: Layers 13–24 (348 Experts) -> Multilingual grammar, Hindi <-> Hinglish <-> English alignment
+    ↓
+Tier 3: Layers 25–36 (348 Experts) -> Indian cultural facts, world knowledge, Python/code syntax
+    ↓
+Tier 4: Layers 37–48 (348 Experts) -> Multi-step logic, math deduction, intent synthesis
+    ↓
+[Output] -> Tied LM Head (30.0 softcap) + 1-Head Multi-Token Prediction (MTP)
+```
+
+### Detailed Layer Parameters:
+* **Hidden Dimension**: 768 ($d_{\text{head}}=64$)
+* **Attention Mechanism**: Multi-Head Latent Attention (MLA)
+  * Query low-rank rank: $c_Q = 256$
+  * Key/Value low-rank rank: $c_{KV} = 256$
+  * Decoupled RoPE dimension: $d_{\text{rope}} = 64$
+  * Attention Heads: 12 Latent Heads
+  * Dual QK-Norm: RMSNorm on $q$ and $k$
+  * Attention Logit Soft-Capping: $50.0 \times \tanh(S / 50.0)$
+  * Hybrid Context: 32 Sliding Window (512 tokens) + 16 Full Causal Attention layers (hybrid ratio 3)
+* **RoPE Base Theta**: 500,000.0 (YaRN ready)
+* **MoE Mixture of Experts**:
+  * 1 Permanent Shared Expert (Universal linguistic syntax & idioms)
+  * 28 Fine-Grained Routed Experts (Specialized cognitive domains, Top-2 routing)
+  * Total Active Experts per layer: 3 experts ($1 \text{ shared} + 2 \text{ routed}$)
+  * Total Experts in network: $48 \times (1 + 28) = \mathbf{1,392\text{ Micro-Experts}}$
+* **Output Logit Soft-Capping**: $30.0 \times \tanh(\text{logits} / 30.0)$
+* **Multi-Token Prediction (MTP)**: 1-Head MTP auxiliary loss with weight 0.1.
+
+---
+
+## 📚 3. Pretraining Dataset
+
+* **Repository**: `ViuAI/viu-mini-raw-pretrain` (Hugging Face Datasets)
+* **Dataset Volume**: **105+ Billion (10,500 Crore) Tokens** across 1,751 unified Parquet files
+* **Language Composition**:
+  * Hinglish: ~78% (Colloquial chat, Bollywood references, social dialogues)
+  * Pure Hindi: ~8% (Devanagari literature, news, formal texts)
+  * English: ~14% (Web knowledge, encyclopedia, science, code)
+* **Ingestion Method**: Native PyArrow streaming over HTTPS directly to GPU memory (Zero disk space required).
+* **In-Flight Blake2b Deduplication Engine**:
+  * Employs 64-bit Blake2b stable integer hashing (`digest_size=8`) on normalized text to prevent loss landscape poisoning, representation collapse, and memorization artifacts.
+  * Audited 2,024,644 candidate rows: **22,268 duplicates dropped** across full-scale ingestion runs.
+  * Exported **2,002,376 UNIQUE rows (778.64 MB ZSTD Parquet, ~750M+ frontier tokens)** into `domains/` across 32 shards:
+    - `finance_tax_full`: 10 shards, **695,434 unique rows (185.62 MB)** — Full Finance Instruct 500k, Sujet Finance 177k, and Financial QA 10k.
+    - `healthcare_clinical_full`: 8 shards, **596,967 unique rows (232.08 MB)** — Full ChatDoctor HealthcareMagic (112k), Medical QA (239k), and AI Medical Dialogues (256k).
+    - `orca_math_full`: 3 shards, **199,468 unique rows (53.09 MB)** — Full Orca Math step-by-step problem solving.
+    - `court_judgments_full`: 3 shards, **183,495 unique rows (169.87 MB)** — Full Indian High Court & Supreme Court case judgments.
+    - `governance_and_law`: 1 shard, **41,987 unique rows (37.45 MB)** — BNS 2023, BNSS 2023, BSA 2023, and Constitution of India.
+    - `literature_and_culture`: 1 shard, **64,609 unique rows (17.24 MB)** — Hindi/Urdu poetry & prose, Tyohar traditions, narrative storytelling.
+    - `deepseek_r1_math`: 1 shard, **50,000 unique rows (21.17 MB)** — DeepSeek-R1 step-by-step reasoning proofs.
+    - `sanatan_heritage`: 1 shard, **37,338 unique rows (7.97 MB)** — Bhagavad Gita, Ramayana, Upanishads, and Indian philosophy records.
+    - `python_code`: 1 shard, **18,612 unique rows (3.74 MB)** — Python code instructions, algorithms, and data structures.
+    - `gsm8k_math`: 1 shard, **7,473 unique rows (1.55 MB)** — GSM8K step-by-step arithmetic word problems.
+* **Native Pretraining Streaming Engine (`train.py`) Unlocks**:
+  * Unlocked `legal/` (**33 shards, 8.48 GB, 6,055,372 Indian High Court & Supreme Court case determinations**) natively via `context/question/response` streaming adapter.
+  * Unlocked `translation/` (**21 shards, 2.03 GB, ~26,000,000 / 2.6 Crore parallel English <-> Hindi sentences** from AI4Bharat Samanantar & Opus) with alternating bidirectional streaming.
+  * Unlocked `distilled/`: **33 shards (3,093,000 rows, 3.82 GB, ~1.5B tokens of DeepSeek-R1 Math Reasoning)** streaming natively.
+  * Unlocked `science/`: **35 shards (3,011,679 rows, 11.58 GB of Science Textbooks & SciQ)** with `generated_full_text` fallback.
+  * Unlocked `wikipedia/`: **41 shards (10.83 GB, ~60 Lakh articles)** in pure Hindi & English.
+  * Unlocked `hinglish/`: **6 shards (~3.4 Lakh rows)** of Alpaca, Dolly, and English-Hinglish Top.
+  * Unlocked `math/`: **4 shards** of GSM8K, GSM8K Socratic, and Orca Math.
+  * Total pretraining dataset files discovered by streaming loader: **1,874 Parquet files (Over 202 GB streamable)**.
+  * Total combined high-impact training volume active: **Over 40+ Million records (~6.5+ Billion tokens)** across all domains.
+* **Frontier Knowledge Domains (Integrated & Active)**:
+  1. *Reasoning & Code*: DeepSeek-R1 math, Orca math, Python algorithms, SQL, GSM8K arithmetic, and `<soch>...</soch>` thinking traces.
+  2. *Parallel Translation*: AI4Bharat Samanantar 26 Million bidirectional English $\leftrightarrow$ Hindi pairs.
+  3. *Bhartiya Dharohar & Culture*: Shreemad Bhagavad Gita, Ramayana, Upanishads, Sant Kabir dohe, Urdu classical poetry (Ghalib, Faiz, Rahat Indori), Panchatantra & Birbal.
+  4. *Indian Governance & Law*: Constitution of India, BNS/BNSS/BSA new criminal codes, landmark Supreme Court judgements, central welfare schemes (PM-Kisan, Ayushman Bharat, India Stack).
+  5. *Indian Finance, Tax & Healthcare*: Income Tax (Old vs New Regime, 80C), GST architecture, Mutual Funds/SIP compounding math, First Aid, Traditional Ayurveda (Tridosha & Dinacharya), and full physician consultations.
+  *(Note: Multilingual Indic languages excluded per architectural directive to concentrate parameter density on Hindi, Hinglish, and English).*
+
+---
+
+## 🖥️ 4. Hardware & Memory Audit
+
+| Hardware | VRAM | Configuration | Optimizer | VRAM Peak | Safety Headroom |
+| :--- | :---: | :--- | :--- | :---: | :---: |
+| **RTX 5090** | **32 GB** | `batch: 4, accum: 32` | `Muon` + `PagedAdamW8bit` | **~23.0 GB** | **~9.0 GB Free** (Rock Solid) |
+| **RTX 4090** | **24 GB** | `batch: 4, accum: 32` | `Muon` + `PagedAdamW8bit` + `grad_ckpt: true` | **~17.5 GB** | **~6.5 GB Free** (Safe) |
+
+* **Optimizer Breakthrough (Muon)**: 
+  * 2D weight matrices (MLA attention & MoE micro-experts) are updated via **Newton-Schulz quintic iteration orthogonalization**.
+  * Eliminates second moment ($v$) buffers on all matrix weights, reducing optimizer VRAM overhead.
+  * Yields **~1.5x faster token convergence** than standard AdamW (800k steps equivalent to 1.2M steps of AdamW).
+  * **Decoupled Weight Decay**: Muon operates at $lr=0.02$ with dedicated `muon_weight_decay: 0.01` (preventing over-regularization and weight shrinkage), while 1D embeddings/norms/routers use AdamW with `weight_decay: 0.1` ($lr=3\times 10^{-4}$).
+* **Global Effective Batch**: $4 \times 32 = \mathbf{128\text{ sequences}}$ = **131,072 tokens/step**
+* **Tokens to Steps**: $105\text{B} \div 131,072 \approx \mathbf{800,000\text{ steps}}$
+
+---
+
+## 📂 5. File Inventory
+
+* `NOTES.md` — This comprehensive reference file
+* `docs/PROGRESS.md` — Milestone log and timestamped audit history
+* `docs/VIU_ARCHITECTURE_SPEC.md` — Formal architecture mathematical specification (48 Layers, 1,392 Experts)
+* `model/configs/model_config.yaml` — 48-layer 1.856B frontier architecture definition
+* `model/configs/train_rtx4090.yaml` — RTX 4090 training profile
+* `model/configs/train_rtx5090.yaml` — RTX 5090 training profile
+* `model/scripts/viu_moe.py` — Frontier MLA + 1,392 Micro-Experts PyTorch neural network
+* `model/scripts/train.py` — Pretraining engine with 8-bit AdamW, Muon optimizer, round-robin domain interleaving, and PyArrow streaming
+* `model/scripts/verify_arch.py` — Architecture verification suite
+* `data/scripts/generate_industrial_enterprise_corpus.py` — Industrial, Enterprise, Office, Excel, Email, QMS (ISO/IATF), SMT & Electronics Knowledge Corpus generator (3 shards uploaded)
+* `data/scripts/ingest_frontier_conversational_master.py` — Conversational & Uncensored Master Booster (Samvaad, UltraChat, Evol-Hindi, Dolphin 2.9.4, Hinglish Chat)
+* `data/scripts/ingest_remaining_99_raw_files.py` — In-flight Blake2b streaming ingest engine for remaining 28 GB raw data
+
+---
+
+## 🏭 6. Industrial, Enterprise, Office, Quality & SMT Intelligence Corpus
+
+Directly ingested into `domains/` in `ViuAI/viu-mini-raw-pretrain`:
+1. **Microsoft Excel & Office Mastery** (`train-frontier_deduped_industrial_enterprise-*.parquet`):
+   - Comprehensive formula mechanics: `XLOOKUP`, `INDEX/MATCH`, `FILTER`, `UNIQUE`, `LET`, `LAMBDA`, `SUMIFS`, `COUNTIFS`, `TEXTJOIN`, `XIRR`, `XNPV`, `VSTACK/HSTACK`.
+   - Advanced VBA automation routines: Multi-sheet consolidation, automated email dispatch via Outlook, ADO SQL database queries, UserForm validation.
+   - Power Query M-Code & DAX: ETL pipelines, time-intelligence measures (`CALCULATE`, `SAMEPERIODLASTYEAR`, `TOTALYTD`).
+   - Bilingual workplace guides in Hindi and Hinglish for Indian office operations.
+2. **All Corporate & Professional Email Types**:
+   - Ingested **10,000 real corporate emails** from `Yale-LILY/aeslc` (Enron corporate corpus).
+   - High-stakes curated templates: P0 executive outage escalations, formal resignation with knowledge transfer plans, vendor price negotiation & procurement, Performance Improvement Plans (PIP), Hinglish project sync & sprint reviews.
+3. **Quality Engineering, QMS, ISO 9001 & IATF 16949**:
+   - **ISO 9001:2015**: 10-clause Annex SL structure, PDCA cycle, risk-based thinking, internal audit frameworks.
+   - **IATF 16949:2016**: Automotive-specific clauses (Product Safety 4.4.1.2, Contingency Plans 6.1.2.3, Control Plans 8.5.1.1, TPM 8.5.1.5, VDA 6.3 audits).
+   - **AIAG Core Tools**:
+     - **APQP**: 5 phases from concept to production ramp-up.
+     - **PPAP**: All 18 submission elements, 5 submission levels, PSW sign-off criteria.
+     - **FMEA**: AIAG-VDA harmonized 7-step method, Action Priority (AP: High, Medium, Low) logic tables.
+     - **MSA**: Gauge R&R (%GRR < 10% acceptable, 10-30% marginal, >30% reject), NDC >= 5, Repeatability (EV) & Reproducibility (AV).
+     - **SPC**: Variable charts (Xbar-R, Xbar-S, I-MR), Attribute charts (p, np, c, u), 8 Nelson out-of-control rules, Process capability formulas ($C_p, C_{pk}, P_p, P_{pk}$).
+   - **Problem Solving & Lean Tools**: Complete 8D case study (automotive SMT short circuit containment & 5-Why root cause), 5S (Seiri, Seiton, Seiso, Seiketsu, Shitsuke), Kaizen, Poka-Yoke (Mistake Proofing), and SMED changeovers.
+4. **SMT Assembly Line, PCB Engineering & Electronic Components**:
+   - **SMT Process Engineering**: Stencil printing (squeegee angles, pressure, speed, aperture area ratio >= 0.66 per IPC-7525), solder paste SAC305 rheology and shelf life, 3D SPI (height, volume, area, coplanarity), high-speed pick & place (feeders, optical alignment, CPH), 10-zone reflow convection profiling (preheat, soak, TAL > 217°C, peak temp 235-245°C, cooling rate 3-4.5°C/s for $Cu_6Sn_5$ IMC), AOI and AXI X-ray inspection.
+   - **IPC Standards**: IPC-A-610 Class 1, 2, and 3 solder joint acceptability criteria (heel fillet, side fillet, wetting angle < 90°), IPC J-STD-001, IPC-7095 (BGA voiding < 15% for Class 3).
+   - **SMT Defect RCA**: Tombstoning (Manhattan effect), solder bridging, voiding in BGAs/QFN thermal pads, Head-in-Pillow (HiP), solder balls & beading.
+   - **Electronic Components Physics**: Passive (MLCC C0G vs X7R DC-bias derating, power inductors $I_{sat}$ vs $I_{rms}$, resistors), Active (Power MOSFETs $R_{ds(on)}, Q_g$, TVS diodes for ESD/Surge, IGBTs), IC packages (QFP, QFN, BGA), MSL moisture sensitivity levels 1-6 per J-STD-020, ANSI/ESD S20.20 electrostatic discharge control.
+
+---
+
+## 💬 7. Conversational, Reasoning & Uncensored Mastery Corpus
+
+Directly ingested into `domains/` in `ViuAI/viu-mini-raw-pretrain`:
+1. **Sarvam AI Samvaad (`conversational_samvaad`)**:
+   - **3 Shards (189.71 MB, 101,424 unique Indian multi-turn dialogues)** in pure Hindi & natural Hinglish.
+2. **HuggingFaceH4 UltraChat 200k (`conversational_ultrachat`)**:
+   - **8 Shards (664.96 MB, 307,428 unique human-AI dialogues)** covering empathetic dialogue, multi-turn follow-ups, and open-domain discussion.
+3. **Freedom Intelligence Evol-Instruct Hindi (`hindi_evol_reasoning`)**:
+   - **2 Shards (83.52 MB, 58,986 unique complex reasoning dialogues)** covering advanced STEM, logic, and coding in pure Devanagari Hindi.
+4. **Everyday Conversational Hinglish (`hinglish_conversational_boost`)**:
+   - **1 Shard (6.72 MB, 10,507 unique dialogues)** from Chatbot Arena & CrossChat.
+5. **Dolphin 2.9.4 Uncensored & Zero-Refusal (`dolphin_uncensored`)**:
+   - **8 Shards (578.16 MB, 376,899 unique multi-turn dialogues)** covering zero-refusal obedience, critical thinking, unfiltered science, and coding assistance.
+6. **Total Active Footprint on Hub**:
+   - **2,073 Active Streamable Parquet Shards (~212.65 GB compressed, ~231.8 Million rows, ~116 to 126 Billion tokens)** across all 15 high-impact categories.
+   - **Toxicity & Offensive Language Integrated**: Restored `toxicity/` (Civil Comments 1.80M rows + Hate Speech 24.7k tweets via `'tweet'` adapter) so the model comprehends unfiltered human slang, street colloquialisms, and abusive language in pretraining.
+   - **Zero Synthetic / Fake Data**: All synthetic stubs permanently deleted; real Lean Six Sigma Q&A (`cw18`) and verified technical standards active under `domains/`.
+   - **Continuous Dynamic Streaming**: Implemented `ds.refill()` in `train.py` for continuous streaming through all 2,073 files without repetitive overfitting.
+   - **Bypassed Metadata/Empty**: Only metadata index (`code/python_edu*`) and 0-row `sharegpt` bypassed.
+
+---
+
+## 🚀 8. Pretraining Launch Commands (Ready Today)
+
+### 1. Architecture Verification Suite
+```bash
+python model/scripts/verify_arch.py
+```
+
+### 2. Fast CPU Pipeline Smoke Test (3 Steps)
+```bash
+python model/scripts/train.py --smoke
+```
+
+### 3. Production Launch on Cloud GPU (RunPod / Vast.ai / Lambda / Kaggle)
+Using `RUN_ON_CLOUD.ipynb` or via terminal:
+
+- **For RTX 5090 (32GB) / A100 (40GB/80GB) / H100 (80GB)**:
+```bash
+python model/scripts/train.py --config model/configs/train_rtx5090.yaml
+```
+
+- **For RTX 4090 (24GB) / L4 (24GB)**:
+```bash
+python model/scripts/train.py --config model/configs/train_rtx4090.yaml
+```
+
+- **Multi-GPU DDP (e.g., 2x or 4x RTX 4090 / 5090)**:
+```bash
+torchrun --nproc_per_node=2 model/scripts/train.py --config model/configs/train_rtx4090.yaml --dist
+```
+
+## 📌 7. Streaming Adapters & Enforced Mix (2026-09-29 audit fixes)
+* Toxicity re-included with label-preserving tags (civil score-to-tag >= 0.5); vortex + math-QA adapters live-verified on Hub.
+* `enforce_mix: true`, quotas `hinglish 0.15 / hindi 0.35 / english 0.5` (counters ~90% English token dominance; graceful degrade).
+* Leftover purge staged at `data/scripts/hub_purge_leftovers_DRYRUN.py` (40 files; needs `--execute` + `HF_TOKEN`).
+* Shared cross-run hash-DB still open (all ingest scripts use per-run memory sets).
+
+## 🛡️ 8. 100% Zero-Exclusion Architecture (Locked & Verified)
+* **Zero Exclusions**: In response to explicit user directive (*"koi file and koi bhi data ko exclude nhi krna bro"*), `NON_UNIFIED_PREFIXES = ()`. Exactly **2,081 of 2,081 Parquet shards (100.0%)** across all 18 categories on Hugging Face (`ViuAI/viu-mini-raw-pretrain`) are actively streaming into pretraining.
+* **Universal Multi-Schema Adapters**:
+  - `instruction` + `output` -> Instruction following (`uncensored/vortex` 8.55M rows, `code/python_code_instructions` 18.6k rows).
+  - `tweet` -> Real social & offensive language (`toxicity/hate_speech_offensive` 24.8k rows).
+  - `question` + `answer` -> Standalone QA pairs (`finance/financial_qa` 7k rows).
+  - `Patient` + `Doctor` -> Healthcare clinical dialogues (`health/ai_medical_dialogues` 256.9k rows).
+  - `context` + `question` + `response` -> Indian court case law (6.05M rows).
+  - `src` + `tgt` -> Samanantar bilingual translation (26M pairs).
+  - `generated_full_text` -> Science textbooks (3.01M rows).
+  - `comment_text` + scores -> Civil comments discussion (1.80M rows).
+* **Live Smoke Verification**: Passed CPU smoke tests on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M` with exit code 0.
+
+## 🏛️ 9. 100% Real Indian History & PIB Corpus (Zero Fake Data)
+* **User Directive Enforced**: Strict prohibition on fake, template, or synthetic data. 100% genuine datasets mined directly from public internet and Hugging Face repositories.
+* **Verified Sources Ingested**:
+  1. **Official Press Information Bureau (PIB) India** (`shivam/hindi_pib_processed`): 269,594 records of official Government of India announcements, cabinet decisions, national schemes, space/defense milestones, and contemporary history in pure Hindi.
+  2. **Complete NCERT History & Political Science Curriculum** (`KadamParth`): 29,395 records across Classes 6, 7, 8, 10, 11, and 12 covering Ancient India (IVC, Vedic, Maurya, Gupta), Medieval India (Chola, Delhi Sultanate, Mughals, Maratha Swarajya), Freedom Movement (1857, INA, Subhash, Gandhi, Patel), Constitution framing, and Post-Independence history.
+  3. **Verified Indian History Chronology** (`BashitAli/Indian_history`): 14,908 in-depth historical Q&A records.
+  4. **Hindi History Q&A** (`kaifahmad/indian-history-hindi-QA-3.4k`): 3,468 Devanagari Hindi records.
+  5. **Curated Expert History Dialogues** (`chungimungi/Indian-History`): 100 records.
+* **Deduplication & Hub Footprint**:
+  - Blake2b 64-bit hashing audited 316,981 records -> **4,186 duplicates dropped**.
+  - **312,795 unique authentic records** uploaded across 7 ZSTD Parquet shards (`domains/train-frontier_deduped_real_indian_history_pib-*.parquet`).
+  - Total pretraining dataset: **2,088 Parquet shards (100% streamed in `train.py`, 0 exclusions)**.
+
+## 🏏 10. 100% Real Cricket & IPL Corpus (Zero Fake Data)
+* **User Directive Enforced**: Complete real cricket dataset fetched from internet/HF, zero synthetic data, uploaded to Hugging Face Hub.
+* **Verified Sources Ingested**:
+  1. **Ball-by-Ball Match Commentary** (`nirmalkumar/cricket-commentary`): 82,480+ authentic match commentary records.
+  2. **Cricket Encyclopedia & Biographies** (`Ankush-Chander/cricket-wiki`): Thousands of comprehensive articles on legendary players (Sachin, Kohli, Dhoni, Rohit, Kapil Dev), grounds, and international tournaments.
+  3. **Official Laws & Rules** (`catyung/cricket-qa-dataset` & `srivats666/cricket-rules`): 1,345+ official rules and QA records on MCC Laws of Cricket, powerplays, LBW, super-overs.
+  4. **Historical Matches & Scorecards** (`bhuvaneshprasad`): 1,025 IPL matches (2008 to modern), 4,717 ODI matches, 2,426 T20I matches, 2,520 Test matches, and 7,400+ player career profiles.
+  5. **Hindi & Hinglish Sports Journalism** (`lallantop/cricket` & `BobbleAI`): 7,370+ real match reviews and Hinglish discussions.
+* **Deduplication & Hub Footprint**:
+  - Blake2b 64-bit hashing audited 191,442 candidate records -> **50,058 duplicates dropped**.
+  - **141,384 unique authentic records** uploaded across 3 dedicated ZSTD Parquet shards (`domains/train-frontier_deduped_real_cricket-*.parquet`).
+  - Total pretraining dataset: **2,091 Parquet shards (100% streamed in `train.py`, 0 exclusions)**.
+
+## 🚀 11. Step-1 Priority Pretraining Integration for Indian History & Cricket (Locked & Verified)
+* **User Directive Enforced**: *"apne new data jo abhi add kiya hai usko traing me add kro"* — Ensure newly added Indian History, PIB, and Cricket data stream directly into active pretraining right from Step 1.
+* **Queue Priority Engine Enhancement**:
+  - Implemented category round-robin interleaving with priority patterns across `train.py` in both `Viu-1.5B-MoE` and `ViuMini-MoE-242M`.
+  - Configured `PRIORITY_PATTERNS = ("real_indian_history_pib", "real_cricket", "real_industrial_qms", "conversational")`.
+  - In each category bucket, newly ingested frontier shards (`real_indian_history_pib` & `real_cricket`) are positioned at the absolute head of the queue, followed by priority conversational/QMS data, followed by remaining shards.
+* **Queue Position Audit**:
+  - Out of 2,091 total Parquet shards across 18 categories:
+    - Queue Position #3: `domains/train-frontier_deduped_real_indian_history_pib-00002.parquet`
+    - Queue Position #21: `domains/train-frontier_deduped_real_indian_history_pib-00001.parquet`
+    - Queue Position #37: `domains/train-frontier_deduped_real_indian_history_pib-00000.parquet`
+    - Queue Position #52: `domains/train-frontier_deduped_real_indian_history_pib-00005.parquet`
+    - Queue Position #65: `domains/train-frontier_deduped_real_cricket-00001.parquet`
+    - Queue Position #76: `domains/train-frontier_deduped_real_indian_history_pib-00006.parquet`
+    - Queue Position #87: `domains/train-frontier_deduped_real_indian_history_pib-00004.parquet`
+    - Queue Position #98: `domains/train-frontier_deduped_real_indian_history_pib-00003.parquet`
+    - Queue Position #108: `domains/train-frontier_deduped_real_cricket-00000.parquet`
+    - Queue Position #118: `domains/train-frontier_deduped_real_cricket-00002.parquet`
+  - All 10 newly added shards (454,179 authentic records) are guaranteed to load within the initial prefetch buffer (`max_blocks=50000` $\approx$ 180,000 blocks) and are actively trained from Step 1.
+* **Verification**:
+  - Smoke tests executed cleanly with Exit Code 0 on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M`.
+  - Synced to Hugging Face Hub `ViuAI/Viu-1.5B-MoE` and `ViuAI/ViuMini-MoE-242M`.
+
+## 🔞 12. 100% Real Adult Erotica Stories Corpus & Pretraining Queue Integration (Locked & Verified)
+* **User Directive Enforced**: *"ingest_erotica_adult.py (Sexy / Adult Stories & Erotica Pipeline) isko run krte ha and ye data dalte hai HF per"* — Run the adult erotica pipeline, upload real literary adult/erotica stories to HF Hub, and ensure it streams into active training.
+* **Safety & Minor Protection Protocol**:
+  - Strict zero-tolerance minor safety guard (`is_blocked` regex filter with exact word boundaries `\b` rejecting underage/non-consensual keywords).
+  - Only genuine, full-length adult creative romance/erotica stories (18+ consensual only).
+  - Standardized to unified 5-column schema: `['text', 'lang', 'source', 'domain', 'safety_tag']`.
+* **Deduplication & Hub Footprint**:
+  - Audited 5,000 raw story records from `detain/literotica-stories`.
+  - Dropped minor-flagged records; filtered short/duplicate narratives via Blake2b 64-bit hashing.
+  - **2,160 unique, full-length literary adult stories** (~25–30 Million tokens, avg 8,000–12,000 chars/story) uploaded to Hugging Face Hub:
+    - Shard: `erotica/train-frontier_deduped_erotica_adult_stories-00000.parquet` (13.5 MB ZSTD).
+  - Pretraining dataset total: **2,092 Parquet shards (100% streamed in `train.py`, 0 exclusions)**.
+* **Step-1 Priority Queue Integration**:
+  - Configured `PRIORITY_PATTERNS = ("real_indian_history_pib", "real_cricket", "erotica_adult_stories", "real_industrial_qms", "conversational")` in `model/scripts/train.py` across both repos.
+  - Added `erotica` domain and safety tag awareness (`allow_erotica` active under default `full` safety mode).
+  - **Queue Position Audit**: `erotica/train-frontier_deduped_erotica_adult_stories-00000.parquet` occupies **Queue Position #4 out of 2,092 files**, streaming into active pretraining immediately at Step 1.
+* **Verification**:
+  - Smoke tests executed cleanly with Exit Code 0 on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M`.
+
+* **Shard 00001 (Antarvasna Hindi Adult Stories)**:
+  - Scraped 101 raw records directly from Antarvasna3 via `data/scripts/scrape_antarvasna.py`.
+  - Processed and uploaded via `ingest_erotica_adult.py --execute --jsonl data/antarvasna_raw.jsonl --skip-hf`.
+  - Uploaded Shard: `erotica/train-frontier_deduped_erotica_adult_stories-00001.parquet` (84 verified full-length Hindi adult stories, 318 KB ZSTD).
+  - Total dataset shards on Hub: **2,093 Parquet files (100% streamed in `train.py`)**.
+  - Background scraper active across 19 sitemaps to continuously collect remaining stories.
+
+## 🎬 13. 100% Real Bollywood Cinema, Indian Mythology & Indian Finance Corpus (Locked & Verified)
+* **User Directive Enforced**: *"is data ko HF per upload kro"* — Ingest open-source authentic datasets across Bollywood Cinema/Music, Indian Mythology/Philosophy, and Indian Finance/Banking, upload to Hugging Face Hub, and integrate into training queue.
+* **Verified Sources Ingested (109,629 authentic records)**:
+  1. **Bollywood Cinema, Dialogues & Songs**:
+     - `HuggMachas/Bollywood_dialogues`: 1,810 iconic scene conversations and dialogues.
+     - `eswardivi/Bollywood_songs`: 999 song lyrics records with instructions and output.
+     - `Vangmayy/bollywood_plots`: 4,016 detailed Hindi movie plot summaries and character narratives.
+     - **Shard**: `domains/train-frontier_deduped_real_cinema-00000.parquet` (6,612 rows, 3.78 MB ZSTD).
+  2. **Indian Mythology & Classical Philosophy**:
+     - `OEvortex/Bhagavad_Gita`: 700 complete Shrimad Bhagavad Gita verses (Sanskrit Shloka + Hindi Anuvad + English translation).
+     - `rahulnyk/mahabharata`: 2,595 comprehensive chapter records covering all 18 Parvas of Mahabharata.
+     - **Shard**: `domains/train-frontier_deduped_real_mythology-00000.parquet` (3,295 rows, 900 KB ZSTD).
+  3. **Indian Finance, RBI & Market**:
+     - `kdave/Indian_Financial_News`: 12,711 full financial news articles covering BSE/NSE, Nifty, and Indian corporate finance.
+     - `AISimplyExplained/RBI_Notifications`: 87,163 official Reserve Bank of India notifications, circulars, and monetary policy statements.
+     - **Shards**: `domains/train-frontier_deduped_real_finance-00000.parquet` to `...-00003.parquet` (99,722 rows total).
+* **Deduplication & Hub Footprint**:
+  - Blake2b 64-bit deduplication audited raw records and dropped 24,672 duplicates.
+  - **109,629 unique authentic records** uploaded across 6 dedicated ZSTD Parquet shards.
+  - Pretraining dataset total on Hub: **2,099 Parquet shards (100% streamed in `train.py`, 0 exclusions)**.
+* **Step-1 Priority Pretraining Integration**:
+  - Configured `PRIORITY_PATTERNS = ("real_indian_history_pib", "real_cricket", "erotica_adult_stories", "real_cinema", "real_mythology", "real_finance", "real_industrial_qms", "conversational")`.
+  - Audited queue positions: All new shards are positioned at the head of category round-robin interleaver (within top ~160 files).
+  - CPU smoke tests executed cleanly with Exit Code 0 on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M`.
+
+## 🏛️ 14. Massive Scale Expansion: Indian Mythology, Vedas, Puranas & Bollywood Cinema Screenplays (Locked & Verified)
+* **User Directive Enforced**: *"ye bollywood and indian mythology ka data bhut kaam aayaa hai"* — Bollywood and Indian Mythology data was previously too small (6,612 and 3,295 rows) compared to Finance (99,722 rows). Massive open-source corpora were identified, ingested, and uploaded to expand both domains by orders of magnitude.
+* **Massive Datasets Ingested (165,617 New Authentic Records)**:
+  1. **Indian Mythology, Vedas & Classical Philosophy (152,662 New Verses / Sections)**:
+     - **All 16 Mahapuranas & Upapuranas (`dataspoof/Puranas-dataset`)**: 120,172 verses with Sanskrit Devanagari, IAST transliteration, and chapter/skandha markers (*Bhagavatam*, *Agni*, *Brahmanda*, *Brahma*, *Devi Gita*, *Garuda*, *Kurma*, *Markandeya*, *Matsya*, *Narada*, *Narasimha*, *Shiva*, *Skanda*, *Vamana*, *Vayu*, *Vishnu*).
+     - **All 4 Vedas (`dataspoof/Vedas`)**: 15,339 Vedic mantras with Devanagari Samhita, Padapatha transliteration, and English/Hindi translations (*Rigveda Complete*, *Yajurveda*, *Atharvaveda*, *Samaveda*).
+     - **Valmiki Ramayana (`sanganaka/ramayana-anvaya`)**: 16,447 verses with original Sanskrit sloka and word-by-word prose anvaya.
+     - **Mahabharata (`nirbhaysinghnarang/Mahabharat`)**: 704 comprehensive narrative sections covering all 18 Parvas (Ganguli English translation).
+     - **Total Mythology Records**: **155,957 records** across **8 Parquet Shards** (`domains/train-frontier_deduped_real_mythology-00000.parquet` to `...-00007.parquet`).
+  2. **Bollywood Cinema, Screenplays & Dialogues (12,955 New Records)**:
+     - **Bollywood Dialogues (`HuggMachas/Bollywood_separate_dailogues`)**: 13,060 multi-turn conversational dialogue scenes generated across 13,271 Hindi movies.
+     - **Hindi Feature Film Screenplays (`pratikkalamkar/Hindi_Movie_Script_Corpus_by_Pratik_Kalamkar`)**: 2,829 multi-page screenplay scenes extracted from 103 complete Hindi movie scripts (*Stree*, *Tumbbad*, *Soorma*, etc.).
+     - **Hindi Movie Reviews (`Process-Venue/Movie_Review_Sentiment_Hindi`)**: 998 detailed film reviews with sentiment annotations.
+     - **Total Cinema Records**: **19,567 records** across **2 Parquet Shards** (`domains/train-frontier_deduped_real_cinema-00000.parquet` and `00001.parquet`).
+* **Hub Footprint & Zero Disk Residue**:
+  - Parquet shards compressed with ZSTD; all local temporary parquet shards unlinked immediately upon upload.
+  - Total shards in `ViuAI/viu-mini-raw-pretrain` expanded to **2,107 Parquet files (~215 GB compressed, ~120B+ tokens)**.
+* **Pretraining Queue Integration & Verification**:
+  - `PRIORITY_PATTERNS` in `model/scripts/train.py` (`"real_mythology"`, `"real_cinema"`) guarantees all 10 Mythology & Cinema shards stream into active training from Step 1.
+  - Smoke tests executed with Exit Code 0 on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M`.
+  - Synced code and pipeline scripts across both model repositories.