feat: 100% zero-exclusion streaming architecture (2,081 shards) and updated docs

docs/PROGRESS.md

@@ -527,3 +527,20 @@ Kaggle par manual file upload se version mismatch hota hai (v1 tokenizer wala ca
   - **Option 1 Standalone Project Initialized**:
     - Created dedicated sister project `Viu-1.5B-MoE`.
     - Golden Development Protocol locked: (1) Prepare file/plan first -> (2) Execute & verify -> (3) Record all work inside NOTES.md & PROGRESS.md.
+
+- [x] 2026-09-29 100% ZERO-EXCLUSION PRETRAINING ARCHITECTURE (2,081 PARQUET SHARDS INGESTED):
+  - **User Directive**: "koi file and koi bhi data ko exclude nhi krna bro" — zero files, zero domains, zero categories excluded from pretraining.
+  - **Engine Zero-Exclusion Modification**:
+    - Cleared `NON_UNIFIED_PREFIXES = ()` across all training pipelines. Exactly **2,081 out of 2,081 Parquet files (100.0%)** stream natively.
+    - Verified 0 files excluded across all 18 categories on Hugging Face Hub (`ViuAI/viu-mini-raw-pretrain`):
+      `code` (3), `distilled` (33), `domains` (271), `english_fixed` (510), `finance` (1), `grammar` (1), `health` (3), `hindi` (434), `hindi_fixed` (661), `hinglish` (7), `legal` (33), `math` (4), `science` (35), `stories` (3), `toxicity` (4), `translation` (21), `uncensored` (16), `wikipedia` (41).
+  - **Universal Multi-Schema Adapters**:
+    1. `instruction` + `output` -> Instruction-following & code (`uncensored/vortex_train_*.parquet` 8.55M rows, `code/python_code_instructions_18k.parquet` 18.6k rows).
+    2. `tweet` -> Social/street slang & offensive language (`toxicity/hate_speech_offensive_train.parquet` 24,783 rows).
+    3. `question` + `answer` -> Standalone QA pairs (`finance/financial_qa_10k.parquet` 7,000 rows).
+    4. `Patient` + `Doctor` -> Healthcare clinical consultations (`health/ai_medical_dialogues.parquet` 256,916 rows).
+    5. `context` + `question` + `response` -> Indian Supreme Court & High Court case judgments (6.05M rows).
+    6. `src` + `tgt` -> AI4Bharat Samanantar English <-> Hindi parallel translations (26M pairs).
+    7. `generated_full_text` -> Science textbooks and SciQ records (3.01M rows).
+    8. `comment_text` + toxicity scores -> Jigsaw Civil Comments (1.80M rows).
+  - **Verification**: Verified via `scratch/verify_zero_exclusions.py` (2,081/2,081 included, 0 excluded) and CPU smoke tests on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M` (exit code 0).