feat: 100% zero-exclusion streaming architecture (2,081 shards) and updated docs
Browse files- docs/PROGRESS.md +12 -0
docs/PROGRESS.md
CHANGED
|
@@ -544,3 +544,15 @@ Kaggle par manual file upload se version mismatch hota hai (v1 tokenizer wala ca
|
|
| 544 |
7. `generated_full_text` -> Science textbooks and SciQ records (3.01M rows).
|
| 545 |
8. `comment_text` + toxicity scores -> Jigsaw Civil Comments (1.80M rows).
|
| 546 |
- **Verification**: Verified via `scratch/verify_zero_exclusions.py` (2,081/2,081 included, 0 excluded) and CPU smoke tests on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M` (exit code 0).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 544 |
7. `generated_full_text` -> Science textbooks and SciQ records (3.01M rows).
|
| 545 |
8. `comment_text` + toxicity scores -> Jigsaw Civil Comments (1.80M rows).
|
| 546 |
- **Verification**: Verified via `scratch/verify_zero_exclusions.py` (2,081/2,081 included, 0 excluded) and CPU smoke tests on both `Viu-1.5B-MoE` and `ViuMini-MoE-242M` (exit code 0).
|
| 547 |
+
|
| 548 |
+
- [x] 2026-09-29 100% REAL AUTHENTIC INDIAN HISTORY & PIB CORPUS INGESTION (ZERO FAKE DATA):
|
| 549 |
+
- **User Directive**: "mujha india ka real data inter net say utha na or na he fack data creat mat karana or iss ko hf per dar dana" — Fetch authentic real data from internet/HF, zero synthetic/fake data, upload directly to Hub.
|
| 550 |
+
- **Ingested Authentic Sources**:
|
| 551 |
+
1. *Official Press Information Bureau (PIB) India (`shivam/hindi_pib_processed`)*: **269,594 authentic records** of official Government of India cabinet declarations, policies, economic schemes, bilateral agreements, defense/space advancements, and modern history in pure Hindi.
|
| 552 |
+
2. *Full NCERT History & Political Science Curriculum (`KadamParth`)*: **29,395 authentic records** covering Classes 6, 7, 8, 10, 11, and 12 (Ancient Harappa, Vedic era, Mahajanapadas, Mauryas, Guptas, Cholas, Delhi Sultanate, Mughals, Maratha Swarajya, 1857 Revolt, Freedom Struggle, Indian Constitution, Post-Independence politics).
|
| 553 |
+
3. *Verified Indian History Chronology (`BashitAli/Indian_history`)*: **14,908 authentic records** of detailed historical Q&A.
|
| 554 |
+
4. *Authentic Hindi Indian History Q&A (`kaifahmad/indian-history-hindi-QA-3.4k`)*: **3,468 authentic records** of Hindi history Q&A.
|
| 555 |
+
5. *Expert History Dialogues (`chungimungi/Indian-History`)*: **100 authentic records**.
|
| 556 |
+
- **In-Flight Blake2b Deduplication**: Audited 316,981 raw records -> **4,186 duplicate rows dropped**; **312,795 UNIQUE authentic records** packaged into 7 dedicated ZSTD Parquet shards (`domains/train-frontier_deduped_real_indian_history_pib-00000.parquet` to `...-00006.parquet`).
|
| 557 |
+
- **Hub Footprint**: Repository `ViuAI/viu-mini-raw-pretrain` expanded to **2,088 Parquet files (~213.5 GB compressed, ~232.1M rows, ~117–127B tokens)**.
|
| 558 |
+
- **Zero-Loss Verification**: `scratch/verify_zero_exclusions.py` confirmed exactly **2,088 out of 2,088 Parquet shards (100.0%)** are actively streamed into `train.py` with **0 exclusions**.
|