Add Pillar 26 Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax)
Browse files- data/scripts/collect_domains.py +14 -0
- docs/DATA_SOURCES.md +4 -0
- docs/PROGRESS.md +2 -2
- docs/ROADMAP_SCRATCH.md +1 -1
- docs/TEST_REPORT.md +1 -1
data/scripts/collect_domains.py
CHANGED
|
@@ -164,6 +164,20 @@ DOMAIN_KNOWLEDGE_PACKS = {
|
|
| 164 |
("Bakchodi band kar aur seedhe kaam ki baat par aa. Tere fukrapane dekhne ka time nahi hai kisi ke paas.", "hinglish"),
|
| 165 |
("Ek dost doosre se bola: 'Bhai usne WhatsApp pe dekha par reply nahi diya, kya karu?' Doosre ne bola: 'Abe champu, usko dimaag se delete kar aur aage badh, faltu rona mat ro yahan!'", "hinglish"),
|
| 166 |
("Ye jo tu hawa-bazi kar raha hai na, do minute me saari hawa nikal jayegi agar reality saamne aa gayi toh.", "hinglish"),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 167 |
]
|
| 168 |
}
|
| 169 |
|
|
|
|
| 164 |
("Bakchodi band kar aur seedhe kaam ki baat par aa. Tere fukrapane dekhne ka time nahi hai kisi ke paas.", "hinglish"),
|
| 165 |
("Ek dost doosre se bola: 'Bhai usne WhatsApp pe dekha par reply nahi diya, kya karu?' Doosre ne bola: 'Abe champu, usko dimaag se delete kar aur aage badh, faltu rona mat ro yahan!'", "hinglish"),
|
| 166 |
("Ye jo tu hawa-bazi kar raha hai na, do minute me saari hawa nikal jayegi agar reality saamne aa gayi toh.", "hinglish"),
|
| 167 |
+
],
|
| 168 |
+
"grammar_and_language_mechanics": [
|
| 169 |
+
# Hindi Vyakaran
|
| 170 |
+
("Hindi me Karta Karak ke sath 'Ne' chinh aane par kriya (verb) Karta ke anusaar nahi, balki Karm (object) ke ling aur vachan ke anusaar badalti hai. Udaharan: 'Ram ne roti khayi' (yahan roti striling hai isliye kriya 'khayi' hui, na ki 'khaya').", "hindi"),
|
| 171 |
+
("Hindi Vakya Shuddhi me aksar 'aneko' shabd ka ashuddh prayog kiya jata hai; 'anek' shabd swayam me hi bahuvachan hai, isliye 'anek log' kehna shuddh hai, 'aneko log' ashuddh hai.", "hindi"),
|
| 172 |
+
("Karak ke 8 bhed hote hain: Karta (ne), Karm (ko), Karan (se/ke dwara), Sampradan (ko/ke liye), Apadan (se prithak), Sambandh (ka/ke/ki), Adhikaran (me/par), aur Sambodhan (he/are).", "hindi"),
|
| 173 |
+
("Sandhi me do varnon ke mel se hone wale vikar ko sandhi kehte hain: Svar Sandhi (jaise Vidya + Aalay = Vidyalay), Vyanjan Sandhi (Sat + Jan = Sajjan), aur Visarga Sandhi (Manah + Har = Manohar).", "hindi"),
|
| 174 |
+
("Muhavara 'Id ka Chand hona' ka arth hai bohot dinon baad dikhayi dena. Vakya: 'Rohan jab se foreign gaya hai, tab se wo Id ka chand ho gaya hai.'", "hindi"),
|
| 175 |
+
# English Grammar
|
| 176 |
+
("Subject-Verb Agreement rule: When two singular subjects are joined by 'neither...nor' or 'either...or', the verb agrees with the subject closest to it. Example: 'Neither the manager nor the employees were informed about the schedule change.'", "english"),
|
| 177 |
+
("Active to Passive Voice transformation: In passive voice, the object becomes the subject and the main verb is always in past participle (V3) form. Active: 'The technician calibrated the SMT machine.' -> Passive: 'The SMT machine was calibrated by the technician.'", "english"),
|
| 178 |
+
("Direct to Indirect Speech: In reporting statements, past tense in the reporting verb shifts the present tense of reported speech to past tense. Direct: He said, 'I am learning Python.' -> Indirect: He said that he was learning Python.", "english"),
|
| 179 |
+
# Hinglish Syntax & Code-Switching
|
| 180 |
+
("Hinglish me English verbs ko Hindi auxiliary 'karna' ya 'hona' ke sath pair kiya jata hai: 'file upload karna', 'payment process hona'. Gender agreement Hindi noun/context ke hisaab se follow hota hai: 'Battery drain ho rahi hai' (striling battery ke anusaar), 'Server restart ho raha hai' (pulling server ke anusaar).", "hinglish"),
|
| 181 |
]
|
| 182 |
}
|
| 183 |
|
docs/DATA_SOURCES.md
CHANGED
|
@@ -124,6 +124,10 @@
|
|
| 124 |
- Relatable Indian Humor: College/hostel banter, corporate employee memes, rishtedaar/shaadi jokes, doctor-patient humor.
|
| 125 |
- Stand-up Comedy & Witty Comebacks: Observational humor (Zakir Khan / Bassi style punchlines).
|
| 126 |
- Colloquial Street Hinglish & Playful Roasting: Street slangs (*dhakkan, chirkut, bakchodi, dimaag ki dahi, hawa-bazi, champu, fukra*), sharp sarcastic comebacks, and friendly desi banter.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 127 |
|
| 128 |
---
|
| 129 |
|
|
|
|
| 124 |
- Relatable Indian Humor: College/hostel banter, corporate employee memes, rishtedaar/shaadi jokes, doctor-patient humor.
|
| 125 |
- Stand-up Comedy & Witty Comebacks: Observational humor (Zakir Khan / Bassi style punchlines).
|
| 126 |
- Colloquial Street Hinglish & Playful Roasting: Street slangs (*dhakkan, chirkut, bakchodi, dimaag ki dahi, hawa-bazi, champu, fukra*), sharp sarcastic comebacks, and friendly desi banter.
|
| 127 |
+
26. **Complete Grammar & Language Mechanics (Vyakaran & Syntax)**:
|
| 128 |
+
- Sampoorn Hindi Vyakaran: Sangya, Sarvanam, Karak ('Ne' rule ergativity concord: 'Ram ne roti khayi'), Ling & Vachan rules, 12 Kaal/Tenses, Vakya Shuddhi ('anek' vs 'aneko'), Sandhi, Samas, Muhavare & Lokoktiyan.
|
| 129 |
+
- English Grammar & Mechanics: 8 Parts of Speech, Subject-Verb Agreement concord, Active & Passive Voice transformations, Direct & Indirect Speech narration, 12 Tenses.
|
| 130 |
+
- Hinglish Syntax & Natural Code-Switching: Verb pairing with auxiliary 'karna'/'hona', gender agreement with loanwords ('battery drain ho rahi hai', 'server restart ho raha hai').
|
| 131 |
|
| 132 |
---
|
| 133 |
|
docs/PROGRESS.md
CHANGED
|
@@ -86,9 +86,9 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
|
|
| 86 |
1. Target scale upgraded to **30B – 50B+ Tokens** (Extreme Over-Training for 242M parameter model).
|
| 87 |
2. Language mix locked: **40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English**.
|
| 88 |
3. **Zero Sub-Sampling / Full Dataset Ingestion Policy**: Ingest 100% of open corpora (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi) without artificial slicing.
|
| 89 |
-
4. Master
|
| 90 |
5. Target HF Dataset Repo: `ViuAI/viu-mini-raw-pretrain` (8TB capacity).
|
| 91 |
-
6. Pipeline: `data/scripts/collect_domains.py` (
|
| 92 |
7. Runbook: `docs/KAGGLE_DATA_COLLECTION.md` with session-proof copy-paste cells including Translation worker.
|
| 93 |
8. Verified: `kaggle_collector.py --dry-run` PASS (clean Parquet shard generated, uploaded-mocked, and deleted).
|
| 94 |
- [ ] TODO: Execute Kaggle data collection workers and monitor shards in `ViuAI/viu-mini-raw-pretrain`.
|
|
|
|
| 86 |
1. Target scale upgraded to **30B – 50B+ Tokens** (Extreme Over-Training for 242M parameter model).
|
| 87 |
2. Language mix locked: **40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English**.
|
| 88 |
3. **Zero Sub-Sampling / Full Dataset Ingestion Policy**: Ingest 100% of open corpora (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi) without artificial slicing.
|
| 89 |
+
4. Master 26-Pillar Knowledge Coverage: All NCERT (Class 1-12), Indian History, Constitution & Law (BNS/CrPC/RTI), Electronics & SMT/PCB, Industrial QMS/FA/ISO, Tech/CS, Corporate Emails, Cinema, Everyday Math with `<soch>`, Finance, Health, Sports, All Indian & Foreign Cuisines, Multilingual Translation (IITB/IndicTrans), Stories & Literature, Movie Plots & Songs/Lyrics, Mobile Tech (Android/iOS/ADB), Jokes & Street Roasts, and Complete Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax).
|
| 90 |
5. Target HF Dataset Repo: `ViuAI/viu-mini-raw-pretrain` (8TB capacity).
|
| 91 |
+
6. Pipeline: `data/scripts/collect_domains.py` (26-pillar knowledge extractor) + `data/scripts/kaggle_collector.py` (Kaggle cloud streaming worker supporting hindi/english/hinglish/domains/translation streams with chunk-upload-delete loop, keeping Kaggle disk < 1GB).
|
| 92 |
7. Runbook: `docs/KAGGLE_DATA_COLLECTION.md` with session-proof copy-paste cells including Translation worker.
|
| 93 |
8. Verified: `kaggle_collector.py --dry-run` PASS (clean Parquet shard generated, uploaded-mocked, and deleted).
|
| 94 |
- [ ] TODO: Execute Kaggle data collection workers and monitor shards in `ViuAI/viu-mini-raw-pretrain`.
|
docs/ROADMAP_SCRATCH.md
CHANGED
|
@@ -20,7 +20,7 @@
|
|
| 20 |
- Evidence: `docs/TEST_REPORT.md` (Step 3 entries), `model/scripts/toy_router_test.py`
|
| 21 |
|
| 22 |
## Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI)
|
| 23 |
-
- Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across
|
| 24 |
- Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested)
|
| 25 |
- Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB)
|
| 26 |
- Evidence: `data/scripts/kaggle_collector.py`, `docs/KAGGLE_DATA_COLLECTION.md`, HF repo `ViuAI/viu-mini-raw-pretrain`
|
|
|
|
| 20 |
- Evidence: `docs/TEST_REPORT.md` (Step 3 entries), `model/scripts/toy_router_test.py`
|
| 21 |
|
| 22 |
## Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI)
|
| 23 |
+
- Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across 26 Exhaustive Pillars (NCERT, History, Law, SMT/PCB/FA, QMS, Tech, Emails, Cinema, Global Cooking, Translation, Stories, Songs, Mobile Tech, Jokes & Desi Roasts, Complete Grammar & Vyakaran)
|
| 24 |
- Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested)
|
| 25 |
- Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB)
|
| 26 |
- Evidence: `data/scripts/kaggle_collector.py`, `docs/KAGGLE_DATA_COLLECTION.md`, HF repo `ViuAI/viu-mini-raw-pretrain`
|
docs/TEST_REPORT.md
CHANGED
|
@@ -268,7 +268,7 @@ Har test ke baad yaha entry hogi: date, env, command, result, verdict.
|
|
| 268 |
- Cmds:
|
| 269 |
- `python data/scripts/kaggle_collector.py --dry-run --stream domains --limit 10`
|
| 270 |
- Upgrades Covered:
|
| 271 |
-
1. Master
|
| 272 |
2. High-Throughput Kaggle Collector (`kaggle_collector.py`): streams raw open corpora (including `cfilt/iitb-english-hindi` translation parallel corpus), performs in-flight 64-bit integer hash deduplication and script filtering, packs into 100k-row Parquet shards, uploads to `ViuAI/viu-mini-raw-pretrain`, and immediately deletes local files to preserve Kaggle disk (<1GB).
|
| 273 |
- Result:
|
| 274 |
- Collector initialized stream `domains`, hit limit of 10 rows.
|
|
|
|
| 268 |
- Cmds:
|
| 269 |
- `python data/scripts/kaggle_collector.py --dry-run --stream domains --limit 10`
|
| 270 |
- Upgrades Covered:
|
| 271 |
+
1. Master 26-Pillar Domain Extractor (`collect_domains.py`): provides curated foundational knowledge across NCERT, Law & Constitution, SMT/PCB, Quality (QMS/FA), Tech/CS, Corporate Emails, Cinema, Everyday Math with `<soch>`, All Indian & Foreign Cuisines, 3-Way Translation (IITB/IndicTrans), Stories & Literature (Premchand/Panchatantra), Movie Plots & Songs/Lyrics (Drishyam/Lagaan/Kishore/Gulzar), Mobile Tech (Android/iOS/ADB/DFU), Jokes, Stand-up & Desi Street Roasts/Slangs, and Complete Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax).
|
| 272 |
2. High-Throughput Kaggle Collector (`kaggle_collector.py`): streams raw open corpora (including `cfilt/iitb-english-hindi` translation parallel corpus), performs in-flight 64-bit integer hash deduplication and script filtering, packs into 100k-row Parquet shards, uploads to `ViuAI/viu-mini-raw-pretrain`, and immediately deletes local files to preserve Kaggle disk (<1GB).
|
| 273 |
- Result:
|
| 274 |
- Collector initialized stream `domains`, hit limit of 10 rows.
|