ViuAI commited on
Commit
cc255c6
·
verified ·
1 Parent(s): 7812c7f

Add Pillar 26 Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax)

Browse files
data/scripts/collect_domains.py CHANGED
@@ -164,6 +164,20 @@ DOMAIN_KNOWLEDGE_PACKS = {
164
  ("Bakchodi band kar aur seedhe kaam ki baat par aa. Tere fukrapane dekhne ka time nahi hai kisi ke paas.", "hinglish"),
165
  ("Ek dost doosre se bola: 'Bhai usne WhatsApp pe dekha par reply nahi diya, kya karu?' Doosre ne bola: 'Abe champu, usko dimaag se delete kar aur aage badh, faltu rona mat ro yahan!'", "hinglish"),
166
  ("Ye jo tu hawa-bazi kar raha hai na, do minute me saari hawa nikal jayegi agar reality saamne aa gayi toh.", "hinglish"),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
167
  ]
168
  }
169
 
 
164
  ("Bakchodi band kar aur seedhe kaam ki baat par aa. Tere fukrapane dekhne ka time nahi hai kisi ke paas.", "hinglish"),
165
  ("Ek dost doosre se bola: 'Bhai usne WhatsApp pe dekha par reply nahi diya, kya karu?' Doosre ne bola: 'Abe champu, usko dimaag se delete kar aur aage badh, faltu rona mat ro yahan!'", "hinglish"),
166
  ("Ye jo tu hawa-bazi kar raha hai na, do minute me saari hawa nikal jayegi agar reality saamne aa gayi toh.", "hinglish"),
167
+ ],
168
+ "grammar_and_language_mechanics": [
169
+ # Hindi Vyakaran
170
+ ("Hindi me Karta Karak ke sath 'Ne' chinh aane par kriya (verb) Karta ke anusaar nahi, balki Karm (object) ke ling aur vachan ke anusaar badalti hai. Udaharan: 'Ram ne roti khayi' (yahan roti striling hai isliye kriya 'khayi' hui, na ki 'khaya').", "hindi"),
171
+ ("Hindi Vakya Shuddhi me aksar 'aneko' shabd ka ashuddh prayog kiya jata hai; 'anek' shabd swayam me hi bahuvachan hai, isliye 'anek log' kehna shuddh hai, 'aneko log' ashuddh hai.", "hindi"),
172
+ ("Karak ke 8 bhed hote hain: Karta (ne), Karm (ko), Karan (se/ke dwara), Sampradan (ko/ke liye), Apadan (se prithak), Sambandh (ka/ke/ki), Adhikaran (me/par), aur Sambodhan (he/are).", "hindi"),
173
+ ("Sandhi me do varnon ke mel se hone wale vikar ko sandhi kehte hain: Svar Sandhi (jaise Vidya + Aalay = Vidyalay), Vyanjan Sandhi (Sat + Jan = Sajjan), aur Visarga Sandhi (Manah + Har = Manohar).", "hindi"),
174
+ ("Muhavara 'Id ka Chand hona' ka arth hai bohot dinon baad dikhayi dena. Vakya: 'Rohan jab se foreign gaya hai, tab se wo Id ka chand ho gaya hai.'", "hindi"),
175
+ # English Grammar
176
+ ("Subject-Verb Agreement rule: When two singular subjects are joined by 'neither...nor' or 'either...or', the verb agrees with the subject closest to it. Example: 'Neither the manager nor the employees were informed about the schedule change.'", "english"),
177
+ ("Active to Passive Voice transformation: In passive voice, the object becomes the subject and the main verb is always in past participle (V3) form. Active: 'The technician calibrated the SMT machine.' -> Passive: 'The SMT machine was calibrated by the technician.'", "english"),
178
+ ("Direct to Indirect Speech: In reporting statements, past tense in the reporting verb shifts the present tense of reported speech to past tense. Direct: He said, 'I am learning Python.' -> Indirect: He said that he was learning Python.", "english"),
179
+ # Hinglish Syntax & Code-Switching
180
+ ("Hinglish me English verbs ko Hindi auxiliary 'karna' ya 'hona' ke sath pair kiya jata hai: 'file upload karna', 'payment process hona'. Gender agreement Hindi noun/context ke hisaab se follow hota hai: 'Battery drain ho rahi hai' (striling battery ke anusaar), 'Server restart ho raha hai' (pulling server ke anusaar).", "hinglish"),
181
  ]
182
  }
183
 
docs/DATA_SOURCES.md CHANGED
@@ -124,6 +124,10 @@
124
  - Relatable Indian Humor: College/hostel banter, corporate employee memes, rishtedaar/shaadi jokes, doctor-patient humor.
125
  - Stand-up Comedy & Witty Comebacks: Observational humor (Zakir Khan / Bassi style punchlines).
126
  - Colloquial Street Hinglish & Playful Roasting: Street slangs (*dhakkan, chirkut, bakchodi, dimaag ki dahi, hawa-bazi, champu, fukra*), sharp sarcastic comebacks, and friendly desi banter.
 
 
 
 
127
 
128
  ---
129
 
 
124
  - Relatable Indian Humor: College/hostel banter, corporate employee memes, rishtedaar/shaadi jokes, doctor-patient humor.
125
  - Stand-up Comedy & Witty Comebacks: Observational humor (Zakir Khan / Bassi style punchlines).
126
  - Colloquial Street Hinglish & Playful Roasting: Street slangs (*dhakkan, chirkut, bakchodi, dimaag ki dahi, hawa-bazi, champu, fukra*), sharp sarcastic comebacks, and friendly desi banter.
127
+ 26. **Complete Grammar & Language Mechanics (Vyakaran & Syntax)**:
128
+ - Sampoorn Hindi Vyakaran: Sangya, Sarvanam, Karak ('Ne' rule ergativity concord: 'Ram ne roti khayi'), Ling & Vachan rules, 12 Kaal/Tenses, Vakya Shuddhi ('anek' vs 'aneko'), Sandhi, Samas, Muhavare & Lokoktiyan.
129
+ - English Grammar & Mechanics: 8 Parts of Speech, Subject-Verb Agreement concord, Active & Passive Voice transformations, Direct & Indirect Speech narration, 12 Tenses.
130
+ - Hinglish Syntax & Natural Code-Switching: Verb pairing with auxiliary 'karna'/'hona', gender agreement with loanwords ('battery drain ho rahi hai', 'server restart ho raha hai').
131
 
132
  ---
133
 
docs/PROGRESS.md CHANGED
@@ -86,9 +86,9 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
86
  1. Target scale upgraded to **30B – 50B+ Tokens** (Extreme Over-Training for 242M parameter model).
87
  2. Language mix locked: **40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English**.
88
  3. **Zero Sub-Sampling / Full Dataset Ingestion Policy**: Ingest 100% of open corpora (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi) without artificial slicing.
89
- 4. Master 25-Pillar Knowledge Coverage: All NCERT (Class 1-12), Indian History, Constitution & Law (BNS/CrPC/RTI), Electronics & SMT/PCB, Industrial QMS/FA/ISO, Tech/CS, Corporate Emails, Cinema, Everyday Math with `<soch>`, Finance, Health, Sports, All Indian & Foreign Cuisines, Multilingual Translation (IITB/IndicTrans), Stories & Literature, Movie Plots & Songs/Lyrics, Mobile Tech (Android/iOS/ADB), and Jokes, Stand-up & Street Roasts/Slangs.
90
  5. Target HF Dataset Repo: `ViuAI/viu-mini-raw-pretrain` (8TB capacity).
91
- 6. Pipeline: `data/scripts/collect_domains.py` (25-pillar knowledge extractor) + `data/scripts/kaggle_collector.py` (Kaggle cloud streaming worker supporting hindi/english/hinglish/domains/translation streams with chunk-upload-delete loop, keeping Kaggle disk < 1GB).
92
  7. Runbook: `docs/KAGGLE_DATA_COLLECTION.md` with session-proof copy-paste cells including Translation worker.
93
  8. Verified: `kaggle_collector.py --dry-run` PASS (clean Parquet shard generated, uploaded-mocked, and deleted).
94
  - [ ] TODO: Execute Kaggle data collection workers and monitor shards in `ViuAI/viu-mini-raw-pretrain`.
 
86
  1. Target scale upgraded to **30B – 50B+ Tokens** (Extreme Over-Training for 242M parameter model).
87
  2. Language mix locked: **40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English**.
88
  3. **Zero Sub-Sampling / Full Dataset Ingestion Policy**: Ingest 100% of open corpora (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi) without artificial slicing.
89
+ 4. Master 26-Pillar Knowledge Coverage: All NCERT (Class 1-12), Indian History, Constitution & Law (BNS/CrPC/RTI), Electronics & SMT/PCB, Industrial QMS/FA/ISO, Tech/CS, Corporate Emails, Cinema, Everyday Math with `<soch>`, Finance, Health, Sports, All Indian & Foreign Cuisines, Multilingual Translation (IITB/IndicTrans), Stories & Literature, Movie Plots & Songs/Lyrics, Mobile Tech (Android/iOS/ADB), Jokes & Street Roasts, and Complete Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax).
90
  5. Target HF Dataset Repo: `ViuAI/viu-mini-raw-pretrain` (8TB capacity).
91
+ 6. Pipeline: `data/scripts/collect_domains.py` (26-pillar knowledge extractor) + `data/scripts/kaggle_collector.py` (Kaggle cloud streaming worker supporting hindi/english/hinglish/domains/translation streams with chunk-upload-delete loop, keeping Kaggle disk < 1GB).
92
  7. Runbook: `docs/KAGGLE_DATA_COLLECTION.md` with session-proof copy-paste cells including Translation worker.
93
  8. Verified: `kaggle_collector.py --dry-run` PASS (clean Parquet shard generated, uploaded-mocked, and deleted).
94
  - [ ] TODO: Execute Kaggle data collection workers and monitor shards in `ViuAI/viu-mini-raw-pretrain`.
docs/ROADMAP_SCRATCH.md CHANGED
@@ -20,7 +20,7 @@
20
  - Evidence: `docs/TEST_REPORT.md` (Step 3 entries), `model/scripts/toy_router_test.py`
21
 
22
  ## Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI)
23
- - Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across 25 Exhaustive Pillars (NCERT, History, Law, SMT/PCB/FA, QMS, Tech, Emails, Cinema, Global Cooking, Translation, Stories, Songs, Mobile Tech, Jokes & Desi Roasts)
24
  - Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested)
25
  - Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB)
26
  - Evidence: `data/scripts/kaggle_collector.py`, `docs/KAGGLE_DATA_COLLECTION.md`, HF repo `ViuAI/viu-mini-raw-pretrain`
 
20
  - Evidence: `docs/TEST_REPORT.md` (Step 3 entries), `model/scripts/toy_router_test.py`
21
 
22
  ## Step 4 — Real data collect [ ] BACHA (sabse important — DATA HI MODEL HAI)
23
+ - Target: 30B – 50B+ tokens 40:30:30 (40% Hinglish Roman + 30% Hindi Devanagari + 30% English) across 26 Exhaustive Pillars (NCERT, History, Law, SMT/PCB/FA, QMS, Tech, Emails, Cinema, Global Cooking, Translation, Stories, Songs, Mobile Tech, Jokes & Desi Roasts, Complete Grammar & Vyakaran)
24
  - Policy: Zero Sub-Sampling / Full Ingestion (IndicCorpV2 152M rows, Sangraha 250M sentences, FineWeb-Edu, Romanized Hindi, IITB translation 100% ingested)
25
  - Execution: 100% Kaggle Cloud streaming with chunk-upload-delete loop (disk < 1GB)
26
  - Evidence: `data/scripts/kaggle_collector.py`, `docs/KAGGLE_DATA_COLLECTION.md`, HF repo `ViuAI/viu-mini-raw-pretrain`
docs/TEST_REPORT.md CHANGED
@@ -268,7 +268,7 @@ Har test ke baad yaha entry hogi: date, env, command, result, verdict.
268
  - Cmds:
269
  - `python data/scripts/kaggle_collector.py --dry-run --stream domains --limit 10`
270
  - Upgrades Covered:
271
- 1. Master 25-Pillar Domain Extractor (`collect_domains.py`): provides curated foundational knowledge across NCERT, Law & Constitution, SMT/PCB, Quality (QMS/FA), Tech/CS, Corporate Emails, Cinema, Everyday Math with `<soch>`, All Indian & Foreign Cuisines, 3-Way Translation (IITB/IndicTrans), Stories & Literature (Premchand/Panchatantra), Movie Plots & Songs/Lyrics (Drishyam/Lagaan/Kishore/Gulzar), Mobile Tech (Android/iOS/ADB/DFU), and Jokes, Stand-up & Desi Street Roasts/Slangs.
272
  2. High-Throughput Kaggle Collector (`kaggle_collector.py`): streams raw open corpora (including `cfilt/iitb-english-hindi` translation parallel corpus), performs in-flight 64-bit integer hash deduplication and script filtering, packs into 100k-row Parquet shards, uploads to `ViuAI/viu-mini-raw-pretrain`, and immediately deletes local files to preserve Kaggle disk (<1GB).
273
  - Result:
274
  - Collector initialized stream `domains`, hit limit of 10 rows.
 
268
  - Cmds:
269
  - `python data/scripts/kaggle_collector.py --dry-run --stream domains --limit 10`
270
  - Upgrades Covered:
271
+ 1. Master 26-Pillar Domain Extractor (`collect_domains.py`): provides curated foundational knowledge across NCERT, Law & Constitution, SMT/PCB, Quality (QMS/FA), Tech/CS, Corporate Emails, Cinema, Everyday Math with `<soch>`, All Indian & Foreign Cuisines, 3-Way Translation (IITB/IndicTrans), Stories & Literature (Premchand/Panchatantra), Movie Plots & Songs/Lyrics (Drishyam/Lagaan/Kishore/Gulzar), Mobile Tech (Android/iOS/ADB/DFU), Jokes, Stand-up & Desi Street Roasts/Slangs, and Complete Grammar & Language Mechanics (Hindi Vyakaran, English Grammar, Hinglish Syntax).
272
  2. High-Throughput Kaggle Collector (`kaggle_collector.py`): streams raw open corpora (including `cfilt/iitb-english-hindi` translation parallel corpus), performs in-flight 64-bit integer hash deduplication and script filtering, packs into 100k-row Parquet shards, uploads to `ViuAI/viu-mini-raw-pretrain`, and immediately deletes local files to preserve Kaggle disk (<1GB).
273
  - Result:
274
  - Collector initialized stream `domains`, hit limit of 10 rows.