Hugging Face's logo

ViuAI
/
ViuMini-Dense-360M

Text Generation
PyTorch
Hindi
English
dense
transformer
indic
hindi
hinglish
english
gqa
gemma-4
Model card Files
xet
 Community
ViuMini-Dense-360M / docs /DATA_SOURCES.md
ViuAI's picture
ViuAI
sync local: enforce_mix sampler, tokenizer v2 (mask fix + 35/30/35), mix optional docs, repair scripts
991af26 verified
preview code
|
Raw History Blame Contribute Delete
19.1 kB

DATA SOURCES — Step 4 (Master 30B–50B+ Token Cloud Ingestion Plan)

Locked Target: 30B – 50B+ Tokens (Extreme Over-Training) across 20 Exhaustive Pillars. Language Ratio (OPTIONAL rule, default OFF): 40% Hinglish (Roman) / 30% Hindi (Devanagari) / 30% English. Training me model/configs/train_config.yaml (enforce_mix: false default, weights lang_mix se) aur tokenizer me --mix/--no-mix se control hota hai. Dataset Destination: ViuAI/viu-mini-raw-pretrain (Hugging Face Dataset Hub). Execution: 100% on Kaggle Cloud (Zero Local PC disk/memory usage).

1. Core Data Policy: Zero Sub-Sampling (Full Dataset Ingestion)

POLICY LOCK (2026-09-18): Agar Hugging Face par koi verified dataset 10B, 20B ya 50B tokens ka hai, toh hum usme se artificial sub-sample (e.g. sirf 5B) nahi kaatenge. Pura ka pura dataset 100% ingest karke sharded Parquet/JSONL format me repo me store kiya jayega.

2. Language Streams & Source Repositories

A. Hinglish Roman (40% — ~12B to 20B+ Tokens)

  1. sk-community/romanized_hindi (~1.82M rows, MIT) — Ingest ALL rows.
  2. cfilt/HiNGE + midas-research/hinglish — Full conversational sets.
  3. Romanized Transliteration Streams: Hindi Wikipedia & Sangraha subsets transformed to high-speed Romanized conversational Hinglish.
  4. Conversational Social Dumps: Cleaned, filtered Hinglish web forums, dialogue corpora, youth chats.
  5. 20 Pillars Synthetic Hinglish Guides & CoT Reasoning (<soch> ... </soch> format).

B. Hindi Devanagari (30% — ~9B to 15B+ Tokens)

  1. ai4bharat/IndicCorpV2 (config indiccorp_v2, split hin_Deva, ~152M rows) — Ingest ALL rows.
  2. ai4bharat/sangraha (verified/hin splits, ~250M sentences) — Ingest ALL rows.
  3. Hindi Wikipedia Dump (wikimedia/wikipedia split hi) — Complete articles.
  4. Official Bilingual Legal/Govt Texts (IndiaCode, Constitution of India Hindi, NCERT Hindi 1-12).

C. English (30% — ~9B to 15B+ Tokens)

  1. HuggingFaceFW/fineweb-edu (Top educational web corpus) — Ingest complete high-quality shards.
  2. English Wikipedia Dump — Complete educational articles.
  3. roneneldan/TinyStories — Full dataset for fundamental narrative coherence and logic.
  4. Technical & Engineering Documentation (Python, Linux, SMT/PCB, Quality standards, Legal acts).

3. The 20 Exhaustive Knowledge Pillars

  1. ALL NCERT (Classes 1–12 Complete):
    • Science (Classes 6-10 General; Classes 11-12 Physics, Chemistry, Biology complete chapters).
    • Mathematics (Classes 1-10 complete theory, formulas, proofs, solved examples).
    • Social Science: History, Geography, Civics/Polity, Economics (Classes 6-12).
    • Computer Science & Informatics Practices (Python, SQL, Networking).
  2. Complete Indian History (Prachin se Aaj Tak):
    • Ancient (Indus Valley, Vedic, Maurya, Gupta, Sangam, Cholas, Pallavas).
    • Medieval (Delhi Sultanate, Bhakti/Sufi movements, Vijayanagara, Mughals, Marathas, Sikhs).
    • Modern & Freedom Movement (1857, Gandhi, Bhagat Singh, Netaji, Patel, Partition, Constitution).
    • Post-Independence (Wars, Space/ISRO, Economic reforms 1991).
  3. Constitution & Lawyer-Level Law (A to Z Kanoon):
    • Full Constitution: Preamble, all Articles (1-395+), Fundamental Rights (12-35), Writs (Art 32), Amendments.
    • BNS / IPC, BNSS / CrPC (FIR, Zero FIR, Arrest rights, Bailable/Non-bailable, Bail types).
    • BSA / Evidence Act, Consumer Protection Act 2019, RTI Act 2005, IT Act 2000 (Cyber Crime).
    • Cheque bounce (Sec 138 NI Act), Motor Vehicles Act, Family & Property laws (ancestral vs self-acquired).
    • Practical Lawyer Drafting: Legal notices, complaint letters, consumer court filings.
  4. Electronics Hardware & SMT Manufacturing (Deep Engineering):
    • PCB Fabrication: Layers (1 to 32+), Substrates (FR-4, Rogers, Aluminum), Copper weight (0.5-2 oz), Trace width/clearance, Vias (Through-hole, Blind, Buried, Micro-via HDI), Surface Finishes (HASL, ENIG, OSP, Immersion Tin/Silver), Gerber files (RS-274X), IPC-2221, IPC-A-600.
    • SMT Line: Stencils, Solder paste (SAC305 vs SnPb, Type 3/4), SPI (3D height/volume), Pick & Place, Reflow profiling (4-zone KIC/Datapaq), Nitrogen reflow, AOI, AXI (BGA/QFN void inspection).
    • SMT Defects & RCA: Tombstoning, bridging, solder balls, voiding, cold joints, Head-in-Pillow (HiP).
    • MI (Manual Insertion / THT): Wave soldering (fluxer, preheat, laminar wave, pot temp 255-265C), selective soldering, hand soldering (IPC-7711/7721).
  5. Industrial Quality Management Systems (QMS & Standards):
    • International Standards: ISO 9001:2015, IATF 16949 (Automotive), ISO 13485 (Medical), ESD S20.20.
    • Automotive 5 Core Tools: APQP (Phases 1-5), FMEA (DFMEA/PFMEA), PPAP (18 elements), MSA (Gage R&R <10%), SPC (Cp, Cpk >= 1.33/1.67, X-bar R charts).
    • Shop Floor Quality: IQC (Sampling MIL-STD-105E AQL), IPQC, OQC/OQA.
    • Continuous Improvement: 5S, Kaizen, Poka-Yoke, CAPA, 7 QC Tools.
    • Workmanship: IPC-A-610 Class 1, Class 2, Class 3 criteria.
  6. Failure Analysis (FA) & Reliability:
    • 8D Methodology (D0 to D8), 5-Whys, Ishikawa Fishbone, Fault Tree Analysis (FTA).
    • NDT: 3D X-ray CT, Scanning Acoustic Microscopy (CSAM), Thermal Imaging.
    • Destructive Testing: Microsectioning (IMC Cu6Sn5/Cu3Sn thickness), SEM + EDX elemental analysis, Dye & Pry.
    • Failure Modes: ESD, EOS, Solder joint fatigue, Dendrites/ECM, Tin whiskers, Creep corrosion.
  7. Complete Tech, CS & Programming (Software Engineering):
    • Hardware & Systems: CPU architecture, RAM, GPU (CUDA/Tensor Cores), Linux kernel & bash scripting.
    • Networking: OSI 7-layer model, TCP/IP, DNS, IPv4/IPv6, HTTP/HTTPS, REST, WebSockets.
    • Programming: Python (OOP, Asyncio, Generators), HTML5/CSS3/JavaScript, SQL (PostgreSQL/MySQL), Git/GitHub, Docker, Nginx.
    • Cybersecurity: Hashing (SHA-256), Encryption (AES/RSA), OWASP Top 10, Phishing, 2FA.
  8. Professional Email & Corporate Communications:
    • HR & Career: Leave applications, Cover letters, Resignations, Salary negotiations, Self-appraisals.
    • Operations: Minutes of Meeting (MOM), Escalation emails, PO, RFQ, Vendor follow-up, Apology & CAPA letters.
  9. Cinema & All Movies (Entertainment & Culture):
    • Bollywood, South Indian Cinema (Telugu, Tamil, Malayalam, Kannada), Hollywood, World Cinema.
    • Plots, iconic dialogues, directors, actors, National Film Awards, Oscars, trivia.
  10. Everyday Math, Commercial Hisaab & Logic:
    • Arithmetic, Dukan Hisaab, Profit/Loss, GST, Unitary Method, Speed-Distance-Time, Train problems, Percentages, Puzzles with step-by-step thinking (<soch> ... </soch>).
  11. Personal Finance, Banking, Tax & Investment:
    • Banking, CIBIL score improvement, Home/Personal loans, Income Tax (New vs Old, ITR-1/4, TDS, Form 16), Mutual Funds/SIP, Stock Market, EPF, Gold, Insurance.
  12. Healthcare, Medicine, First Aid & Wellness:
    • First Aid (Burns, fractures, CPR), Common ailments, Lifestyle diseases (Diabetes, BP, Thyroid), Medical lab test ranges (CBC, LFT, KFT), Ayurveda & Desi Nuskhe.
  13. Sports, Cricket & Athletics:
    • Cricket (World Cups, IPL, Rules, Records, Legends), Olympics, Football, Pro Kabaddi, Kushti.
  14. Competitive Exams & Sarkari Naukri Guidance:
    • UPSC CSE, SSC CGL/CHSL, Banking (IBPS/SBI), Railways, State PCS, syllabus, reasoning & aptitude tricks.
  15. Automobile, Two-Wheelers & Daily Maintenance:
    • Petrol/Diesel/CNG vs EV, Engine oil, Coolant, Mileage tips, Troubleshooting, RTO rules (DL, RC, NOC, PUC).
  16. Rasoi, Cooking & Global Nutrition (All Indian & Foreign Cuisines):
    • Complete Indian Regional Cuisines: North Indian (Dal Makhani, Shahi Paneer, Biryani dum), South Indian (Dosa/Idli fermentation, Sambar, Rasam), West Indian (Dhokla, Dal Baati Churma), East Indian (Machher Jhol, Panch Phoron, Mishti Doi).
    • Complete Foreign & International Cuisines: Italian (Neapolitan Pizza dough fermentation, Pasta al dente, Carbonara emulsion, Risotto), Pan-Asian & Chinese (Wok Hei high-heat searing, Kung Pao, Dim Sum, Ramen dashi broth), Mexican (Masa Harina corn tortillas, Guacamole, Salsa Verde, Carnitas), Middle Eastern (Creamy Tahini Hummus, Falafel from raw chickpeas, Shawarma marinades), French Continental (5 Mother Sauces: Béchamel, Velouté, Espagnole, Sauce Tomat, Hollandaise).
    • Culinary Science: Emulsification, baking hydration ratios, maillard reaction, spice & herb pairings, dietary nutrition (protein calculation, calories).
  17. Soft Skills & Interview Preparation:
    • HR interview questions, Public speaking, Body language, Confidence, Time management (Pomodoro).
  18. World Geography & International Affairs (Geopolitics):
    • 7 Continents, Oceans, Capitals, Currencies, UN, WHO, G20, BRICS, NATO, World War 1 & 2, Cold War.
  19. Bhartiya Sanskriti, Sangeet, Kala & Dharam:
    • Indian Classical Music (Hindustani, Carnatic, Ragas, Instruments), Folk Arts, Multi-faith teachings (Geeta, Gurubani, Sufi, Jain, Buddhist).
  20. Massive Bol-Chal Hinglish & Dialogue:
    • Natural colloquial dialogue, youth chat, workplace talk, slang-free conversational flows.
  21. Complete Multilingual Translation & Transliteration (Anuvad & Lipyantaran):
    • 3-Way Parallel Corpora: English $\leftrightarrow$ Hindi (Devanagari) $\leftrightarrow$ Hinglish (Roman).
    • Sources: cfilt/iitb-english-hindi (IITB 1.6M parallel pairs), ai4bharat/IndicTrans2, sk-community/romanized_hindi.
    • Cross-lingual terminology across Legal, SMT/PCB, Tech, and Medical domains.
  22. Kahaniyan, Stories & Creative Sahitya:
    • Prachin & Prernadayak: Panchatantra, Hitopadesh, Jataka Tales, Akbar-Birbal, Tenali Raman, Vikram-Betal.
    • Amar Sahitya: Munshi Premchand (Idgah, Godan, Kafan, Namak ka Daroga), Harivansh Rai Bachchan, Dinkar.
    • Short fiction, moral stories, mystery & suspense (TinyStories format).
  23. Detailed Movie Stories, Full Plots, Songs, Lyrics & Ghazals:
    • Full Movie Plots: Drishyam, Lagaan, Sholay, 3 Idiots, Dangal, Baahubali, KGF, RRR, Inception, Interstellar.
    • Sangeet & Filmy Gaane: Kishore Kumar, Mohammed Rafi, Lata Mangeshkar, Mukesh, R.D. Burman, Arijit Singh, A.R. Rahman.
    • Poetry & Ghazals: Gulzar, Sahir Ludhianvi, Mirza Ghalib, Jagjit Singh ki ghazals aur sher-o-shayari.
  24. Mobile Tech: Android, iOS, ADB/Fastboot & Smartphone Engineering:
    • Android: Linux Kernel, HAL, ART (Ahead-of-Time & JIT compilation), APK architecture, Permissions, Play Services.
    • Developer Tools: USB Debugging, OEM unlock, ADB commands (adb devices, adb reboot bootloader, bloatware uninstaller), Fastboot flashing, Recovery mode (wipe cache/factory reset).
    • iOS / Apple Ecosystem: iOS architecture, Secure Enclave, iCloud, AirDrop, DFU mode restore, battery health optimization (80% rule).
    • Everyday Repairs & Connectivity: Phone heating fix, storage cleanup, eSIM vs physical SIM, APN network settings, phone migration.
  25. Desi Jokes, Stand-up Humor, Memes, Relatable Roasts & Street Slangs:
    • Relatable Indian Humor: College/hostel banter, corporate employee memes, rishtedaar/shaadi jokes, doctor-patient humor.
    • Stand-up Comedy & Witty Comebacks: Observational humor (Zakir Khan / Bassi style punchlines).
    • Colloquial Street Hinglish & Playful Roasting: Street slangs (dhakkan, chirkut, bakchodi, dimaag ki dahi, hawa-bazi, champu, fukra), sharp sarcastic comebacks, and friendly desi banter.
  26. Complete Grammar & Language Mechanics (Vyakaran & Syntax):
    • Sampoorn Hindi Vyakaran: Sangya, Sarvanam, Karak ('Ne' rule ergativity concord: 'Ram ne roti khayi'), Ling & Vachan rules, 12 Kaal/Tenses, Vakya Shuddhi ('anek' vs 'aneko'), Sandhi, Samas, Muhavare & Lokoktiyan.
    • English Grammar & Mechanics: 8 Parts of Speech, Subject-Verb Agreement concord, Active & Passive Voice transformations, Direct & Indirect Speech narration, 12 Tenses.
    • Hinglish Syntax & Natural Code-Switching: Verb pairing with auxiliary 'karna'/'hona', gender agreement with loanwords ('battery drain ho rahi hai', 'server restart ho raha hai').

4. Cloud Streaming & Sharding Architecture 

[Open Datasets on HF / Web] 
             │ (Streaming Generator)
             ▼
[Kaggle Cloud Worker]
   ├── Clean & Script Check
   ├── 64-bit Integer Hash-Dedup
   ├── Accumulate 100k rows
   ├── Write /tmp/shard_00000.parquet
   ├── Upload via HfApi to ViuAI/viu-mini-raw-pretrain
   └── Delete local shard immediately (<1GB disk used)
             │
             ▼
[HF Dataset Repo: ViuAI/viu-mini-raw-pretrain] (8TB Free Capacity)
   ├── hindi/train-shard-00000.parquet ...
   ├── english_fixed/train-00000.parquet ...
   ├── hindi_fixed/train-00000.parquet ...
   ├── hinglish/train-*.parquet ...
   └── domains/train-*.parquet ...

5. Expansion Roadmap: P1 & P2 Missing Knowledge Pillars (2026-09-23)

Priority 1 (P1) — Core Missing Pillars & Hinglish Boosters

  1. Sports / Cricket (domains/sports_cricket):
    • Cricket rules (LBW, DRS, DLS, Powerplay), IPL, T20/ODI/Test records, player stats, commentary lingo ("kya shot mara hai", "clean bowled", "googly").
    • Mix: 60% Hinglish, 40% Hindi.
  2. Sarkari Naukri (domains/sarkari_naukri):
    • UPSC Civil Services, SSC CGL/CHSL, Railway RRB NTPC/Group D, Banking (IBPS/SBI).
    • Syllabus, exam patterns, eligibility, GK/Current Affairs, reasoning/aptitude questions, notifications.
    • Mix: 50% Hindi, 50% Hinglish.
  3. Automobile / RTO (domains/automobile_rto):
    • Learner & Permanent Driving License (DL) process, Sarathi Parivahan, RC transfer, NOC, fitness.
    • Motor Vehicles Act traffic rules, fines/e-challans, vehicle maintenance (engine oil, brakes, tyre pressure), Electric Vehicles (EV) charging & battery care.
    • Mix: 60% Hinglish, 40% Hindi.
  4. NCERT SST Hindi (education/ncert_sst_hindi):
    • Classes 6 to 12 Complete: Itihas (Ancient, Medieval, Modern), Bhugol (Indian & World Geography), Nagrik Shastra/Rajneeti Vigyan (Civics & Indian Polity), Arthashastra (Economics).
    • Mix: 100% Devanagari Hindi.
  5. Cyber-Fraud Awareness (domains/cyber_fraud_awareness):
    • Online scam prevention: OTP fraud, UPI/QR code scams, lottery/job scams, Digital Arrest scams, fake loan apps.
    • Actionable guides: National Cyber Crime Helpline (1930), cybercrime.gov.in complaint process, bank account freezing.
    • Mix: 60% Hinglish, 40% Hindi.
  6. Tyohar Practical (domains/tyohar_practical):
    • Major Indian festivals: Diwali, Holi, Eid, Chhath Puja, Raksha Bandhan, Durga Puja, Makar Sankranti, Ganesh Utsav.
    • History, cultural significance, puja vidhi, dates, traditional sweets/recipes, family customs.
    • Mix: 50% Hindi, 50% Hinglish.
  7. Health Hinglish Dialogues (dialogues/health_hinglish):
    • Doctor-patient consultations: common symptoms (bukhar, sardi, pet dard, BP, sugar), diagnosis talk.
    • Home remedies (gharelu nuskhe), medication precautions, lifestyle, diet & fitness advice in natural conversational Hinglish.
    • Mix: 90% Conversational Hinglish, 10% Hindi.
  8. Interview & Jobs Hinglish (dialogues/interview_jobs_hinglish):
    • Mock HR interviews: "Tell me about yourself", salary negotiation, career gaps, strengths/weaknesses.
    • Technical interview QA, resume formatting, corporate email etiquette, workplace communication.
    • Mix: 80% Conversational Hinglish, 20% English.

Priority 2 (P2) — Practical Utilities & Public Services

  1. E-commerce Support (domains/ecommerce_support):
    • Customer care chats: order tracking, delayed delivery, return/replacement, refund status, damaged product complaints.
    • Mix: 70% Hinglish, 30% English.
  2. IRCTC / Railway Utilities (domains/irctc_railway):
    • Ticket booking guides (General, Tatkal, Premium Tatkal), PNR status, cancellation & refund rules, RAC vs WL, chart timing, luggage rules.
    • Mix: 60% Hinglish, 40% Hindi.
  3. Real Estate (domains/real_estate):
    • Buying & renting property, registry process, stamp duty, tenant verification, rent agreement clauses, home loan process, RERA rights.
    • Mix: 50% Hindi, 50% Hinglish.
  4. Constitution Full Articles (legal/constitution_articles):
    • Articles 1 to 395+ complete bilingual text (Hindi & English), Preamble, Fundamental Rights (Art 12-35), Directive Principles, Writs, Major Amendments.
    • Mix: 50% Hindi, 50% English.

5.1 Real Public Datasets Registry (100% Ingestion, ZERO Sub-Sampling)

Pillar Topic Source HF Repository Real Rows Available Format & Content
1. Sports/Cricket Cricket commentary & Wiki nirmalkumar/cricket-commentary + Ankush-Chander/cricket-wiki 165,700 Ball-by-ball commentary (Hinglish/English) + encyclopedic wiki
2. Sarkari Naukri SSC CGL, UP CET & UPSC FAQ sharad461/SSC-CGL-2023-hi + rmahesh/UP_CET_Hindi_exams + prnv19/UPSC_FAQ 3,100+ Official Hindi exam papers with options, answers & UPSC rules
3. Automobile/RTO Traffic law & fines QA Pravincoder/Indian_traffic_law_QA 200+ Indian Motor Vehicles Act, DL process, fines
4. NCERT SST Hindi History, Geography, Civics kaifahmad/indian-history-hindi-QA-3.4k + oss-codes/NCERT-Conversational-Dataset-Indic 265,800 Full NCERT textbook dialogues & Indian history QA in Hindi
5. Cyber-Fraud Scam prevention & reporting MaestroDev19/Cyber-fraud-guide 16,000 Phishing, banking fraud, OTP scams, cybercrime helpline
6. Tyohar Practical Festivals & culture deepkaria/indian-culture-dataset 506 Ministry of Culture festival guides & customs
7. Health Hinglish Consultations & doctor QA knowrohit07/know_medical_dialogue_v2 + mahfoos/Patient-Doctor-Conversation 9,600 Doctor-patient dialogues, symptoms, consultations
8. Interview/Jobs HR, Mock interview & jobs AI-Mock-Interviewer/Train_data + shristi79/Popular_Interview_questions_and_answers 7,025 Mock interview turns, coding and behavioral QA
9. E-Commerce Order support & queries bitext/Bitext-customer-support-llm-chatbot-training-dataset 26,800 Real customer service tickets, returns, cancellations
10. IRCTC/Railway Passenger complaints & train QA faizmubeen/indian_railways_complaint_tweets 4,500 Real passenger complaints & queries (@RailMinIndia)
11. Real Estate Property laws & legal QA mratanusarkar/Indian-Laws + Alok2304/Indian_Property_Law 34,700 Indian property law, RERA, registration, legal QA
12. Constitution Full constitutional articles nisaar/Constitution_of_India 933 Complete articles and provisions of the Constitution

Ingestion Rule: 100% of rows from every dataset are ingested without arbitrary max_rows cutoffs. Automatically packed into 100k row shards (`150MB) matching domains/train-p1p2_-.parquet`.