ViuAI commited on
Commit
e1dae91
·
verified ·
1 Parent(s): 2c8f401

feat/docs: sync dedicated toxicity folder script & docs for docs/PROGRESS.md

Browse files
Files changed (1) hide show
  1. docs/PROGRESS.md +9 -10
docs/PROGRESS.md CHANGED
@@ -125,16 +125,15 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
125
  - [x] 2026-09-19: Translation & Hinglish Massive Scale-Up (17:58 IST):
126
  1. **AI4Bharat Samanantar Translation Ingested**: All 8 Parquet shards (1.83 GB, 10,125,706 parallel English-Hindi sentence pairs) from `ai4bharat/samanantar` ingested into `translation/`. Total translation corpus jumped from 252 MB to **2.03 GB across 21 shards (~1.17 Crore sentence pairs, 8x increase)**.
127
  2. **Curated Hinglish Dialogue & Instruction Sets Ingested**: 9 curated multi-turn datasets (`NebulaByte/alpaca-gpt4`, `BhabhaAI/Cross-chat`, `one-thing/chatbot-arena`, `ShareGPT-Hinglish`, `Dolly15k-Hinglish`, `LIMA-Hinglish`, `rvv-karma/English-Hinglish-TOP`) ingested into `hinglish/`. Total Hinglish jumped from 53 MB to **202.61 MB across 10 files (1.5M+ rich conversational rows, 4x increase)**.
128
- - [x] 2026-09-19: 90GB+ / 70 Billion Token Landmark Smashed (21:25 IST Audit):
129
- 1. Live Hub audit confirmed **91.81 GB** total dataset on `ViuAI/viu-mini-raw-pretrain` across **431 Parquet shards** (~260+ GB uncompressed raw text).
130
- 2. Total tokens accumulated: **~68 Billion to 70 Billion Tokens**!
131
- 3. English (`english/` FineWeb-Edu): 36 shards (**56.10 GB**, ~2.1 Crore educational articles, ~40B tokens).
132
- 4. Hindi (`hindi/` IndicCorpV2): 319 shards (**22.64 GB**, 7.6 Crore clean sentences, ~18B tokens).
133
- 5. Wikipedia (`wikipedia/`): 41 shards (**10.83 GB**, 64.5 Lakh complete articles, ~5B tokens).
134
- 6. Translation (`translation/`): 21 shards (**2.03 GB**, 1.17 Crore parallel English-Hindi pairs, ~2.5B tokens).
135
- 7. Hinglish (`hinglish/`): 10 files (**202.61 MB**, 15 Lakh+ dialogues, ~400M tokens).
136
- 8. Domains (`domains/`): 3 shards (Comprehensive 8D, SMT, History, Literature, Vyakaran, ADB).
137
- 9. Pretraining Data Ingestion is now in extreme over-training territory for a 242M parameter model (over 280 tokens per parameter ratio!). Next: Pretraining launch on rental GPU.
138
  - [ ] Next: Prepare pretraining streaming dataset loader in `train.py` and finalize pretraining launch config.
139
 
140
  ## Kaise note likhna hai
 
125
  - [x] 2026-09-19: Translation & Hinglish Massive Scale-Up (17:58 IST):
126
  1. **AI4Bharat Samanantar Translation Ingested**: All 8 Parquet shards (1.83 GB, 10,125,706 parallel English-Hindi sentence pairs) from `ai4bharat/samanantar` ingested into `translation/`. Total translation corpus jumped from 252 MB to **2.03 GB across 21 shards (~1.17 Crore sentence pairs, 8x increase)**.
127
  2. **Curated Hinglish Dialogue & Instruction Sets Ingested**: 9 curated multi-turn datasets (`NebulaByte/alpaca-gpt4`, `BhabhaAI/Cross-chat`, `one-thing/chatbot-arena`, `ShareGPT-Hinglish`, `Dolly15k-Hinglish`, `LIMA-Hinglish`, `rvv-karma/English-Hinglish-TOP`) ingested into `hinglish/`. Total Hinglish jumped from 53 MB to **202.61 MB across 10 files (1.5M+ rich conversational rows, 4x increase)**.
128
+ - [x] 2026-09-19: Dedicated Toxicity & Profanity Lexicon Folder Ingestion (21:40 IST):
129
+ 1. **New Dedicated Folder Created (`toxicity/`)**: Created and uploaded `toxicity/train-00000.parquet` to `ViuAI/viu-mini-raw-pretrain`.
130
+ 2. **Comprehensive Hindi, Hinglish & English Profanity Taxonomy**:
131
+ - Hardcore familial & sexualized slurs (MC, BC, TMKC, BKL, betichod, chutiya, lund, gaand, randi, bhosdike, etc. with semantic roots and severity tiers 1 to 5).
132
+ - English hardcore profanities (C-word, F-word, M-word, asshole, dickhead, bastard, etc.).
133
+ - Regional Indian cuss dialects (Mumbai Bambaiya, Delhi/NCR, Punjabi, UP/Bihar).
134
+ - Pragmatic context classification: Distinguishing friendly peer banter ("kaisa hai re harami") vs malicious harassment attacks.
135
+ - Conversational Self-Defense & Counter-Roast protocols: Razor-sharp, witty, dignified shutdowns for models when abused by hostile users.
136
+ 3. **Updated Grand Total**: **93.36 GB across 433 files (~72 Billion Tokens)** locked on Hub!
 
137
  - [ ] Next: Prepare pretraining streaming dataset loader in `train.py` and finalize pretraining launch config.
138
 
139
  ## Kaise note likhna hai