feat/docs: sync dedicated toxicity folder script & docs for docs/PROGRESS.md
Browse files- docs/PROGRESS.md +9 -10
docs/PROGRESS.md
CHANGED
|
@@ -125,16 +125,15 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
|
|
| 125 |
- [x] 2026-09-19: Translation & Hinglish Massive Scale-Up (17:58 IST):
|
| 126 |
1. **AI4Bharat Samanantar Translation Ingested**: All 8 Parquet shards (1.83 GB, 10,125,706 parallel English-Hindi sentence pairs) from `ai4bharat/samanantar` ingested into `translation/`. Total translation corpus jumped from 252 MB to **2.03 GB across 21 shards (~1.17 Crore sentence pairs, 8x increase)**.
|
| 127 |
2. **Curated Hinglish Dialogue & Instruction Sets Ingested**: 9 curated multi-turn datasets (`NebulaByte/alpaca-gpt4`, `BhabhaAI/Cross-chat`, `one-thing/chatbot-arena`, `ShareGPT-Hinglish`, `Dolly15k-Hinglish`, `LIMA-Hinglish`, `rvv-karma/English-Hinglish-TOP`) ingested into `hinglish/`. Total Hinglish jumped from 53 MB to **202.61 MB across 10 files (1.5M+ rich conversational rows, 4x increase)**.
|
| 128 |
-
- [x] 2026-09-19:
|
| 129 |
-
1.
|
| 130 |
-
2.
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
9. Pretraining Data Ingestion is now in extreme over-training territory for a 242M parameter model (over 280 tokens per parameter ratio!). Next: Pretraining launch on rental GPU.
|
| 138 |
- [ ] Next: Prepare pretraining streaming dataset loader in `train.py` and finalize pretraining launch config.
|
| 139 |
|
| 140 |
## Kaise note likhna hai
|
|
|
|
| 125 |
- [x] 2026-09-19: Translation & Hinglish Massive Scale-Up (17:58 IST):
|
| 126 |
1. **AI4Bharat Samanantar Translation Ingested**: All 8 Parquet shards (1.83 GB, 10,125,706 parallel English-Hindi sentence pairs) from `ai4bharat/samanantar` ingested into `translation/`. Total translation corpus jumped from 252 MB to **2.03 GB across 21 shards (~1.17 Crore sentence pairs, 8x increase)**.
|
| 127 |
2. **Curated Hinglish Dialogue & Instruction Sets Ingested**: 9 curated multi-turn datasets (`NebulaByte/alpaca-gpt4`, `BhabhaAI/Cross-chat`, `one-thing/chatbot-arena`, `ShareGPT-Hinglish`, `Dolly15k-Hinglish`, `LIMA-Hinglish`, `rvv-karma/English-Hinglish-TOP`) ingested into `hinglish/`. Total Hinglish jumped from 53 MB to **202.61 MB across 10 files (1.5M+ rich conversational rows, 4x increase)**.
|
| 128 |
+
- [x] 2026-09-19: Dedicated Toxicity & Profanity Lexicon Folder Ingestion (21:40 IST):
|
| 129 |
+
1. **New Dedicated Folder Created (`toxicity/`)**: Created and uploaded `toxicity/train-00000.parquet` to `ViuAI/viu-mini-raw-pretrain`.
|
| 130 |
+
2. **Comprehensive Hindi, Hinglish & English Profanity Taxonomy**:
|
| 131 |
+
- Hardcore familial & sexualized slurs (MC, BC, TMKC, BKL, betichod, chutiya, lund, gaand, randi, bhosdike, etc. with semantic roots and severity tiers 1 to 5).
|
| 132 |
+
- English hardcore profanities (C-word, F-word, M-word, asshole, dickhead, bastard, etc.).
|
| 133 |
+
- Regional Indian cuss dialects (Mumbai Bambaiya, Delhi/NCR, Punjabi, UP/Bihar).
|
| 134 |
+
- Pragmatic context classification: Distinguishing friendly peer banter ("kaisa hai re harami") vs malicious harassment attacks.
|
| 135 |
+
- Conversational Self-Defense & Counter-Roast protocols: Razor-sharp, witty, dignified shutdowns for models when abused by hostile users.
|
| 136 |
+
3. **Updated Grand Total**: **93.36 GB across 433 files (~72 Billion Tokens)** locked on Hub!
|
|
|
|
| 137 |
- [ ] Next: Prepare pretraining streaming dataset loader in `train.py` and finalize pretraining launch config.
|
| 138 |
|
| 139 |
## Kaise note likhna hai
|