feat/docs: sync stories & uncensored scripts & docs for docs/PROGRESS.md
Browse files- docs/PROGRESS.md +9 -1
docs/PROGRESS.md
CHANGED
|
@@ -130,7 +130,15 @@ Har kaam ke baad yaha entry karo taaki yaad rahe kab kya kiya tha.
|
|
| 130 |
2. **Dual-Type Toxicity & Profanity Architecture (Curated + Open-Source)**:
|
| 131 |
- **Type 1: Deep Linguistic Taxonomy & Self-Defense Blueprint**: Hardcore familial & sexualized slurs (MC, BC, TMKC, BKL, betichod, chutiya, lund, gaand, randi, bhosdike, etc. with semantic roots and severity tiers 1 to 5), English hardcore profanities (C-word, F-word, M-word, asshole), regional cuss dialects, friendly banter vs hostile harassment context classification, and savage self-defense protocols (`toxicity/train-00000.parquet`).
|
| 132 |
- **Type 2: Open-Source In-the-Wild Internet Datasets**: Ingested 6 real-world open-source datasets (Google Civil Comments 2M toxic/obscene comments, Davidson Hate Speech, Prism Hinglish Hate Speech, HASOC Hindi Offensive text, HateCheck Hindi) totaling **373.61 MB across 7 files** in `toxicity/`.
|
| 133 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
- [ ] Next: Prepare pretraining streaming dataset loader in `train.py` and finalize pretraining launch config.
|
| 135 |
|
| 136 |
## Kaise note likhna hai
|
|
|
|
| 130 |
2. **Dual-Type Toxicity & Profanity Architecture (Curated + Open-Source)**:
|
| 131 |
- **Type 1: Deep Linguistic Taxonomy & Self-Defense Blueprint**: Hardcore familial & sexualized slurs (MC, BC, TMKC, BKL, betichod, chutiya, lund, gaand, randi, bhosdike, etc. with semantic roots and severity tiers 1 to 5), English hardcore profanities (C-word, F-word, M-word, asshole), regional cuss dialects, friendly banter vs hostile harassment context classification, and savage self-defense protocols (`toxicity/train-00000.parquet`).
|
| 132 |
- **Type 2: Open-Source In-the-Wild Internet Datasets**: Ingested 6 real-world open-source datasets (Google Civil Comments 2M toxic/obscene comments, Davidson Hate Speech, Prism Hinglish Hate Speech, HASOC Hindi Offensive text, HateCheck Hindi) totaling **373.61 MB across 7 files** in `toxicity/`.
|
| 133 |
+
- [x] 2026-09-19: Dedicated Stories & Uncensored Folders Ingestion (22:15 IST):
|
| 134 |
+
1. **New `stories/` Folder Created (2 files, 237.22 MB)**:
|
| 135 |
+
- English: Premier Microsoft Research `roneneldan/TinyStories` narrative coherent story shard (`stories/tinystories_train_00000.parquet`).
|
| 136 |
+
- Hindi & English: Complete classical Panchatantra, Akbar-Birbal, and Vikram-Betal wisdom stories (`stories/indian_classics_hindi_english.parquet`).
|
| 137 |
+
2. **New `uncensored/` Folder Created (3 files, 288.39 MB)**:
|
| 138 |
+
- Eric Hartford's Wizard Vicuna 70k Unfiltered (`uncensored/wizard_vicuna_70k_unfiltered.json`, 144.5 MB).
|
| 139 |
+
- Jon Durbin's Airoboros Uncensored Multi-turn Dialogues (`uncensored/airoboros_instructions_uncensored.jsonl`, 304.2 MB).
|
| 140 |
+
- Curated Unfiltered Direct Persona & Gritty Creative Noir Fiction (`uncensored/unfiltered_direct_qa.parquet`).
|
| 141 |
+
3. **Updated Grand Total**: **94.24 GB across 444 files (~74 Billion Tokens)** locked on Hub!
|
| 142 |
- [ ] Next: Prepare pretraining streaming dataset loader in `train.py` and finalize pretraining launch config.
|
| 143 |
|
| 144 |
## Kaise note likhna hai
|