File size: 3,640 Bytes
991af26 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
language:
- hi
- en
- multilingual
language_bcp47:
- hi-Latn
license: apache-2.0
task_categories:
- text-generation
tags:
- hinglish
- hindi
- pretraining
- code
- science
- legal
- distillation
- reasoning
pretty_name: Viu Mini Raw Pretrain
size_categories:
- 100M<n<1B
configs:
- config_name: default
data_files:
- split: train
path:
- hindi/train-*.parquet
- hindi_fixed/train-*.parquet
- english_fixed/train-*.parquet
- hinglish/*.parquet
- domains/*.parquet
- distilled/*.parquet
- translation/train-*.parquet
- translation/samanantar-*.parquet
- wikipedia/train-*.parquet
- science/*.parquet
- math/*.parquet
- code/*.parquet
- legal/train-*.parquet
- health/*.parquet
- stories/*.parquet
- grammar/*.parquet
- finance/*.parquet
- dictionary/*.parquet
- toxicity/*.parquet
- uncensored/*.parquet
---
# ViuAI/viu-mini-raw-pretrain
Raw pretraining corpus for **ViuMini-MoE-242M** (Hinglish-first; language-mix rule OPTIONAL, default OFF — see `enforce_mix` in training config).
Unified schema: `text, lang, source, domain, safety_tag`.
- `text`: training text (plain, or `<|user|> ... <|assistant|> ...` templated for QA/instruct/distillation sources)
- `lang`: `hinglish` | `hindi` | `english` | `bilingual`
- `source`: origin dataset name (e.g. `indiccorp_v2`, `fineweb-edu`, `bespoke-stratos-r1`, `smoltalk`, `numina_math_cot`)
- `domain`: knowledge pillar (e.g. `general_hindi`, `distilled_reasoning`, `distilled_math_cot`, `sports_cricket`, `automobile_rto`)
- `safety_tag`: `safe` | `toxic` | `uncensored`
## Distillation & Reasoning Corpus (`distilled/`)
Includes 33 high-quality reasoning shards (3.73 GB, ~1.7B tokens) distilled from frontier models (DeepSeek-R1, SmolTalk, NuminaMath, Magpie Llama 3.1, FineTome 100k, Bespoke Stratos R1) with native `<soch> ... </soch>` chain-of-thought isolation.
## Safety notice (intentional inclusion)
`toxicity/` and `uncensored/` folders are **intentionally included** for robustness research
(hate-speech detection, refusal training, red-teaming). They are TAGGED, not hidden:
```python
from datasets import load_dataset
ds = load_dataset("ViuAI/viu-mini-raw-pretrain", split="train", streaming=True)
def keep(row, allow_toxic=False, allow_uncensored=False):
tag = (row.get("safety_tag") or "safe").lower()
if tag == "safe": return True
if tag == "toxic": return allow_toxic
if tag == "uncensored": return allow_uncensored
return True
# default pretraining: safe only
ds_safe = ds.filter(lambda r: keep(r))
# robustness research: safe+toxic
ds_research = ds.filter(lambda r: keep(r, allow_toxic=True))
# red-team only, controlled: everything
ds_full = ds.filter(lambda r: keep(r, allow_toxic=True, allow_uncensored=True))
```
Do NOT train user-facing models on `full` without alignment (SFT/DPO + refusal).
## Repair & Clean Status (2026-09-23)
1. Hindi missing shards: 100% Backfilled (447 shards total).
2. All oversized files resharded to 150MB standard shards (`english_fixed/` & `hindi_fixed/`).
3. 50 legacy oversized files deleted — ZERO storage duplication.
4. Schema unified across all domains (`grammar`, `math`, `science`, `toxicity`, `uncensored`, `domains`, `distilled`).
5. Frontier distillation corpus: 33 shards (3.73 GB) integrated into `distilled/`.
6. Net clean dataset size: ~229.5 GB (>105.1B tokens across 1,926 clean files).
## License / citation
Apache-2.0. Upstream sources retain their own licenses (Wikipedia CC-BY-SA, Samanantar CC-BY, etc.).
If you use this dataset, cite upstream sources plus `ViuAI/viu-mini-raw-pretrain`.
|