---
language:
- hi
- en
- multilingual
language_bcp47:
- hi-Latn
license: apache-2.0
task_categories:
- text-generation
tags:
- hinglish
- hindi
- pretraining
- code
- science
- legal
pretty_name: Viu Mini Raw Pretrain
size_categories:
- 100M<n<1B
configs:
- config_name: default
  data_files:
  - split: train
    path:
    - hindi/train-*.parquet
    - hindi_fixed/train-*.parquet
    - english_fixed/train-*.parquet
    - hinglish/train-*.parquet
    - domains/train-*.parquet
    - translation/train-*.parquet
    - translation/samanantar-*.parquet
    - wikipedia/train-*.parquet
    - science/*.parquet
    - math/*.parquet
    - code/*.parquet
    - legal/train-*.parquet
    - health/*.parquet
    - stories/*.parquet
    - grammar/*.parquet
    - finance/*.parquet
    - dictionary/*.parquet
    - toxicity/*.parquet
    - uncensored/*.parquet
---

# ViuAI/viu-mini-raw-pretrain

Raw pretraining corpus for **ViuMini-MoE-242M** (Hinglish-first; language-mix rule OPTIONAL, default OFF — training config me `enforce_mix` dekho).
Unified schema: `text, lang, source, domain, safety_tag` (older files still migrating — see Known issues).

- `text`: training text (plain, or `<|user|> ... <|assistant|> ...` templated for QA/instruct sources)
- `lang`: `hinglish` | `hindi` | `english` | `bilingual`
- `source`: origin dataset/generator name (e.g. `indiccorp_v2`, `fineweb-edu`, `vyakaran_master`, `sciq`)
- `domain`: knowledge pillar (e.g. `general_hindi`, `electronics_smt_pcb`, `science_qa`, `hindi_vyakaran_sandhi`)
- `safety_tag`: `safe` | `toxic` | `uncensored`

## Safety notice (intentional inclusion)

`toxicity/` and `uncensored/` folders are **intentionally included** for robustness research
(hate-speech detection, refusal training, red-teaming). They are TAGGED, not hidden:

```python
from datasets import load_dataset
ds = load_dataset("ViuAI/viu-mini-raw-pretrain", split="train", streaming=True)

def keep(row, allow_toxic=False, allow_uncensored=False):
    tag = (row.get("safety_tag") or "safe").lower()
    if tag == "safe": return True
    if tag == "toxic": return allow_toxic
    if tag == "uncensored": return allow_uncensored
    return True

# default pretraining: safe only
ds_safe = ds.filter(lambda r: keep(r))
# robustness research: safe+toxic
ds_research = ds.filter(lambda r: keep(r, allow_toxic=True))
# red-team only, controlled: everything
ds_full = ds.filter(lambda r: keep(r, allow_toxic=True, allow_uncensored=True))
```

Do NOT train user-facing models on `full` without alignment (SFT/DPO + refusal).

## Known issues (2026-09-22 audit, fixing in progress)

1. Hindi shards `train-00115..00178`, `train-00307..00370` missing (upload gaps) — backfilling.
2. `english/` 37x~1.6GB + `hindi/` 13x~1.15GB oversized shards — resharding to ~150MB/100k rows.
3. Legacy schema drift (`question/answer`, `instruction/response`, `topic/difficulty`, `title/text`) —
   migrating to unified schema via `data/scripts/standardize_to_unified.py`.
4. Viewer HTTP 500 on some offsets until (2)+(3) complete.
5. Duplicate curated pillar rows (`domains/` vs generated) — dedup via stable blake2b hash.

See `docs/HF_REPAIR_RUNBOOK.md` in the model repo for repair commands.

## License / citation

Apache-2.0. Upstream sources retain their own licenses (Wikipedia CC-BY-SA, Samanantar CC-BY, etc.).
If you use this dataset, cite upstream sources plus `ViuAI/viu-mini-raw-pretrain`.