# Tokenizer — NOTES
Module: 48k byte-level BPE, Hinglish-focus. Mix rule OPTIONAL (`--mix` / `--no-mix`, default: 40/30/30 local, 35/45/20 Hub).

## Structure
```
tokenizer/
  configs/tokenizer_config.yaml
  scripts/tokenizer_train.py
  outputs/tokenizer.json (final), tokenizer_test.json (sample test)
  eval/ (coverage reports future)
```

## Log
- 2026-09-15: train script banaya, sample data (35 lines) pe test pass — vocab 714 bana (data chhota tha), <unk>=0, fertility 1.77
- 2026-09-15: modular move — `src/` -> `tokenizer/scripts/`, `tokenizers/` -> `tokenizer/outputs/`
- TODO: real 500k lines pe final training, Hindi fertility 1.5-1.8 target

## Run
```powershell
# tokenizer/scripts se:
python tokenizer_train.py --data_dir ../../data/raw --out ../outputs/tokenizer.json --vocab_size 48000 --max_lines 500000
# project root se:
python tokenizer/scripts/tokenizer_train.py --data_dir ./data/raw --out ./tokenizer/outputs/tokenizer.json --vocab_size 48000 --max_lines 500000
```