| # Tokenizer — NOTES | |
| Module: 48k byte-level BPE, Hinglish-focus. Mix rule OPTIONAL (`--mix` / `--no-mix`, default: 40/30/30 local, 35/45/20 Hub). | |
| ## Structure | |
| ``` | |
| tokenizer/ | |
| configs/tokenizer_config.yaml | |
| scripts/tokenizer_train.py | |
| outputs/tokenizer.json (final), tokenizer_test.json (sample test) | |
| eval/ (coverage reports future) | |
| ``` | |
| ## Log | |
| - 2026-09-15: train script banaya, sample data (35 lines) pe test pass — vocab 714 bana (data chhota tha), <unk>=0, fertility 1.77 | |
| - 2026-09-15: modular move — `src/` -> `tokenizer/scripts/`, `tokenizers/` -> `tokenizer/outputs/` | |
| - TODO: real 500k lines pe final training, Hindi fertility 1.5-1.8 target | |
| ## Run | |
| ```powershell | |
| # tokenizer/scripts se: | |
| python tokenizer_train.py --data_dir ../../data/raw --out ../outputs/tokenizer.json --vocab_size 48000 --max_lines 500000 | |
| # project root se: | |
| python tokenizer/scripts/tokenizer_train.py --data_dir ./data/raw --out ./tokenizer/outputs/tokenizer.json --vocab_size 48000 --max_lines 500000 | |
| ``` | |