tokenizer/NOTES.md

Tokenizer — NOTES

Module: 48k byte-level BPE, Hinglish-focus. Mix rule OPTIONAL (--mix / --no-mix, default: 40/30/30 local, 35/45/20 Hub).

Structure

tokenizer/
  configs/tokenizer_config.yaml
  scripts/tokenizer_train.py
  outputs/tokenizer.json (final), tokenizer_test.json (sample test)
  eval/ (coverage reports future)

Log

• 2026-09-15: train script banaya, sample data (35 lines) pe test pass — vocab 714 bana (data chhota tha), =0, fertility 1.77
• 2026-09-15: modular move — src/ -> tokenizer/scripts/, tokenizers/ -> tokenizer/outputs/
• TODO: real 500k lines pe final training, Hindi fertility 1.5-1.8 target

Run

# tokenizer/scripts se:
python tokenizer_train.py --data_dir ../../data/raw --out ../outputs/tokenizer.json --vocab_size 48000 --max_lines 500000
# project root se:
python tokenizer/scripts/tokenizer_train.py --data_dir ./data/raw --out ./tokenizer/outputs/tokenizer.json --vocab_size 48000 --max_lines 500000