salyamq/smq-tokenizer-kz

github repo: https://github.com/salyamq/smq_kk_tokenizer

SMQ Kazakh Tokenizer

SentencePiece BPE tokenizer trained for the Kazakh language.

Details

• Model type: SentencePiece (BPE)
• Vocab size: 32,000
• Language: Kazakh (kk)

Files

• smq_kk_tokenizer.model - SentencePiece model file
• smq_kk_tokenizer.vocab - vocabulary file

Usage

from huggingface_hub import hf_hub_download
import sentencepiece as spm

model_path = hf_hub_download(
    repo_id="salyamq/smq-tokenizer-kz",
    filename="smq_kk_tokenizer.model"
)

sp = spm.SentencePieceProcessor()
sp.load(model_path)

tokens = sp.encode("Бұл сөйлем қазақ тілінде жазылған", out_type=str)
print(tokens)

Evaluation

Measured on a held-out Kazakh text chunk (~443K chars):

Metric	Value
Chars/token	5.37
Bytes/token	9.93
Tokens/word	1.49
UNK rate	0.00%

Notes

Trained with byte_fallback True using the spm.SentencePieceTrainer.train pipeline. Not yet compatible with AutoTokenizer.from_pretrained — load via sentencepiece directly (see usage above).