You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

ViLLM Tokenizer v3

Unified BPE tokenizer for Vietnamese + English, built on tiktoken cl100k_base with 37,986 Vietnamese-specific merge rules.

Architecture

Component	IDs	Description
BPE pieces	0 – 138,241	tiktoken `cl100k_base` (100k) + Vietnamese merges (38k). Rank = token ID
Compounds	138,242 – 163,241	25,000 `_`-joined Vietnamese compounds (e.g., `học_sinh`)
Special/code/byte	163,242 – 163,563	Special tokens, byte fallbacks, code punctuation
Total	163,564	Effective embedding slots needed

Reduced vocab

vocab.json contains only 25,322 entries (non-BPE tokens). BPE piece IDs come directly from tiktoken ranks — the redundant 90k en_bpe entries and 55k en_sp (SentencePiece) entries are eliminated.

Quick Start

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("vlinhd11/villm-tokenizer", trust_remote_code=True)
print(tok.vocab_size)  # 25322
ids = tok.encode("Xin chào, tôi là một trợ lý AI.")
print(tok.decode(ids))

Performance

80/80 roundtrip benchmark: 100% exact match, 0% char error
English tokenization: identical to GPT-4 (cl100k_base)
Vietnamese: each syllable = 1 BPE piece (100% syllable coverage)

Requirements

transformers >= 4.30
villm-tok-rs >= 0.3.0 (Rust fast path, 8x faster). Install: pip install villm-tok-rs

vlinhd11
/

villm-tokenizer