Jommarn/thai-0.1b-tokenizer

🇹🇭 Thai-English 16K ByteLevel BPE Tokenizer

A compact, highly efficient 16,384 vocabulary Byte-Level Byte-Pair Encoding (BPE) tokenizer tailored specifically for Thai and English Small Language Models (SLMs).

By limiting the vocabulary to 16,384 tokens, an embedding layer requires only ~12.5M parameters (at d_model=768), leaving the remaining ~95M+ parameters for 14 deep Transformer layers in a 108M parameter budget!

---

🚀 Key Features

1. High Compression for Thai: Efficient subword and byte-level representations preventing token bloat on Thai text.
2. Special Tokens Built-in:
   • ChatML Tokens: <|im_start|>, <|im_end|>
   • DeepSeek-R1 CoT Tokens: <think>, </think>
   • Standard Delimiters: <s>, </s>, <unk>, <pad>, <eos>
3. Training Data:
   • Thai Wikipedia (cleaned dump)
   • Thai conversational dialogues (ThaiInstruct)
   • NECTEC Thai Food & Cultural lexicon
   • Simple English texts

---

💻 How to Use

from transformers import PreTrainedTokenizerFast

# Load tokenizer from Hugging Face Hub
tokenizer = PreTrainedTokenizerFast.from_pretrained("Jommarn/thai-0.1b-tokenizer")

# Example Encoding
text = "สวัสดีครับ! โมเดลภาษาไทยขนาด 0.1B สร้างขึ้นโดยจอมมาร (Jommarn)"
tokens = tokenizer.encode(text)
print("Tokens:", tokens)
print("Decoded:", tokenizer.decode(tokens))

# ChatML Formatting Example
chat = "<|im_start|>user\nสวัสดีครับคุณคือใคร?<|im_end|>\n<|im_start|>assistant\n<think>\n"
print("Chat token IDs:", tokenizer.encode(chat))

---

📜 License

Apache-2.0. Free for open research and commercial use.