File size: 1,978 Bytes
65c182b 4a6ac08 65c182b 4a6ac08 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
language:
- th
- en
license: apache-2.0
---
# ⚡ KordAI TokenizerFast
**KordAI TokenizerFast** is a fast Hugging Face tokenizer designed for efficient text preprocessing for **Thai and English** language models.
It is fully compatible with the 🤗 Transformers ecosystem and can be used for both training and inference with compatible models.
---
# ✨ Features
- 🚀 Fast tokenizer implementation
- 🇹🇭 Thai language support
- 🇺🇸 English language support
- 🤗 Compatible with Hugging Face Transformers
- ⚡ Efficient tokenization and decoding
- 💻 Suitable for both training and inference
---
# 📋 Model Details
| Item | Value |
|------|-------|
| Repository | `KordAI/TokenizerFast` |
| Type | Fast Tokenizer |
| Languages | Thai, English |
| Library | Transformers |
| License | Apache-2.0 |
---
# 🎯 Intended Use
This tokenizer is designed for:
- Thai text tokenization
- English text tokenization
- Multilingual NLP pipelines
- Dataset preprocessing
- Language model training
- Language model inference
---
# 🐍 Usage
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"KordAI/TokenizerFast"
)
text = "สวัสดี Hello!"
tokens = tokenizer(text)
print(tokens)
decoded = tokenizer.decode(tokens["input_ids"])
print(decoded)
```
---
# ⚠️ Notes
- This repository contains only the tokenizer.
- It should be used together with a compatible language model.
- Performance depends on the model paired with this tokenizer.
---
# 🙏 Acknowledgments
Special thanks to:
- **Hugging Face** for the Transformers ecosystem.
- **KordAI** for developing and distributing the tokenizer.
- The open-source AI community for advancing multilingual NLP.
---
# 📖 Citation
```bibtex
@misc{kordaitokenizerfast2026,
title={KordAI TokenizerFast},
author={KordAI},
year={2026},
publisher={Hugging Face},
howpublished={https://huggingface.co/KordAI/TokenizerFast}
}
``` |