Model Card for Model ID
This is a Byte-Pair Encoding (BPE) tokenizer trained from scratch on Turkish Wikipedia text using the 🤗 Hugging Face tokenizers library. It is intended for use as the tokenizer component of Turkish-language NLP models.
Model Details
Model Description
This is a Byte-Pair Encoding (BPE) tokenizer trained from scratch on Turkish Wikipedia text using the 🤗 Hugging Face tokenizers library. It is intended for use as the tokenizer component of Turkish-language NLP models.
- Developed by: Talayhan
- Model type: BPE Tokenizer (vocabulary size: 8,192)
- Language(s) (NLP): Turkish (tr)
Model Sources
- Repository: Trained using the train_hf_tokenizer.py script from malibayram/single_letter_transformers
Uses
Direct Use
This tokenizer can be used to convert Turkish text into subword tokens (BPE) for downstream use in training or fine-tuning Turkish NLP models (e.g., language models, classifiers).
Out-of-Scope Use
Not recommended for languages other than Turkish, or for domains substantially different from Wikipedia text (e.g., code, social media slang, dialectal variants), where the vocabulary coverage may be poor.
Bias, Risks, and Limitations
As the tokenizer was trained solely on Turkish Wikipedia, it may underrepresent informal language, dialectal variation, code-switching, and domain-specific vocabulary (e.g., medical, legal, colloquial). A vocabulary size of 8,192 is relatively small, which may lead to more aggressive subword splitting for rare or out-of-domain words.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the tokenizer. Consider evaluating tokenization quality (fertility, coverage) on your target domain before use, especially if it differs from encyclopedic text.
Training Details
Training Data
Turkish Wikipedia dump, sourced from kaan39/turkish-wikipedia-dataset on the Hugging Face Hub. The dataset contains 26,871 rows with a total file size of 37.7 MB.
Training Procedure
Preprocessing
Only the messages (text content) field was extracted from the dataset and written to a plain .txt file; the tokenizer was trained on this extracted text.
Training Hyperparameters
- Algorithm: Byte-Pair Encoding (BPE)
- Vocabulary size: 8,192 (2^13)
Technical Specifications
Model Architecture and Objective
BPE (Byte-Pair Encoding) subword tokenizer, vocabulary size 8,192, trained with the Hugging Face tokenizers library.