metadata
language:
- en
library_name: tokenizers
license: mit
tags:
- tokenizer
- bpe
scribe-tok-5k
A small byte-pair encoding (BPE) tokenizer with a 5,000-token vocabulary, trained on clean English book text. It is intended for experimentation, learning, and small language models where a compact vocabulary is preferred over a large production tokenizer.
Details
| Model type | BPE |
| Vocab size | 5,000 |
| Pre-tokenizer | Whitespace |
| Min frequency | 2 |
| Corpus | kd13/bookcorpus-clean (100,000 lines) |
| Trained with | tokenizers 0.23.2 / transformers 5.17.0 |
Special tokens
| Token | ID |
|---|---|
[PAD] |
0 |
[UNK] |
1 |
[CLS] |
2 |
[SEP] |
3 |
[MASK] |
4 |
The whitespace pre-tokenizer means tokens never include leading spaces, and no lowercasing is applied (the corpus is already lowercase).
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-5k")
text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)
print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))
Or directly with tokenizers:
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")
print(output.tokens)
print(output.ids)
Training
The tokenizer was trained from scratch:
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(
vocab_size=5000,
min_frequency=2,
special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
)
tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
Limitations
- Trained only on English book text; performance on other languages, code, or social/web text will be poor.
- No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
- Vocabulary is intentionally small (5k), so rare words are split into many subword pieces.
License
Released under the MIT License.