| --- |
| language: |
| - en |
| library_name: tokenizers |
| license: mit |
| tags: |
| - tokenizer |
| - bpe |
| --- |
| |
| # scribe-tok-5k |
|
|
| [](https://github.com/coderianx/scribe-tok-5k) |
| [](https://huggingface.co/coderian/scribe-tok-5k) |
| [](https://opensource.org/licenses/MIT) |
|
|
| A small byte-pair encoding (BPE) tokenizer with a 5,000-token vocabulary, trained on clean English book text. It is intended for experimentation, learning, and small language models where a compact vocabulary is preferred over a large production tokenizer. |
|
|
| ## Details |
|
|
| | | | |
| |---|---| |
| | Model type | BPE | |
| | Vocab size | 5,000 | |
| | Pre-tokenizer | Whitespace | |
| | Min frequency | 2 | |
| | Corpus | [kd13/bookcorpus-clean](https://huggingface.co/datasets/kd13/bookcorpus-clean) (100,000 lines) | |
| | Trained with | `tokenizers` 0.23.2 / `transformers` 5.17.0 | |
|
|
| ### Special tokens |
|
|
| | Token | ID | |
| |---|---| |
| | `[PAD]` | 0 | |
| | `[UNK]` | 1 | |
| | `[CLS]` | 2 | |
| | `[SEP]` | 3 | |
| | `[MASK]` | 4 | |
|
|
| The whitespace pre-tokenizer means tokens never include leading spaces, and no lowercasing is applied (the corpus is already lowercase). |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-5k") |
| |
| text = "the mysterious creature suddenly disappeared into the shadows" |
| encoded = tokenizer(text) |
| |
| print(encoded["input_ids"]) |
| print(tokenizer.convert_ids_to_tokens(encoded["input_ids"])) |
| ``` |
|
|
| Or directly with `tokenizers`: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| |
| tokenizer = Tokenizer.from_file("tokenizer.json") |
| output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows") |
| |
| print(output.tokens) |
| print(output.ids) |
| ``` |
|
|
| ## Training |
|
|
| The tokenizer was trained from scratch: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| from tokenizers.models import BPE |
| from tokenizers.trainers import BpeTrainer |
| from tokenizers.pre_tokenizers import Whitespace |
| |
| tokenizer = Tokenizer(BPE(unk_token="[UNK]")) |
| tokenizer.pre_tokenizer = Whitespace() |
| |
| trainer = BpeTrainer( |
| vocab_size=5000, |
| min_frequency=2, |
| special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"], |
| ) |
| |
| tokenizer.train(["corpus.txt"], trainer=trainer) |
| tokenizer.save("tokenizer.json") |
| ``` |
|
|
| ## Limitations |
|
|
| - Trained only on English book text; performance on other languages, code, or social/web text will be poor. |
| - No normalization or lowercasing is applied, so casing variations are treated as distinct tokens. |
| - Vocabulary is intentionally small (5k), so rare words are split into many subword pieces. |
|
|
| ## License |
|
|
| Released under the [MIT License](LICENSE). |
|
|
