| --- |
| language: |
| - en |
| library_name: tokenizers |
| license: mit |
| tags: |
| - tokenizer |
| - bpe |
| --- |
| |
| # scribe-tok-50k |
|
|
| [](https://github.com/coderianx/scribe-tok-50k) |
| [](https://huggingface.co/coderian/scribe-tok-50k) |
| [](https://opensource.org/licenses/MIT) |
|
|
| A byte-pair encoding (BPE) tokenizer with a 50,004-token vocabulary, trained from scratch on 2,000,000 lines of clean English book text. It is intended for experimentation, learning, and small-to-medium language models that need a compact, from-scratch vocabulary instead of a large production tokenizer. |
|
|
| ## Details |
|
|
| | | | |
| |---|---| |
| | Model type | BPE | |
| | Vocab size | 50,004 | |
| | Merges | 49,928 | |
| | Pre-tokenizer | Whitespace | |
| | Normalizer | None | |
| | Decoder | None | |
| | Min frequency | 2 | |
| | Corpus | [kd13/bookcorpus-clean](https://huggingface.co/datasets/kd13/bookcorpus-clean) (2,000,000 lines, ~151 MB) | |
| | Trained with | `tokenizers` 0.23.2 / `transformers` 5.17.0 | |
|
|
| ### Special tokens |
|
|
| | Token | ID | |
| |---|---| |
| | `[PAD]` | 0 | |
| | `[UNK]` | 1 | |
| | `[BOS]` | 2 | |
| | `[EOS]` | 3 | |
|
|
| ## Usage |
|
|
| With `transformers`: |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-50k") |
| |
| text = "the mysterious creature suddenly disappeared into the shadows" |
| encoded = tokenizer(text) |
| |
| print(encoded["input_ids"]) |
| print(tokenizer.convert_ids_to_tokens(encoded["input_ids"])) |
| ``` |
|
|
| Or directly with `tokenizers`: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| |
| tokenizer = Tokenizer.from_file("tokenizer.json") |
| output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows") |
| |
| print(output.tokens) |
| print(output.ids) |
| ``` |
|
|
| ## Training |
|
|
| The tokenizer was trained from scratch on the English `kd13/bookcorpus-clean` dataset: |
|
|
| ```python |
| from tokenizers import Tokenizer |
| from tokenizers.models import BPE |
| from tokenizers.trainers import BpeTrainer |
| from tokenizers.pre_tokenizers import Whitespace |
| |
| tokenizer = Tokenizer(BPE(unk_token="[UNK]")) |
| tokenizer.pre_tokenizer = Whitespace() |
| |
| trainer = BpeTrainer( |
| vocab_size=50_004, |
| min_frequency=2, |
| special_tokens=["[PAD]", "[UNK]", "[BOS]", "[EOS]"], |
| ) |
| |
| tokenizer.train(["corpus.txt"], trainer=trainer) |
| tokenizer.save("tokenizer.json") |
| ``` |
|
|
| The full pipeline is available on [GitHub](https://github.com/coderianx/scribe-tok-50k): |
|
|
| 1. `config.py` - streams 2,000,000 lines from the dataset and writes `corpus.txt` |
| 2. `train.py` - trains the BPE tokenizer, saves `tokenizer.json`, and pushes it to the Hub |
| 3. `test.py` - encodes a sample sentence and prints tokens/IDs |
|
|
| ## Limitations |
|
|
| - Trained only on English book text; performance on other languages, code, or social/web text will be poor. |
| - No normalization or lowercasing is applied, so casing variations are treated as distinct tokens. |
| - Pre-tokenization is whitespace-only, so tokens never include leading spaces and punctuation is treated as standalone tokens. |
| - No decoder is configured; `decode` reflects the whitespace-separated format of the training corpus. |
|
|
| ## License |
|
|
| Released under the [MIT License](https://opensource.org/licenses/MIT). |
|
|
