File size: 2,832 Bytes
31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 31afde9 00b18a3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
language:
- en
library_name: tokenizers
license: mit
tags:
- tokenizer
- bpe
---
# scribe-tok-5k
[](https://github.com/coderianx/scribe-tok-5k)
[](https://huggingface.co/coderian/scribe-tok-5k)
[](https://opensource.org/licenses/MIT)
A small byte-pair encoding (BPE) tokenizer with a 5,000-token vocabulary, trained on clean English book text. It is intended for experimentation, learning, and small language models where a compact vocabulary is preferred over a large production tokenizer.
## Details
| | |
|---|---|
| Model type | BPE |
| Vocab size | 5,000 |
| Pre-tokenizer | Whitespace |
| Min frequency | 2 |
| Corpus | [kd13/bookcorpus-clean](https://huggingface.co/datasets/kd13/bookcorpus-clean) (100,000 lines) |
| Trained with | `tokenizers` 0.23.2 / `transformers` 5.17.0 |
### Special tokens
| Token | ID |
|---|---|
| `[PAD]` | 0 |
| `[UNK]` | 1 |
| `[CLS]` | 2 |
| `[SEP]` | 3 |
| `[MASK]` | 4 |
The whitespace pre-tokenizer means tokens never include leading spaces, and no lowercasing is applied (the corpus is already lowercase).
## Usage
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-5k")
text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)
print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))
```
Or directly with `tokenizers`:
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")
print(output.tokens)
print(output.ids)
```
## Training
The tokenizer was trained from scratch:
```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(
vocab_size=5000,
min_frequency=2,
special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
)
tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
```
## Limitations
- Trained only on English book text; performance on other languages, code, or social/web text will be poor.
- No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
- Vocabulary is intentionally small (5k), so rare words are split into many subword pieces.
## License
Released under the [MIT License](LICENSE).
|