File size: 3,357 Bytes
b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc b9a2321 bc2c2cc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
language:
- en
library_name: tokenizers
license: mit
tags:
- tokenizer
- bpe
---
# scribe-tok-50k
[](https://github.com/coderianx/scribe-tok-50k)
[](https://huggingface.co/coderian/scribe-tok-50k)
[](https://opensource.org/licenses/MIT)
A byte-pair encoding (BPE) tokenizer with a 50,004-token vocabulary, trained from scratch on 2,000,000 lines of clean English book text. It is intended for experimentation, learning, and small-to-medium language models that need a compact, from-scratch vocabulary instead of a large production tokenizer.
## Details
| | |
|---|---|
| Model type | BPE |
| Vocab size | 50,004 |
| Merges | 49,928 |
| Pre-tokenizer | Whitespace |
| Normalizer | None |
| Decoder | None |
| Min frequency | 2 |
| Corpus | [kd13/bookcorpus-clean](https://huggingface.co/datasets/kd13/bookcorpus-clean) (2,000,000 lines, ~151 MB) |
| Trained with | `tokenizers` 0.23.2 / `transformers` 5.17.0 |
### Special tokens
| Token | ID |
|---|---|
| `[PAD]` | 0 |
| `[UNK]` | 1 |
| `[BOS]` | 2 |
| `[EOS]` | 3 |
## Usage
With `transformers`:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-50k")
text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)
print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))
```
Or directly with `tokenizers`:
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")
print(output.tokens)
print(output.ids)
```
## Training
The tokenizer was trained from scratch on the English `kd13/bookcorpus-clean` dataset:
```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(
vocab_size=50_004,
min_frequency=2,
special_tokens=["[PAD]", "[UNK]", "[BOS]", "[EOS]"],
)
tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
```
The full pipeline is available on [GitHub](https://github.com/coderianx/scribe-tok-50k):
1. `config.py` - streams 2,000,000 lines from the dataset and writes `corpus.txt`
2. `train.py` - trains the BPE tokenizer, saves `tokenizer.json`, and pushes it to the Hub
3. `test.py` - encodes a sample sentence and prints tokens/IDs
## Limitations
- Trained only on English book text; performance on other languages, code, or social/web text will be poor.
- No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
- Pre-tokenization is whitespace-only, so tokens never include leading spaces and punctuation is treated as standalone tokens.
- No decoder is configured; `decode` reflects the whitespace-separated format of the training corpus.
## License
Released under the [MIT License](https://opensource.org/licenses/MIT).
|