---
language:
- en
library_name: tokenizers
license: mit
tags:
- tokenizer
- bpe
---

# scribe-tok-50k

[![GitHub](https://img.shields.io/badge/GitHub-coderianx%2Fscribe--tok--50k-181717?logo=github&logoColor=white)](https://github.com/coderianx/scribe-tok-50k)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-coderian%2Fscribe--tok--50k-FFD21E)](https://huggingface.co/coderian/scribe-tok-50k)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A byte-pair encoding (BPE) tokenizer with a 50,004-token vocabulary, trained from scratch on 2,000,000 lines of clean English book text. It is intended for experimentation, learning, and small-to-medium language models that need a compact, from-scratch vocabulary instead of a large production tokenizer.

## Details

| | |
|---|---|
| Model type | BPE |
| Vocab size | 50,004 |
| Merges | 49,928 |
| Pre-tokenizer | Whitespace |
| Normalizer | None |
| Decoder | None |
| Min frequency | 2 |
| Corpus | [kd13/bookcorpus-clean](https://huggingface.co/datasets/kd13/bookcorpus-clean) (2,000,000 lines, ~151 MB) |
| Trained with | `tokenizers` 0.23.2 / `transformers` 5.17.0 |

### Special tokens

| Token | ID |
|---|---|
| `[PAD]` | 0 |
| `[UNK]` | 1 |
| `[BOS]` | 2 |
| `[EOS]` | 3 |

## Usage

With `transformers`:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-50k")

text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)

print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))
```

Or directly with `tokenizers`:

```python
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")

print(output.tokens)
print(output.ids)
```

## Training

The tokenizer was trained from scratch on the English `kd13/bookcorpus-clean` dataset:

```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = BpeTrainer(
    vocab_size=50_004,
    min_frequency=2,
    special_tokens=["[PAD]", "[UNK]", "[BOS]", "[EOS]"],
)

tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
```

The full pipeline is available on [GitHub](https://github.com/coderianx/scribe-tok-50k):

1. `config.py` - streams 2,000,000 lines from the dataset and writes `corpus.txt`
2. `train.py` - trains the BPE tokenizer, saves `tokenizer.json`, and pushes it to the Hub
3. `test.py` - encodes a sample sentence and prints tokens/IDs

## Limitations

- Trained only on English book text; performance on other languages, code, or social/web text will be poor.
- No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
- Pre-tokenization is whitespace-only, so tokens never include leading spaces and punctuation is treated as standalone tokens.
- No decoder is configured; `decode` reflects the whitespace-separated format of the training corpus.

## License

Released under the [MIT License](https://opensource.org/licenses/MIT).