---
language:
- en
library_name: tokenizers
license: mit
tags:
- tokenizer
- bpe
---

# scribe-tok-5k

[![GitHub](https://img.shields.io/badge/GitHub-coderianx%2Fscribe--tok--5k-181717?logo=github&logoColor=white)](https://github.com/coderianx/scribe-tok-5k)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-coderian%2Fscribe--tok--5k-FFD21E)](https://huggingface.co/coderian/scribe-tok-5k)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A small byte-pair encoding (BPE) tokenizer with a 5,000-token vocabulary, trained on clean English book text. It is intended for experimentation, learning, and small language models where a compact vocabulary is preferred over a large production tokenizer.

## Details

| | |
|---|---|
| Model type | BPE |
| Vocab size | 5,000 |
| Pre-tokenizer | Whitespace |
| Min frequency | 2 |
| Corpus | [kd13/bookcorpus-clean](https://huggingface.co/datasets/kd13/bookcorpus-clean) (100,000 lines) |
| Trained with | `tokenizers` 0.23.2 / `transformers` 5.17.0 |

### Special tokens

| Token | ID |
|---|---|
| `[PAD]` | 0 |
| `[UNK]` | 1 |
| `[CLS]` | 2 |
| `[SEP]` | 3 |
| `[MASK]` | 4 |

The whitespace pre-tokenizer means tokens never include leading spaces, and no lowercasing is applied (the corpus is already lowercase).

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("coderian/scribe-tok-5k")

text = "the mysterious creature suddenly disappeared into the shadows"
encoded = tokenizer(text)

print(encoded["input_ids"])
print(tokenizer.convert_ids_to_tokens(encoded["input_ids"]))
```

Or directly with `tokenizers`:

```python
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
output = tokenizer.encode("the mysterious creature suddenly disappeared into the shadows")

print(output.tokens)
print(output.ids)
```

## Training

The tokenizer was trained from scratch:

```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

trainer = BpeTrainer(
    vocab_size=5000,
    min_frequency=2,
    special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
)

tokenizer.train(["corpus.txt"], trainer=trainer)
tokenizer.save("tokenizer.json")
```

## Limitations

- Trained only on English book text; performance on other languages, code, or social/web text will be poor.
- No normalization or lowercasing is applied, so casing variations are treated as distinct tokens.
- Vocabulary is intentionally small (5k), so rare words are split into many subword pieces.

## License

Released under the [MIT License](LICENSE).