MadTokenizer-50K

Byte-level BPE tokenizer for English, trained from scratch on wikitext-103. Vocabulary size: 50,000 tokens.

About

• Type: byte-level BPE (same recipe as GPT-2)
• Vocab size: 50,000
• Corpus: wikitext-103 (raw, ~538M characters)
• Special tokens: <s>, <pad>, </s>, <unk>, <mask>
• Use case: tokenizer-only model (no language-model weights yet)

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("MadSensiro/MadTokenizer-50K")

ids = tok("Hello, world! How are you today?").input_ids
# [24420, 16, 1302, 5, 3469, 475, 1616, 4329, 35]

print(tok.decode(ids))   # ' Hello, world! How are you today?'

The tokenizer is byte-level, so it handles arbitrary text (unicode, spaces, emojis) without producing <unk>, because every byte sequence is decodable.

Files

config.json              # model config + metadata
tokenizer.json           # serialized fast tokenizer (loaded by HF)
tokenizer_config.json    # tokenizer options
special_tokens_map.json  # special token mapping
vocab.json               # token -> id
merges.txt               # BPE merge rules

Training

Trained with the Hugging Face tokenizers library (BpeTrainer, ByteLevel pre-tokenizer, min_frequency=2).

Note

BPE uses a leading whitespace marker (Ġ) for beginning-of-word tokens, so decode of a standalone string may prepend a space — this is standard GPT-2 behavior and expected.