MadTokenizer-100K

Byte-level BPE tokenizer for English, trained from scratch on wikitext-103 and fine-tuned on multi_news. Vocabulary size: 100,000 tokens.

About

• Type: byte-level BPE (same recipe as GPT-2)
• Vocab size: 100,000
• Corpus: wikitext-103 (raw, ~538M characters) + multi_news
• Special tokens: <s>, <pad>, </s>, <unk>, <mask>
• Use case: tokenizer-only model (no language-model weights yet)

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("MadSensiro/MadTokenizer-100K")

ids = tok("Hello, world! How are you today?").input_ids
# [652, 39, 21, 99, 12, 9, 21, 92, 70]  (example)

print(tok.decode(ids))   # ' Hello, world! How are you today?'

The tokenizer is byte-level, so it handles arbitrary text (unicode, spaces, emojis) without producing <unk>, because every byte sequence is decodable.

Files

config.json              # model config + metadata
tokenizer.json           # serialized fast tokenizer (loaded by HF)
tokenizer_config.json    # tokenizer options
special_tokens_map.json  # special token mapping
vocab.json               # token -> id
merges.txt               # BPE merge rules

Training

Trained with the Hugging Face tokenizers library (BpeTrainer, ByteLevel pre-tokenizer, min_frequency=2).

Note

BPE uses a leading whitespace marker (Ġ) for beginning-of-word tokens, so decode of a standalone string may prepend a space — this is standard GPT-2 behavior and expected.