cro-bpe-64k

Croatian byte-level BPE tokenizer with a vocabulary of 64,000 tokens, trained only on Croatian.

General-purpose tokenizers are built mostly for English and split Croatian words into many pieces, so Croatian text costs more tokens, more context and more compute than it should. cro-bpe is trained on 12 GB sampled from ten openly licensed Croatian corpora (web crawls, Wikipedia, parliamentary proceedings, EU legislation) after exact and near-duplicate removal.

This repository has the tokenizer in two formats with identical ids:

• tokenizer.json, for Hugging Face tokenizers and transformers;
• cro-bpe-64k.tiktoken + config.json (also published as tiktoken.json), for tiktoken, which encodes the same ids several times faster.

Other sizes: cro-bpe-32k, cro-bpe-48k. Code, training pipeline and full evaluation: github.com/ibizaca/cro-bpe.

Results

On a holdout set of 18,131 Croatian documents (40 MB) from all sources. "Tokens vs. cro-bpe-32k" is the number of tokens needed for the same text; lossless means the tokenizer reproduces the text exactly:

Tokenizer	Vocabulary	bytes/token	tokens/word	tokens vs. cro-bpe-32k	lossless
cro-bpe-64k	64k	4.76	1.41	0.92	yes
cro-bpe-48k	48k	4.62	1.46	0.95	yes
cro-bpe-32k	32k	4.39	1.53	1.00	yes
BERTić (BCMS encoder)	32k	4.36	1.54	1.01	no
EuroLLM	128k	3.35	2.01	1.31	yes
GPT-4o (o200k_base)	200k	3.28	2.05	1.34	yes
Gemma 3	262k	3.21	2.10	1.37	yes
Llama 3	128k	2.74	2.46	1.61	yes
Llama 2	32k	2.50	2.70	1.76	yes

Metrics, throughput and the other baselines: docs/evaluation.md.

Usage

tokenizers

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("ibizaca/cro-bpe-64k")
encoding = tokenizer.encode("Dobar dan, kako ste?")
encoding.tokens
tokenizer.decode(encoding.ids)  # 'Dobar dan, kako ste?'

transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ibizaca/cro-bpe-64k")
ids = tokenizer("Dobar dan, kako ste?")["input_ids"]

No special tokens are added automatically; <|endoftext|> is the EOS token and <|pad|> the padding token.

tiktoken

tiktoken has no normalizer, so apply NFC first, as tokenizer.json does:

import json
import unicodedata

import tiktoken
from huggingface_hub import hf_hub_download
from tiktoken.load import load_tiktoken_bpe

config = json.load(open(hf_hub_download("ibizaca/cro-bpe-64k", "config.json")))
encoding = tiktoken.Encoding(
    name=config["name"],
    pat_str=config["pattern"],
    mergeable_ranks=load_tiktoken_bpe(hf_hub_download("ibizaca/cro-bpe-64k", config["rank_file"])),
    special_tokens=config["special_tokens"],
)

text = unicodedata.normalize("NFC", "Dobar dan, kako ste?")
ids = encoding.encode_ordinary(text)
encoding.decode(ids)  # 'Dobar dan, kako ste?'

Special tokens

The same ids in every vocabulary size:

Token	Id
<|endoftext|>	0
<|pad|>	1
<|mask|>	2
<|cls|>	3
<|sep|>	4
<|reserved_0|> … <|reserved_63|>	5–68

mask, cls and sep serve encoder models; the reserved slots leave room for chat, tool or fill-in-the-middle markers without resizing the vocabulary.

Tokenizer details

• Normalization: NFC.
• Pre-tokenization: a GPT-4-style split without the English contraction branch (letter runs with at most one leading non-letter, digit groups of up to three, punctuation runs, whitespace), then byte-level mapping. No prefix space is added.
• Byte-level, so it is lossless on any UTF-8 text.
• training_config.json holds the training settings, evaluation.json this model's full results.

Training data

Corpus	License
Croatian Wikipedia	CC-BY-SA 4.0
HPLT 3.0 hrv_Latn	CC0 1.0
FineWeb-2 hrv_Latn	ODC-By 1.0
CLASSLA-web.hr 2.0	CC0 1.0
CC-100 Croatian	Public domain
MaCoCu-hr 2.0	CC0 1.0
hrWaC 2.1	CC-BY-SA 4.0
ParlaMint-HR 5.0	CC-BY 4.0
MultiEURLEX hr	CC-BY 4.0
ParlaSpeech-HR 2.0 (text)	CC-BY 4.0

Sizes, versions and how the corpora were cleaned, deduplicated and sampled: docs/data.md.

Citation

@software{bizaca2026crobpe,
  author  = {Bizaca, Ivan},
  title   = {cro-bpe: A Byte-Level {BPE} Tokenizer for {Croatian}},
  year    = {2026},
  url     = {https://github.com/ibizaca/cro-bpe},
  license = {Apache-2.0}
}