| --- |
| language: |
| - hu |
| license: apache-2.0 |
| library_name: tokenizers |
| tags: |
| - picur |
| - tokenizer |
| - hungarian |
| - magyar |
| - morphology |
| - sentencepiece |
| - unigram |
| --- |
| |
| # picur-tokenizer |
|
|
| A 49,152-piece SentencePiece Unigram tokenizer for Hungarian, |
| fitted so that it segments on real morpheme boundaries, |
| rather than on whatever the compression objective prefers. |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("picur/picur-tokenizer") |
| tok.tokenize("a házakban lakó emberekkel") |
| # ['▁a', '▁ház', 'ak', 'ban', '▁lakó', '▁ember', 'ek', 'kel'] |
| ``` |
|
|
| ## Why another tokenizer |
|
|
| > - Nevezz meg három állatot. |
| > - Kutyácska, lovacska, verebecske. |
| > - Jó, de kicsinyítő képző nélkül. |
| > - Kecs, macs, szar. |
|
|
| Hungarian is agglutinative: one stem carries a long tail of suffixes, so `ház` (house) surfaces as `házban`, `házakban`, `házaimban`. |
| A tokenizer fitted on compression alone shatters that paradigm across unrelated rows, |
| and the model then has to learn `ház` several times over - once per spelling of the same word. |
| Training effort that should have gone into knowledge goes into re-learning morphology instead. |
|
|
| This vocabulary is fitted to keep a stem on one row and let the suffixes be their own pieces, |
| so a paradigm shares its knowledge instead of spreading it. |
|
|
| ## How it segments |
|
|
| | word | pieces | | |
| |---|---|---| |
| | `házakban` | `▁ház` `ak` `ban` | stem + plural + inessive | |
| | `könyvekkel` | `▁könyv` `ek` `kel` | stem + plural + instrumental | |
| | `emberekkel` | `▁ember` `ek` `kel` | the same two suffixes, the same rows | |
| | `magyarokat` | `▁magyar` `ok` `at` | back-vowel harmony of the pair above | |
| | `csapadékvíz` | `▁csapadék` `víz` | compound, cut at the declared seam | |
| | `tavak` | `▁tav` `ak` | v-stem alternation (`tó` -> `tav`) | |
|
|
| Vowel harmony is why `ban`/`ben`, `kel`/`kal` and `hoz`/`hez`/`höz` all exist as separate rows: |
| they are the same grammatical function wearing different vowels. |
| See [Tied rows](#tied-rows) for what is shipped to help a model learn them as one. |
|
|
| ## Measured |
|
|
| Every number below is reproducible from public artifacts by `benchmark.py` in this repository: |
|
|
| ``` |
| pip install transformers datasets |
| python benchmark.py |
| ``` |
|
|
| It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five published tokenizers. |
| Nothing from the pipeline that produced this vocabulary is involved. |
|
|
| ### Paradigm sharing |
|
|
| The property this vocabulary is built for. |
| On [UniMorph Hungarian](https://github.com/unimorph/hun) first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs: |
|
|
| | tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form | |
| |---|---|---|---|---|---| |
| | **picur/picur-tokenizer** | 49,152 | **45.4%** | **73.1%** | **55.1%** | 3.04 | |
| | NYTK/PULI-GPT-3SX | 50,000 | 24.5% | 54.9% | 38.5% | 2.98 | |
| | SZTAKI-HLT/hubert-base-cc | 32,001 | 23.6% | 55.5% | 28.1% | 3.07 | |
| | FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 | |
| | Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 | |
|
|
| **one stem row + suffixes** asks both halves of the question at once: |
| the lemma is a SINGLE row, and its inflected form begins with that row. |
| So every mention of the word teaches one embedding. |
|
|
| Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys `lemma is one token`. |
| Swallowing whole inflected forms does the reverse. |
| Only the conjunction says a paradigm shares a row, and it does so at the same token cost per form as the closest baseline. |
|
|
| ### Fertility |
|
|
| Characters a token buys, on the first 2,000 documents of each dataset: |
|
|
| | corpus | picur | PULI-GPT-3SX | huBERT | XLM-R | Qwen3 | |
| |---|---|---|---|---|---| |
| | `lazos/gutenberg-hu` | 3.740 | **4.050** | 3.915 | 3.381 | 2.417 | |
| | `lazos/gazette-hu` | 2.960 | **3.808** | 3.665 | 3.392 | 1.997 | |
| | `SZTAKI-HLT/HunSum-2-abstractive` | 4.001 | **4.544** | 4.487 | 3.803 | 2.367 | |
| | `open-web-math/open-web-math` | **2.953** | 2.394 | 2.385 | 3.412 | 3.669 | |
| | `bigcode/starcoderdata` (python) | **2.498** | 1.721 | 2.454 | 3.205 | 4.121 | |
|
|
| **On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a defect.** |
| PULI buys more characters per token by cutting `házakban` into `ház` `akban`, keeping stem and endings fused; |
| this vocabulary spends one more token to cut it into `▁ház` `ak` `ban`, and gets the paradigm table above in return. |
|
|
| Where the comparison is not about Hungarian morphology it inverts: |
| on mathematics and code this vocabulary beats both Hungarian baselines, |
| because a slice of each was in the fit corpus so technical text does not fall back to bytes. |
| Qwen wins those outright, as an English-first tokenizer should. |
|
|
| ### Special tokens |
|
|
| | id | token | | |
| |---|---|---| |
| | 0 | `<\|unk\|>` | | |
| | 1 | `<\|bos\|>` | one per document | |
| | 2 | `<\|eos\|>` | | |
| | 3 | `<\|pad\|>` | | |
| | 4 | `<\|im_start\|>` | ChatML turn open | |
| | 5 | `<\|im_end\|>` | ChatML turn close; the stop token for chat | |
| | 6-21 | `<\|reserved_0..15\|>` | unused, for later extension | |
|
|
| Tool-call markers (`<tools>`, `<tool_call>`, `<tool_response>` and their closers) are present as ordinary pieces, following the Hermes convention. |
|
|
| Note the ChatML split: `<|im_end|>` terminates a *turn*, `<|eos|>` terminates a *sequence*. A chat model should stop on `<|im_end|>`. |
|
|
| ### Tied rows |
|
|
| `ties.json` maps 1,357 allomorph rows onto the one row a model should learn first: `ban` -> `ben`, `höz` -> `hoz`, across 100 suffix families. |
| Vowel harmony makes these variants of one morpheme, |
| so initialising their embeddings from a shared base gives the model the paradigm for free rather than making it discover the relationship from data. |
|
|
| This is an initialisation aid, not part of tokenization, nothing reads it automatically. |
|
|
| ## How it was fitted |
|
|
| The vocabulary is overfitted well past its target size and then **selected down**, |
| with a declared Hungarian lexicon deciding what survives: |
|
|
| 1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities, |
| 1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords. |
| 2. **Overfit** - a large Unigram vocabulary is trained on the corpus. |
| 3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `magyarokat` becomes `▁magyar` `ok` `at`: |
| the whole ending `okat` is not a row, while the plural `ok` and the accusative `at` each are, and every other paradigm reuses them. |
| 4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should. |
| 5. **Prune** - rows the corpus never reaches are dropped. |
|
|
| Corpus: 700 Mchar drawn across 57 Hungarian sources: |
| encyclopedic, news, legal, parliamentary, academic, literary, subtitles, web and forum. |
| Plus small code and mathematics slices, so that technical text does not fall back to bytes. |
|
|
| ## Limitations |
|
|
| - **Hungarian-first.** English and code are handled but not optimized; other languages fall back to bytes. |
| - **Coverage, not rules, sets the floor.** A stem outside the 28,000-stem seed can still be mis-split, |
| and an alternating stem can lose on score: `művek` cuts as `mű` + `vek` where morphology says `műv` + `ek`. |
| - **Compound seams come from a declared lexicon**, so a compound absent from it gets no guaranteed seam. |
| - **Cased entity handling is partial.** 1,000 capitalised forms are seeded; beyond those, capitalisation costs a row. |
| - **Two losses are declared, not accidental.** Metaspace writes the word mark as |
| U+2581, so a literal U+2581 in the input is indistinguishable from a space. The |
| tokenizer declares a `Replace(U+2581, " ")` normalizer ahead of Metaspace, which |
| makes the collision part of the contract instead of a surprise - text carrying |
| block-drawing characters (a sparkline, say) round-trips with that one character |
| turned into a space. Leading whitespace on an input is dropped for the same |
| reason: Metaspace prepends a mark unconditionally. Everything else round-trips |
| byte-exactly, including every script outside Latin, via byte fallback. |
|
|
| ## Attribution |
|
|
| This tokenizer is a derivative work of the following resouces, see `NOTICE`. |
|
|
| - **[morphdb.hu](https://github.com/morphdb/morphdb.hu)** - CC-BY 2.5, (c) Nemeth |
| Laszlo and Godo Ferenc. The declared lexicon: 351,917 headwords, 37,042 compound |
| seams, alternation classes and vowel harmony. This decided what the fit protects |
| and what it evicts. |
| - **[hunmorph-foma](https://github.com/r0ller/hunmorph-foma)** - BSD-2-Clause, |
| (c) 2016 r0ller. Finite-state transducers that analysed corpus words into lemma |
| plus tags, bootstrapping the segmentation this vocabulary is scored against. |
| - **[UniMorph Hungarian](https://github.com/unimorph/hun)** - CC BY-SA 3.0, derived |
| from English Wiktionary. Used by `benchmark.py` only: it is scored against and |
| never built from, which is what makes the paradigm table above independent of |
| everything that produced the vocabulary. |
|
|
| The tokenizer itself is Apache-2.0. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{picur-tokenizer, |
| title = {picur-tokenizer: a morphology-aware Hungarian tokenizer}, |
| author = {Bencz, Lajos}, |
| year = {2026}, |
| url = {https://huggingface.co/picur/picur-tokenizer} |
| } |
| ``` |
|
|
