docs: correct the fertility delta and spell out the eviction example
Browse files
README.md
CHANGED
|
@@ -66,13 +66,13 @@ pip install transformers datasets
|
|
| 66 |
python benchmark.py
|
| 67 |
```
|
| 68 |
|
| 69 |
-
It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five
|
| 70 |
-
|
| 71 |
|
| 72 |
### Paradigm sharing
|
| 73 |
|
| 74 |
-
The property this vocabulary is built for.
|
| 75 |
-
(first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs
|
| 76 |
|
| 77 |
| tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form |
|
| 78 |
|---|---|---|---|---|---|
|
|
@@ -82,13 +82,13 @@ The property this vocabulary is built for. On [UniMorph Hungarian](https://githu
|
|
| 82 |
| FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 |
|
| 83 |
| Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 |
|
| 84 |
|
| 85 |
-
**one stem row + suffixes** asks both halves of the question at once:
|
| 86 |
-
|
|
|
|
| 87 |
|
| 88 |
-
Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
cost per form as the closest baseline.
|
| 92 |
|
| 93 |
### Fertility
|
| 94 |
|
|
@@ -102,16 +102,15 @@ Characters a token buys, on the first 2,000 documents of each dataset:
|
|
| 102 |
| `open-web-math/open-web-math` | **2.923** | 2.394 | 2.385 | 3.412 | 3.669 |
|
| 103 |
| `bigcode/starcoderdata` (python) | **2.465** | 1.721 | 2.454 | 3.205 | 4.121 |
|
| 104 |
|
| 105 |
-
**On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a
|
| 106 |
-
|
| 107 |
-
spends
|
| 108 |
-
|
| 109 |
-
ahead.
|
| 110 |
|
| 111 |
-
Where the comparison is not about Hungarian morphology it inverts:
|
| 112 |
-
vocabulary beats both Hungarian baselines,
|
| 113 |
-
technical text does not fall back to bytes.
|
| 114 |
-
tokenizer should.
|
| 115 |
|
| 116 |
### Special tokens
|
| 117 |
|
|
@@ -145,7 +144,8 @@ with a declared Hungarian lexicon deciding what survives:
|
|
| 145 |
1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities,
|
| 146 |
1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
|
| 147 |
2. **Overfit** - a large Unigram vocabulary is trained on the corpus.
|
| 148 |
-
3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `
|
|
|
|
| 149 |
4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should.
|
| 150 |
5. **Prune** - rows the corpus never reaches are dropped.
|
| 151 |
|
|
|
|
| 66 |
python benchmark.py
|
| 67 |
```
|
| 68 |
|
| 69 |
+
It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five published tokenizers.
|
| 70 |
+
Nothing from the pipeline that produced this vocabulary is involved.
|
| 71 |
|
| 72 |
### Paradigm sharing
|
| 73 |
|
| 74 |
+
The property this vocabulary is built for.
|
| 75 |
+
On [UniMorph Hungarian](https://github.com/unimorph/hun) first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs:
|
| 76 |
|
| 77 |
| tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form |
|
| 78 |
|---|---|---|---|---|---|
|
|
|
|
| 82 |
| FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 |
|
| 83 |
| Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 |
|
| 84 |
|
| 85 |
+
**one stem row + suffixes** asks both halves of the question at once:
|
| 86 |
+
the lemma is a SINGLE row, and its inflected form begins with that row.
|
| 87 |
+
So every mention of the word teaches one embedding.
|
| 88 |
|
| 89 |
+
Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys `lemma is one token`.
|
| 90 |
+
Swallowing whole inflected forms does the reverse.
|
| 91 |
+
Only the conjunction says a paradigm shares a row, and it does so at the same token cost per form as the closest baseline.
|
|
|
|
| 92 |
|
| 93 |
### Fertility
|
| 94 |
|
|
|
|
| 102 |
| `open-web-math/open-web-math` | **2.923** | 2.394 | 2.385 | 3.412 | 3.669 |
|
| 103 |
| `bigcode/starcoderdata` (python) | **2.465** | 1.721 | 2.454 | 3.205 | 4.121 |
|
| 104 |
|
| 105 |
+
**On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a defect.**
|
| 106 |
+
PULI buys more characters per token by cutting `házakban` into `ház` `akban`, keeping stem and endings fused;
|
| 107 |
+
this vocabulary spends one more token to cut it into `▁ház` `ak` `ban`, and gets the paradigm table above in return.
|
| 108 |
+
A model reading 4% fewer tokens that has to learn `ház` four times over has not come out ahead.
|
|
|
|
| 109 |
|
| 110 |
+
Where the comparison is not about Hungarian morphology it inverts:
|
| 111 |
+
on mathematics and code this vocabulary beats both Hungarian baselines,
|
| 112 |
+
because a slice of each was in the fit corpus so technical text does not fall back to bytes.
|
| 113 |
+
Qwen wins those outright, as an English-first tokenizer should.
|
| 114 |
|
| 115 |
### Special tokens
|
| 116 |
|
|
|
|
| 144 |
1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities,
|
| 145 |
1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
|
| 146 |
2. **Overfit** - a large Unigram vocabulary is trained on the corpus.
|
| 147 |
+
3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `magyarokat` becomes `▁magyar` `ok` `at`:
|
| 148 |
+
the whole ending `okat` is not a row, while the plural `ok` and the accusative `at` each are, and every other paradigm reuses them.
|
| 149 |
4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should.
|
| 150 |
5. **Prune** - rows the corpus never reaches are dropped.
|
| 151 |
|