lazos commited on
Commit
eb271df
·
verified ·
1 Parent(s): 4079ffe

docs: correct the fertility delta and spell out the eviction example

Browse files
Files changed (1) hide show
  1. README.md +20 -20
README.md CHANGED
@@ -66,13 +66,13 @@ pip install transformers datasets
66
  python benchmark.py
67
  ```
68
 
69
- It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five
70
- published tokenizers. Nothing from the pipeline that produced this vocabulary is involved.
71
 
72
  ### Paradigm sharing
73
 
74
- The property this vocabulary is built for. On [UniMorph Hungarian](https://github.com/unimorph/hun)
75
- (first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs):
76
 
77
  | tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form |
78
  |---|---|---|---|---|---|
@@ -82,13 +82,13 @@ The property this vocabulary is built for. On [UniMorph Hungarian](https://githu
82
  | FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 |
83
  | Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 |
84
 
85
- **one stem row + suffixes** asks both halves of the question at once: the lemma is a SINGLE row,
86
- *and* its inflected form begins with that row. So every mention of the word teaches one embedding.
 
87
 
88
- Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys
89
- `lemma is one token` -- that is the Qwen row, 54.9% and 2.0%. Swallowing whole inflected forms does
90
- the reverse. Only the conjunction says a paradigm shares a row, and it does so at the same token
91
- cost per form as the closest baseline.
92
 
93
  ### Fertility
94
 
@@ -102,16 +102,15 @@ Characters a token buys, on the first 2,000 documents of each dataset:
102
  | `open-web-math/open-web-math` | **2.923** | 2.394 | 2.385 | 3.412 | 3.669 |
103
  | `bigcode/starcoderdata` (python) | **2.465** | 1.721 | 2.454 | 3.205 | 4.121 |
104
 
105
- **On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a
106
- defect.** PULI buys more characters per token by keeping `házakban` as one row; this vocabulary
107
- spends an extra token to cut it into `▁ház` `ak` `ban`, and gets the paradigm table above in
108
- return. A model reading 4% fewer tokens that has to learn `ház` four times over has not come out
109
- ahead.
110
 
111
- Where the comparison is not about Hungarian morphology it inverts: on mathematics and code this
112
- vocabulary beats both Hungarian baselines, because a slice of each was in the fit corpus so
113
- technical text does not fall back to bytes. Qwen wins those outright, as an English-first
114
- tokenizer should.
115
 
116
  ### Special tokens
117
 
@@ -145,7 +144,8 @@ with a declared Hungarian lexicon deciding what survives:
145
  1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities,
146
  1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
147
  2. **Overfit** - a large Unigram vocabulary is trained on the corpus.
148
- 3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `okát` is not a row; `ok` and `át` are.
 
149
  4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should.
150
  5. **Prune** - rows the corpus never reaches are dropped.
151
 
 
66
  python benchmark.py
67
  ```
68
 
69
+ It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five published tokenizers.
70
+ Nothing from the pipeline that produced this vocabulary is involved.
71
 
72
  ### Paradigm sharing
73
 
74
+ The property this vocabulary is built for.
75
+ On [UniMorph Hungarian](https://github.com/unimorph/hun) first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs:
76
 
77
  | tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form |
78
  |---|---|---|---|---|---|
 
82
  | FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 |
83
  | Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 |
84
 
85
+ **one stem row + suffixes** asks both halves of the question at once:
86
+ the lemma is a SINGLE row, and its inflected form begins with that row.
87
+ So every mention of the word teaches one embedding.
88
 
89
+ Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys `lemma is one token`.
90
+ Swallowing whole inflected forms does the reverse.
91
+ Only the conjunction says a paradigm shares a row, and it does so at the same token cost per form as the closest baseline.
 
92
 
93
  ### Fertility
94
 
 
102
  | `open-web-math/open-web-math` | **2.923** | 2.394 | 2.385 | 3.412 | 3.669 |
103
  | `bigcode/starcoderdata` (python) | **2.465** | 1.721 | 2.454 | 3.205 | 4.121 |
104
 
105
+ **On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a defect.**
106
+ PULI buys more characters per token by cutting `házakban` into `ház` `akban`, keeping stem and endings fused;
107
+ this vocabulary spends one more token to cut it into `▁ház` `ak` `ban`, and gets the paradigm table above in return.
108
+ A model reading 4% fewer tokens that has to learn `ház` four times over has not come out ahead.
 
109
 
110
+ Where the comparison is not about Hungarian morphology it inverts:
111
+ on mathematics and code this vocabulary beats both Hungarian baselines,
112
+ because a slice of each was in the fit corpus so technical text does not fall back to bytes.
113
+ Qwen wins those outright, as an English-first tokenizer should.
114
 
115
  ### Special tokens
116
 
 
144
  1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities,
145
  1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
146
  2. **Overfit** - a large Unigram vocabulary is trained on the corpus.
147
+ 3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `magyarokat` becomes `▁magyar` `ok` `at`:
148
+ the whole ending `okat` is not a row, while the plural `ok` and the accusative `at` each are, and every other paradigm reuses them.
149
  4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should.
150
  5. **Prune** - rows the corpus never reaches are dropped.
151