a2B: reproducible seed, public benchmarks, benchmark.py

README.md

@@ -59,23 +59,59 @@ See [Tied rows](#tied-rows) for what is shipped to help a model learn them as on
 
 ## Measured
 
-Scored on 20,000 documents withheld from the fit (1.42M weighted pairs),
-against morpheme segmentations derived from finite-state analyses of the same sources:
+Every number below is reproducible from public artifacts by `benchmark.py` in this repository:
 
-| metric | value | what it means |
-|---|---|---|
-| **morph-f1** | **79.9%** | precision 90.0% / recall 71.7% against verified morpheme boundaries |
-| **stem seam** | **79.8%** | inflected words cut at the stem boundary |
-| **one-piece function** | **71.3%** | grammatical functions that are a single row (98 functions) |
-| **single-token lemma** | **94.9%** | dictionary headwords that survive as one piece |
-| **paradigm spread** | **1.23 rows** | distinct stem rows a paradigm spends; 1.0 is ideal |
-| chars/token | 3.662 | fertility, on Hungarian prose |
-| round-trip failures | 0 | on the full evaluation sample |
-| byte fallback | 0.084% | |
-
-Fertility is reported for reference only.
-It was **not** the objective, and this vocabulary deliberately trades a little of it for grammar:
-the fit that maximised chars/token scored 53.2% morph-f1, versus 79.9% here.
+```
+pip install transformers datasets
+python benchmark.py
+```
+
+It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five
+published tokenizers. Nothing from the pipeline that produced this vocabulary is involved.
+
+### Paradigm sharing
+
+The property this vocabulary is built for. On [UniMorph Hungarian](https://github.com/unimorph/hun)
+(first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs):
+
+| tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form |
+|---|---|---|---|---|---|
+| **picur/picur-tokenizer** | 49,152 | **44.1%** | **72.4%** | **54.3%** | 3.04 |
+| NYTK/PULI-GPT-3SX | 50,000 | 24.5% | 54.9% | 38.5% | 2.98 |
+| SZTAKI-HLT/hubert-base-cc | 32,001 | 23.6% | 55.5% | 28.1% | 3.07 |
+| FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 |
+| Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 |
+
+**one stem row + suffixes** asks both halves of the question at once: the lemma is a SINGLE row,
+*and* its inflected form begins with that row. So every mention of the word teaches one embedding.
+
+Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys
+`lemma is one token` -- that is the Qwen row, 54.9% and 2.0%. Swallowing whole inflected forms does
+the reverse. Only the conjunction says a paradigm shares a row, and it does so at the same token
+cost per form as the closest baseline.
+
+### Fertility
+
+Characters a token buys, on the first 2,000 documents of each dataset:
+
+| corpus | picur | PULI-GPT-3SX | huBERT | XLM-R | Qwen3 |
+|---|---|---|---|---|---|
+| `lazos/gutenberg-hu` | 3.761 | **4.050** | 3.915 | 3.381 | 2.417 |
+| `lazos/gazette-hu` | 2.959 | **3.808** | 3.665 | 3.392 | 1.997 |
+| `SZTAKI-HLT/HunSum-2-abstractive` | 3.988 | **4.544** | 4.487 | 3.803 | 2.367 |
+| `open-web-math/open-web-math` | **2.923** | 2.394 | 2.385 | 3.412 | 3.669 |
+| `bigcode/starcoderdata` (python) | **2.465** | 1.721 | 2.454 | 3.205 | 4.121 |
+
+**On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a
+defect.** PULI buys more characters per token by keeping `házakban` as one row; this vocabulary
+spends an extra token to cut it into `▁ház` `ak` `ban`, and gets the paradigm table above in
+return. A model reading 4% fewer tokens that has to learn `ház` four times over has not come out
+ahead.
+
+Where the comparison is not about Hungarian morphology it inverts: on mathematics and code this
+vocabulary beats both Hungarian baselines, because a slice of each was in the fit corpus so
+technical text does not fall back to bytes. Qwen wins those outright, as an English-first
+tokenizer should.
 
 ### Special tokens
 
@@ -95,7 +131,7 @@ Note the ChatML split: `<|im_end|>` terminates a *turn*, `<|eos|>` terminates a
 
 ### Tied rows
 
-`ties.json` maps 1,365 allomorph rows onto the one row a model should learn first: `ban` -> `ben`, `höz` -> `hoz`, across 101 suffix families.
+`ties.json` maps 1,353 allomorph rows onto the one row a model should learn first: `ban` -> `ben`, `höz` -> `hoz`, across 100 suffix families.
 Vowel harmony makes these variants of one morpheme,
 so initialising their embeddings from a shared base gives the model the paradigm for free rather than making it discover the relationship from data.
 
@@ -106,8 +142,8 @@ This is an initialisation aid, not part of tokenization, nothing reads it automa
 The vocabulary is overfitted well past its target size and then **selected down**,
 with a declared Hungarian lexicon deciding what survives:
 
-1. **Seed** - 32,910 pieces required up front: 28,000 frequent stems, 1,000 named entities,
-   1,000 capitalised forms, 445 suffixes, 2,011 alternating stems and 448 programming keywords.
+1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities,
+   1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
 2. **Overfit** - a large Unigram vocabulary is trained on the corpus.
 3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `okát` is not a row; `ok` and `át` are.
 4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should.
@@ -123,7 +159,7 @@ Plus small code and mathematics slices, so that technical text does not fall bac
 - **Coverage, not rules, sets the floor.** A stem outside the 28,000-stem seed can still be mis-split,
   and an alternating stem can lose on score: `művek` cuts as `mű` + `vek` where morphology says `műv` + `ek`.
 - **Compound seams come from a declared lexicon**, so a compound absent from it gets no guaranteed seam.
-- **Cased entity handling is partial.** 4,000 capitalised forms are seeded; beyond those, capitalisation costs a row.
+- **Cased entity handling is partial.** 1,000 capitalised forms are seeded; beyond those, capitalisation costs a row.
 - **Two losses are declared, not accidental.** Metaspace writes the word mark as
   U+2581, so a literal U+2581 in the input is indistinguishable from a space. The
   tokenizer declares a `Replace(U+2581, " ")` normalizer ahead of Metaspace, which
@@ -144,6 +180,10 @@ This tokenizer is a derivative work of the following resouces, see `NOTICE`.
 - **[hunmorph-foma](https://github.com/r0ller/hunmorph-foma)** - BSD-2-Clause,
   (c) 2016 r0ller. Finite-state transducers that analysed corpus words into lemma
   plus tags, bootstrapping the segmentation this vocabulary is scored against.
+- **[UniMorph Hungarian](https://github.com/unimorph/hun)** - CC BY-SA 3.0, derived
+  from English Wiktionary. Used by `benchmark.py` only: it is scored against and
+  never built from, which is what makes the paradigm table above independent of
+  everything that produced the vocabulary.
 
 The tokenizer itself is Apache-2.0.
 

benchmark.py

@@ -0,0 +1,124 @@
+"""Reproduce the tables in this model card.
+
+    pip install transformers datasets
+    python benchmark.py            # both tables
+    python benchmark.py paradigms  # UniMorph only
+    python benchmark.py fertility  # corpora only
+
+Everything it reads is public and pinned: a UniMorph commit, five Hugging Face datasets at a fixed
+document count, and five published tokenizers. No part of the pipeline that produced this
+vocabulary is involved, which is the point -- the numbers are meant to be checkable by someone who
+has never seen it.
+"""
+import re
+import sys
+import unicodedata
+import urllib.request
+from collections import defaultdict
+
+from datasets import load_dataset
+from transformers import AutoTokenizer
+
+TOKENIZERS = (
+    "picur/picur-tokenizer",
+    "NYTK/PULI-GPT-3SX",
+    "SZTAKI-HLT/hubert-base-cc",
+    "Qwen/Qwen3-8B",
+    "FacebookAI/xlm-roberta-base",
+)
+
+UNIMORPH = ("https://raw.githubusercontent.com/unimorph/hun/"
+            "20d5e04211bbab3dd689c99637356a8eab7c5fd7/hun")
+LEMMAS = 20_000
+
+CORPORA = (
+    ("lazos/gutenberg-hu", None, "data", "text"),
+    ("lazos/gazette-hu", None, "data", "text"),
+    ("SZTAKI-HLT/HunSum-2-abstractive", None, "data", "article"),
+    ("open-web-math/open-web-math", None, "data", "text"),
+    ("bigcode/starcoderdata", "python", None, "content"),
+)
+DOCUMENTS = 2_000
+MIN_CHARS = 200
+WORD = re.compile(r"\w+", re.UNICODE)
+
+
+def load():
+    found = {}
+    for name in TOKENIZERS:
+        try:
+            found[name] = AutoTokenizer.from_pretrained(name)
+        except Exception as error:            # a gated model is skipped, not fatal
+            print(f"skip {name}: {type(error).__name__}", file=sys.stderr)
+    return found
+
+
+def paradigms():
+    """Does an inflected form begin with its lemma, and is that lemma a single row?"""
+    with urllib.request.urlopen(UNIMORPH, timeout=120) as handle:
+        raw = handle.read().decode("utf-8")
+    table = defaultdict(list)
+    for line in raw.splitlines():
+        parts = line.split("\t")
+        if len(parts) < 3 or not parts[0] or not parts[1]:
+            continue
+        if len(table) >= LEMMAS and parts[0] not in table:
+            continue
+        if parts[1] != parts[0]:
+            table[parts[0]].append(parts[1])
+    pairs = sum(len(v) for v in table.values())
+    print(f"\nUniMorph hun: {len(table):,} lemmas, {pairs:,} lemma/form pairs")
+    print(f"{'tokenizer':32s} {'stem row':>9s} {'prefix':>8s} {'lemma=1':>8s} {'tok/form':>9s}")
+    for name, tok in load().items():
+        cache = {}
+
+        def cut(word, tok=tok, cache=cache):
+            if word not in cache:
+                cache[word] = tuple(tok.tokenize(" " + word))
+            return cache[word]
+
+        rows = shared = whole = cost = 0
+        for lemma, forms in table.items():
+            head = cut(lemma)
+            entire = len(head) == 1
+            whole += entire
+            for form in forms:
+                piece = cut(form)
+                keeps = len(piece) >= len(head) and piece[:len(head)] == head
+                shared += keeps
+                rows += keeps and entire
+                cost += len(piece)
+        print(f"{name:32s} {rows/pairs:8.1%} {shared/pairs:8.1%} "
+              f"{whole/len(table):8.1%} {cost/pairs:9.2f}")
+
+
+def fertility():
+    """Characters a token buys, on a fixed prefix of each dataset."""
+    loaded = load()
+    for repo, config, folder, column in CORPORA:
+        stream = load_dataset(repo, config, data_dir=folder, split="train", streaming=True)
+        texts = []
+        for row in stream:
+            text = (row.get(column) or "").strip()
+            if len(text) >= MIN_CHARS:
+                texts.append(unicodedata.normalize("NFC", text))
+            if len(texts) >= DOCUMENTS:
+                break
+        chars = sum(map(len, texts))
+        words = [w for t in texts for w in WORD.findall(t)]
+        print(f"\n{repo}  {len(texts):,} documents, {chars/1e6:.2f} Mchar")
+        for name, tok in loaded.items():
+            total = sum(len(tok.encode(t, add_special_tokens=False)) for t in texts)
+            cost = {w: len(tok.encode(" " + w, add_special_tokens=False)) for w in set(words)}
+            counts = [cost[w] for w in words]
+            print(f"    {name:32s} {chars/total:6.3f} chars/token   "
+                  f"{sum(counts)/len(counts):4.2f} pieces/word   "
+                  f"{sum(n >= 4 for n in counts)/len(counts):5.1%} need 4+")
+
+
+if __name__ == "__main__":
+    wanted = sys.argv[1] if len(sys.argv) > 1 else "all"
+    if wanted in ("all", "paradigms"):
+        paradigms()
+    if wanted in ("all", "fertility"):
+        fertility()

ties.json

@@ -1 +1 @@
-{"832": 289, "881": 1507, "1036": 363, "1430": 33281, "1464": 1264, "1836": 1427, "2002": 1427, "2104": 1610, "2127": 555, "2927": 393, "3073": 868, "3300": 2543, "3573": 2520, "3776": 555, "3946": 555, "4080": 2826, "4085": 1264, "4229": 6547, "4580": 2257, "4651": 1023, "4834": 3774, "4882": 3546, "5204": 5108, "5573": 44976, "5643": 4760, "5992": 4828, "6110": 2826, "6128": 621, "6391": 1722, "6717": 1264, "6760": 4760, "7534": 345, "7739": 2210, "8006": 3268, "8129": 4590, "8315": 5640, "8488": 2197, "8640": 7439, "8724": 3063, "8833": 6105, "8900": 4167, "8903": 8646, "9102": 594, "9173": 7271, "9395": 7952, "9452": 5057, "9502": 5143, "9509": 1127, "9521": 1388, "9662": 6302, "9816": 8646, "9893": 1678, "10073": 2281, "10210": 7343, "10307": 8042, "10433": 8646, "10434": 5423, "10527": 10361, "10628": 1989, "10671": 723, "10879": 8898, "10937": 5128, "11113": 2126, "11506": 2102, "11582": 9473, "11618": 2826, "11882": 4760, "12155": 4986, "12475": 6823, "12527": 17365, "12687": 8917, "12718": 10840, "12842": 651, "12894": 9593, "13191": 21879, "13237": 10739, "13250": 8949, "13316": 11133, "13359": 10341, "13634": 8221, "13807": 39934, "13909": 16366, "14253": 10824, "14263": 8773, "14649": 4590, "14657": 9846, "14907": 12891, "14957": 6069, "14997": 4590, "15007": 9794, "15053": 10819, "15137": 10915, "15204": 34357, "15221": 6474, "15265": 1118, "15430": 4893, "15731": 8841, "15839": 2302, "15910": 3492, "16522": 2893, "16550": 5206, "16562": 4655, "16609": 36674, "16685": 8898, "16816": 2653, "16941": 2137, "16990": 16847, "16994": 9473, "17108": 21949, "17126": 3093, "17266": 28660, "17364": 13208, "17462": 1338, "17839": 9672, "17934": 6740, "18070": 16802, "18080": 8465, "18082": 19743, "18406": 1425, "18520": 16034, "18766": 7389, "18936": 30722, "19215": 30263, "19340": 8646, "19438": 4434, "19469": 3091, "19653": 3415, "19697": 40396, "19702": 9492, "19734": 18767, "19815": 2281, "19839": 2543, "19867": 17024, "20101": 4089, "20185": 2985, "20248": 27793, "20317": 44763, "20392": 21811, "20778": 1221, "20823": 8898, "20934": 6816, "20942": 14127, "21037": 26418, "21056": 30456, "21133": 823, "21232": 33309, "21251": 27899, "21322": 39063, "21622": 5108, "21950": 14062, "21966": 8797, "22105": 1599, "22134": 22031, "22136": 6087, "22164": 18973, "22379": 1290, "22417": 6081, "22486": 26600, "22508": 30386, "22530": 4844, "22568": 18288, "22606": 5780, "22658": 14704, "23156": 555, "23228": 1610, "23317": 13333, "23420": 10755, "23527": 11361, "23542": 12561, "23543": 11709, "23644": 19396, "23657": 41454, "24164": 28112, "24276": 23385, "24596": 15987, "24634": 46767, "24861": 6740, "24881": 35159, "24893": 15829, "24924": 40261, "24942": 30836, "25194": 21015, "25402": 15082, "25710": 3320, "25772": 5647, "25825": 4761, "25892": 4344, "26025": 26591, "26081": 1498, "26087": 13705, "26269": 20956, "26290": 32329, "26310": 26484, "26361": 18185, "26392": 34852, "26419": 30692, "26420": 5128, "26455": 36972, "26459": 35515, "26463": 22759, "26566": 34219, "26590": 31181, "26746": 28493, "26798": 44758, "26823": 28500, "26848": 22668, "26865": 16612, "26867": 33750, "26997": 924, "27079": 26682, "27083": 28739, "27097": 19451, "27174": 31214, "27213": 27629, "27223": 26708, "27250": 28228, "27284": 3774, "27304": 13321, "27318": 37417, "27397": 4205, "27439": 6764, "27461": 26529, "27526": 7271, "27533": 1691, "27560": 31992, "27602": 20750, "27619": 39209, "27645": 31321, "27660": 29665, "27784": 7217, "27824": 15559, "27844": 30826, "27905": 29095, "27943": 2826, "27945": 28027, "28034": 31811, "28092": 7384, "28098": 18306, "28105": 12286, "28187": 31979, "28195": 3650, "28212": 47298, "28298": 47291, "28338": 14077, "28384": 9837, "28468": 26595, "28524": 32469, "28525": 11650, "28529": 28639, "28588": 4753, "28654": 25887, "28684": 18269, "28754": 35961, "28814": 42090, "28857": 43190, "28870": 43668, "28879": 3320, "28928": 20644, "28944": 28317, "29074": 27608, "29102": 33822, "29110": 28592, "29186": 6894, "29194": 23249, "29218": 45476, "29264": 38966, "29395": 34476, "29442": 29358, "29464": 1501, "29481": 31590, "29486": 30770, "29538": 22043, "29608": 30175, "29640": 41593, "29653": 31919, "29757": 26741, "29785": 28983, "29802": 41578, "29803": 44193, "29808": 47285, "29817": 2514, "29820": 37030, "29847": 30195, "29869": 24563, "29914": 34850, "29945": 31800, "29968": 30161, "29990": 17417, "29998": 43952, "30026": 29860, "30054": 6759, "30061": 42098, "30089": 21516, "30104": 28168, "30199": 48562, "30202": 41558, "30215": 47470, "30235": 26725, "30236": 25267, "30239": 34814, "30244": 4463, "30301": 48438, "30334": 28217, "30337": 29329, "30374": 3185, "30441": 18602, "30509": 46967, "30520": 40746, "30538": 47143, "30541": 31926, "30556": 48570, "30572": 2850, "30595": 27088, "30599": 14796, "30632": 26715, "30650": 16903, "30698": 36216, "30706": 44462, "30737": 30736, "30780": 31521, "30783": 34161, "30876": 49024, "30894": 15903, "30915": 46249, "30935": 48175, "30968": 27363, "31017": 32937, "31019": 48782, "31030": 25380, "31035": 19154, "31111": 32286, "31149": 28499, "31232": 41497, "31266": 17024, "31350": 32430, "31383": 3443, "31392": 41083, "31393": 16327, "31456": 5272, "31463": 6750, "31474": 29090, "31479": 37949, "31494": 38497, "31571": 43076, "31630": 46050, "31682": 46346, "31701": 6951, "31710": 22870, "31728": 39462, "31742": 26514, "31764": 30961, "31823": 22583, "31836": 17024, "31839": 4985, "31869": 42560, "31877": 47292, "31897": 44996, "31924": 11940, "31933": 33710, "31955": 30255, "31956": 1454, "31971": 46551, "31995": 3147, "31997": 45514, "32011": 35866, "32020": 9324, "32028": 26369, "32039": 32426, "32063": 48058, "32076": 27186, "32105": 47987, "32120": 12410, "32146": 16378, "32159": 49056, "32163": 20015, "32167": 28538, "32186": 45487, "32196": 43143, "32217": 3733, "32277": 43833, "32311": 27513, "32336": 36095, "32374": 47885, "32376": 30664, "32377": 31928, "32384": 34978, "32415": 1411, "32441": 48373, "32443": 41511, "32445": 42827, "32485": 32313, "32489": 46484, "32499": 47772, "32524": 43878, "32536": 31941, "32549": 28236, "32558": 32397, "32559": 48097, "32586": 47932, "32591": 31373, "32603": 2523, "32650": 4699, "32727": 2520, "32748": 2147, "32794": 8648, "32829": 2716, "32863": 17558, "32961": 700, "32965": 1118, "33011": 3053, "33028": 41222, "33029": 4828, "33032": 1327, "33049": 8367, "33084": 3147, "33129": 6503, "33163": 7551, "33165": 1206, "33182": 6925, "33247": 15234, "33263": 10146, "33269": 10852, "33295": 23927, "33315": 1937, "33369": 24582, "33382": 4842, "33424": 16381, "33445": 5864, "33486": 15966, "33547": 5681, "33555": 8873, "33565": 1787, "33612": 20692, "33622": 7282, "33655": 5398, "33704": 1144, "33705": 35838, "33720": 16212, "33728": 2661, "33745": 11325, "33761": 4578, "33787": 9013, "33808": 21453, "33839": 32799, "33941": 3486, "33944": 2624, "33978": 33840, "33982": 16752, "34049": 6053, "34065": 7185, "34073": 18428, "34077": 35518, "34088": 35536, "34107": 3062, "34177": 3745, "34182": 35023, "34202": 8666, "34211": 8572, "34262": 35236, "34276": 33110, "34280": 28853, "34290": 13466, "34300": 9568, "34312": 25010, "34323": 16436, "34324": 6083, "34366": 41904, "34385": 35600, "34387": 23915, "34407": 35030, "34427": 8175, "34452": 36323, "34463": 460, "34479": 10679, "34485": 26579, "34502": 10304, "34515": 14214, "34537": 12353, "34560": 25740, "34561": 36733, "34591": 35095, "34627": 26788, "34646": 29628, "34658": 34421, "34662": 34414, "34752": 25842, "34773": 5108, "34783": 44924, "34800": 12468, "34819": 34655, "34853": 14682, "34892": 25338, "34958": 47493, "35037": 30875, "35053": 37967, "35056": 35014, "35063": 6476, "35115": 36349, "35141": 11194, "35142": 11125, "35199": 15982, "35232": 13107, "35252": 19919, "35285": 30335, "35330": 18151, "35339": 32923, "35385": 14685, "35410": 16081, "35414": 21856, "35423": 36173, "35426": 17943, "35437": 7290, "35442": 34620, "35451": 22353, "35477": 36502, "35478": 25216, "35480": 45607, "35485": 8251, "35491": 30473, "35525": 44386, "35528": 41137, "35546": 5709, "35584": 34066, "35596": 16847, "35643": 46580, "35692": 46606, "35712": 29609, "35721": 35990, "35722": 37480, "35723": 36417, "35737": 32002, "35741": 26431, "35749": 579, "35769": 47305, "35780": 27221, "35791": 43538, "35795": 47867, "35798": 32673, "35830": 15393, "35854": 30111, "35862": 605, "35873": 27989, "35880": 26246, "35883": 47406, "35903": 3774, "35909": 10844, "35950": 41722, "35952": 34540, "35956": 14645, "35984": 42726, "35986": 15765, "35992": 842, "36021": 692, "36028": 36451, "36055": 28732, "36072": 48496, "36084": 14507, "36086": 26700, "36114": 29951, "36118": 17394, "36135": 31275, "36136": 8931, "36140": 8898, "36160": 6302, "36185": 35526, "36191": 38036, "36242": 32287, "36261": 31026, "36282": 7439, "36284": 35228, "36289": 35825, "36295": 47404, "36296": 26432, "36305": 4249, "36329": 14498, "36353": 31938, "36358": 44750, "36367": 48847, "36386": 37735, "36393": 32786, "36410": 37027, "36420": 36280, "36489": 7383, "36496": 19585, "36523": 35034, "36536": 24559, "36570": 17024, "36579": 28436, "36589": 16981, "36607": 47695, "36624": 37016, "36627": 48983, "36636": 27123, "36661": 47893, "36679": 20891, "36687": 26394, "36717": 31806, "36757": 36833, "36828": 46852, "36841": 47122, "36850": 45114, "36853": 7271, "36921": 4865, "36930": 1526, "36933": 46678, "36944": 36049, "36945": 39083, "36951": 9283, "36967": 34440, "36975": 25047, "36993": 37181, "36998": 48166, "37018": 40467, "37024": 28013, "37054": 25267, "37063": 1002, "37068": 19141, "37157": 41150, "37224": 22554, "37245": 36315, "37283": 8170, "37291": 38219, "37352": 7271, "37394": 478, "37418": 17487, "37459": 2589, "37481": 1579, "37584": 28225, "37610": 6302, "37687": 19585, "37715": 37953, "37736": 1128, "37753": 2059, "37841": 8029, "37868": 32030, "37897": 2543, "37900": 1467, "37928": 27021, "37934": 27970, "37938": 38188, "37983": 38262, "37992": 35452, "38000": 4659, "38020": 38042, "38053": 1610, "38069": 1833, "38091": 4557, "38092": 4230, "38102": 7186, "38195": 35739, "38201": 26986, "38221": 33601, "38235": 2126, "38245": 26842, "38247": 2681, "38287": 2117, "38288": 11416, "38327": 24909, "38331": 9031, "38341": 8784, "38392": 35375, "38411": 13192, "38413": 38837, "38452": 38744, "38473": 31122, "38482": 3443, "38516": 21876, "38577": 15202, "38600": 24884, "38631": 1721, "38654": 9094, "38743": 1671, "38777": 42658, "38822": 9531, "38828": 1640, "38884": 35428, "38890": 24835, "38917": 35705, "38986": 11351, "39003": 34890, "39006": 9194, "39012": 19145, "39016": 39334, "39022": 32967, "39028": 28894, "39042": 9482, "39045": 521, "39050": 10998, "39062": 38814, "39074": 29852, "39107": 26558, "39114": 39437, "39148": 27828, "39199": 20404, "39207": 35858, "39265": 14255, "39389": 26719, "39432": 21774, "39452": 43734, "39459": 25267, "39465": 6618, "39478": 21816, "39496": 2917, "39527": 38011, "39534": 25078, "39538": 41303, "39556": 36744, "39560": 16974, "39574": 4098, "39578": 40379, "39638": 40663, "39644": 17287, "39660": 25047, "39686": 3784, "39719": 35672, "39751": 38386, "39761": 27212, "39771": 7536, "39777": 2876, "39791": 28134, "39797": 15514, "39810": 15924, "39889": 5687, "39906": 40838, "39960": 17492, "39964": 1386, "39989": 39274, "39991": 7765, "40077": 18159, "40088": 1048, "40091": 41017, "40131": 13436, "40135": 40093, "40142": 14854, "40147": 4502, "40155": 39341, "40157": 36381, "40224": 44441, "40234": 44450, "40264": 14581, "40312": 7001, "40320": 5149, "40364": 39891, "40384": 39254, "40400": 34696, "40402": 43471, "40411": 39277, "40461": 31066, "40495": 4244, "40501": 35591, "40535": 36187, "40537": 7673, "40561": 34863, "40569": 3302, "40573": 12332, "40605": 25162, "40638": 36508, "40664": 38941, "40686": 8327, "40698": 38963, "40699": 32475, "40739": 15312, "40821": 39932, "40879": 1184, "40919": 19846, "41007": 45018, "41012": 3459, "41019": 41050, "41046": 42176, "41047": 16770, "41051": 36842, "41086": 31031, "41113": 34599, "41138": 39374, "41143": 8919, "41144": 43968, "41151": 42031, "41153": 1716, "41155": 40311, "41186": 39931, "41194": 16053, "41219": 4491, "41221": 12924, "41236": 36120, "41274": 42314, "41279": 31271, "41280": 43919, "41296": 45915, "41300": 42342, "41337": 39947, "41338": 42270, "41357": 40152, "41370": 40165, "41489": 47499, "41492": 35603, "41576": 36383, "41615": 37131, "41669": 27070, "41746": 36017, "41769": 19877, "41780": 1210, "41814": 38448, "41820": 41366, "41828": 15669, "41843": 35809, "41853": 30540, "41854": 15041, "41858": 44332, "41871": 40470, "41876": 42611, "41893": 27453, "41895": 41446, "41920": 41467, "41961": 43172, "42039": 41098, "42043": 43058, "42070": 34786, "42071": 30897, "42119": 35391, "42128": 43984, "42168": 26568, "42187": 44566, "42203": 42908, "42219": 44505, "42232": 40222, "42240": 23075, "42243": 31896, "42277": 42217, "42309": 30672, "42418": 15877, "42433": 4018, "42438": 42545, "42439": 5084, "42440": 40957, "42445": 40244, "42460": 8069, "42463": 8701, "42493": 13975, "42521": 26512, "42524": 20799, "42549": 42760, "42556": 37504, "42571": 31083, "42595": 32381, "42662": 41660, "42721": 43406, "42782": 45861, "42806": 40672, "42812": 39076, "42858": 31886, "42861": 41402, "42891": 9380, "42946": 23432, "42957": 6694, "42981": 39212, "42993": 42033, "43005": 41528, "43021": 42059, "43063": 43917, "43073": 25148, "43100": 26548, "43115": 42984, "43156": 42178, "43180": 44077, "43193": 16239, "43196": 40835, "43338": 37916, "43358": 11258, "43365": 20712, "43368": 26841, "43384": 42701, "43413": 41925, "43426": 44235, "43431": 43786, "43476": 15139, "43488": 44055, "43510": 38006, "43563": 43127, "43624": 38081, "43654": 37797, "43656": 46733, "43728": 42863, "43752": 44499, "43778": 42393, "43797": 42419, "43810": 7491, "43824": 42453, "43861": 11342, "43872": 43001, "43875": 13004, "43885": 34802, "43891": 31722, "43939": 40082, "43957": 31616, "43993": 16041, "44008": 37636, "44020": 44453, "44062": 36504, "44083": 15421, "44118": 44061, "44125": 43272, "44138": 33206, "44150": 44944, "44206": 38467, "44222": 43880, "44224": 44859, "44283": 43608, "44289": 16904, "44297": 39650, "44304": 38555, "44308": 6426, "44322": 18032, "44394": 45384, "44396": 30634, "44410": 45908, "44420": 38711, "44437": 6273, "44446": 43360, "44458": 26810, "44470": 38756, "44476": 18977, "44477": 45200, "44573": 44001, "44576": 38852, "44582": 44084, "44592": 13475, "44603": 12560, "44618": 39919, "44620": 41577, "44626": 38900, "44645": 38918, "44658": 38929, "44665": 42035, "44713": 44902, "44736": 11144, "44740": 44451, "44751": 39333, "44755": 28876, "44756": 39468, "44771": 23057, "44775": 43093, "44778": 39052, "44790": 38510, "44794": 39067, "44814": 43153, "44817": 39092, "44850": 12299, "44852": 39133, "44854": 39136, "44897": 38609, "44910": 39179, "44914": 39194, "44916": 39195, "44921": 39201, "44943": 45066, "44970": 45389, "44993": 44099, "44995": 39282, "45021": 39314, "45022": 43023, "45027": 39325, "45031": 13557, "45047": 39346, "45052": 13231, "45063": 39368, "45065": 39372, "45079": 28167, "45103": 46203, "45129": 24449, "45133": 11894, "45154": 45159, "45157": 42225, "45161": 6157, "45167": 46666, "45177": 44542, "45189": 42424, "45225": 39641, "45242": 43637, "45256": 43481, "45288": 41237, "45314": 47568, "45320": 13044, "45326": 29800, "45334": 34416, "45345": 27728, "45359": 45234, "45370": 15284, "45373": 37540, "45400": 42772, "45402": 45319, "45431": 45869, "45441": 45878, "45470": 48136, "45479": 18231, "45489": 45608, "45490": 41573, "45497": 45330, "45532": 45619, "45540": 45954, "45551": 2909, "45552": 21937, "45560": 44752, "45565": 22722, "45591": 30354, "45600": 13987, "45617": 36406, "45620": 30760, "45623": 2810, "45640": 42787, "45641": 12022, "45642": 45102, "45656": 45390, "45679": 47121, "45696": 5845, "45709": 45987, "45719": 45956, "45725": 30240, "45742": 38835, "45778": 35490, "45783": 47097, "45786": 43176, "45792": 46152, "45858": 45442, "45864": 32168, "45871": 47116, "45894": 23015, "45920": 30700, "45947": 42488, "45971": 46133, "45975": 9934, "45977": 42970, "45988": 45973, "46022": 42879, "46048": 38859, "46054": 3187, "46055": 42706, "46058": 44726, "46122": 46261, "46124": 30276, "46132": 41152, "46139": 39627, "46166": 4042, "46168": 46965, "46226": 44754, "46260": 7135, "46265": 47293, "46289": 4112, "46299": 20288, "46313": 36762, "46314": 41065, "46337": 576, "46351": 27369, "46384": 44624, "46386": 46628, "46394": 44314, "46400": 43492, "46407": 34815, "46408": 42656, "46413": 30223, "46442": 34447, "46447": 29857, "46450": 44126, "46453": 8573, "46479": 27403, "46488": 43508, "46500": 44177, "46509": 43114, "46530": 40290, "46556": 43349, "46566": 43388, "46585": 38959, "46615": 43572, "46636": 43313, "46674": 28194, "46694": 36427, "46713": 38769, "46740": 2318, "46745": 44169, "46758": 41763, "46761": 35884, "46786": 47617, "46799": 44367, "46805": 43179, "46833": 45260, "46838": 27909, "46853": 30341, "46866": 20453, "46867": 48454, "46881": 39836, "46899": 41581, "46901": 23851, "46924": 30413, "46929": 43311, "46935": 32470, "46942": 47000, "46968": 8928, "46980": 34206, "47004": 39955, "47005": 46776, "47049": 42546, "47053": 17957, "47069": 26336, "47080": 27652, "47094": 31916, "47102": 7891, "47103": 34330, "47114": 17278, "47119": 45237, "47126": 47402, "47142": 46742, "47148": 28918, "47154": 47179, "47169": 47941, "47177": 30828, "47216": 44128, "47217": 3404, "47220": 47765, "47236": 29494, "47246": 47876, "47258": 46958, "47263": 6091, "47316": 29771, "47321": 46478, "47333": 44792, "47336": 46956, "47351": 45131, "47354": 37042, "47359": 45246, "47360": 45521, "47369": 24263, "47370": 45983, "47372": 45690, "47384": 44960, "47387": 46067, "47398": 36148, "47400": 47184, "47405": 42291, "47420": 46382, "47425": 12852, "47434": 45213, "47443": 42183, "47457": 43323, "47458": 31715, "47463": 47377, "47464": 44294, "47466": 31935, "47475": 16935, "47476": 34942, "47478": 44013, "47483": 14291, "47507": 44973, "47528": 41263, "47529": 31256, "47538": 46562, "47547": 47491, "47550": 40727, "47571": 47012, "47572": 43959, "47574": 47019, "47575": 30591, "47586": 45918, "47597": 18489, "47605": 41778, "47618": 41900, "47637": 41001, "47642": 6413, "47643": 43452, "47645": 29794, "47659": 46665, "47665": 5645, "47674": 46013, "47676": 26907, "47743": 46842, "47746": 46326, "47754": 47794, "47756": 46378, "47766": 29437, "47795": 32238, "47796": 29725, "47804": 47526, "47808": 31769, "47815": 28562, "47836": 46917, "47842": 24492, "47849": 46926, "47852": 46933, "47861": 48046, "47862": 40680, "47870": 46890, "47877": 20911, "47897": 45134, "47908": 41173, "47917": 31289, "47919": 47013, "47920": 46686, "47937": 47521, "47940": 47196, "47943": 41270, "47949": 41452, "47951": 8967, "47964": 45281, "47966": 47064, "47967": 41514, "47979": 41714, "47983": 47087, "48003": 41755, "48006": 46700, "48022": 47718, "48032": 47242, "48052": 29767, "48059": 42317, "48078": 41691, "48079": 42388, "48085": 42446, "48086": 13432, "48092": 3452, "48117": 47229, "48132": 26197, "48139": 42156, "48146": 44351, "48156": 29969, "48162": 46033, "48165": 46020, "48194": 7258, "48204": 47365, "48210": 43421, "48213": 45513, "48215": 43552, "48223": 47128, "48225": 44871, "48227": 43539, "48231": 30882, "48238": 36150, "48246": 47902, "48249": 45008, "48251": 45981, "48271": 43858, "48272": 47393, "48274": 36194, "48275": 47397, "48277": 45621, "48289": 20241, "48294": 44123, "48301": 28299, "48308": 46365, "48312": 45243, "48318": 9589, "48323": 46070, "48326": 47792, "48340": 46527, "48344": 17684, "48351": 44393, "48361": 26531, "48363": 48569, "48368": 29844, "48386": 48484, "48389": 7938, "48402": 22220, "48453": 44826, "48458": 16490, "48461": 47035, "48508": 47687, "48514": 47691, "48519": 47841, "48528": 48126, "48533": 46493, "48554": 28787, "48556": 32587, "48565": 28490, "48571": 47212, "48572": 47805, "48581": 17107, "48583": 46550, "48592": 30970, "48614": 30484, "48620": 27726, "48621": 47604, "48641": 46746, "48643": 48075, "48644": 32531, "48652": 30351, "48682": 46041, "48689": 46851, "48715": 49040, "48719": 48128, "48734": 4970, "48736": 48189, "48746": 48749, "48750": 32491, "48764": 47822, "48766": 48802, "48769": 43149, "48772": 46959, "48778": 47357, "48781": 47651, "48790": 47283, "48794": 46949, "48806": 47403, "48844": 27724, "48866": 48031, "48868": 47386, "48869": 48853, "48873": 28361, "48874": 35249, "48886": 48619, "48893": 47409, "48894": 31689, "48895": 48922, "48896": 44060, "48900": 47622, "48907": 34466, "48916": 32067, "48933": 48096, "48943": 47811, "48950": 47779, "48961": 48355, "48965": 47889, "48972": 43746, "48973": 47948, "48974": 48561, "48978": 16513, "48979": 30933, "48988": 48827, "48990": 48610, "48991": 48888, "48999": 49028, "49002": 36370, "49009": 48197, "49011": 27128, "49013": 36783, "49014": 48742, "49015": 48858, "49030": 49034, "49032": 31770, "49037": 48420, "49043": 48683, "49045": 48713, "49057": 32519, "49063": 48499, "49064": 26068, "49075": 48958, "49078": 48600, "49082": 30648, "49085": 48630, "49086": 36908, "49090": 43625, "49091": 29970, "49094": 29467, "49095": 48707, "49096": 49007, "49102": 48745, "49103": 48774, "49104": 47136, "49107": 48930, "49108": 48335, "49109": 31936, "49114": 48822, "49115": 48902, "49119": 48927, "49132": 48955, "49137": 49138, "49143": 31409, "49145": 32589, "49146": 30962, "49151": 49148}
+{"832": 289, "881": 1506, "1036": 363, "1429": 33297, "1463": 1263, "1835": 1426, "2001": 1426, "2103": 1609, "2126": 555, "2926": 393, "3072": 868, "3299": 2542, "3572": 2519, "3775": 555, "3945": 555, "4079": 2825, "4084": 1263, "4228": 6548, "4579": 2256, "4650": 1023, "4833": 3773, "4881": 3545, "5203": 5107, "5572": 44982, "5642": 4759, "5993": 4827, "6111": 2825, "6129": 621, "6392": 1721, "6718": 1263, "6761": 4759, "7535": 345, "7740": 2209, "8007": 3267, "8130": 4589, "8316": 5639, "8490": 2196, "8643": 7440, "8727": 3062, "8837": 6106, "8904": 4166, "8907": 8649, "9106": 594, "9177": 7272, "9399": 7953, "9456": 5056, "9506": 5142, "9513": 1126, "9525": 1387, "9666": 6303, "9820": 8649, "9852": 39288, "9897": 1677, "10077": 2280, "10214": 7344, "10311": 8043, "10437": 8649, "10438": 5422, "10531": 10365, "10632": 1988, "10675": 723, "10882": 8902, "10940": 5127, "11117": 2125, "11510": 2101, "11586": 9477, "11622": 2825, "11886": 4759, "12160": 4985, "12482": 6824, "12534": 17371, "12694": 8921, "12725": 10843, "12849": 651, "12901": 9597, "13198": 21886, "13244": 10743, "13257": 8953, "13323": 11137, "13366": 10345, "13641": 8222, "13814": 39948, "13916": 16372, "14260": 10827, "14270": 8776, "14656": 4589, "14664": 9850, "14914": 12898, "14964": 6070, "15004": 4589, "15014": 9798, "15060": 10822, "15144": 10918, "15211": 34373, "15228": 6475, "15272": 1118, "15437": 4892, "15737": 8845, "15845": 2301, "15916": 3491, "16528": 2892, "16556": 5205, "16568": 4654, "16615": 36688, "16691": 8902, "16822": 2652, "16947": 2136, "16996": 16853, "17000": 9477, "17114": 21956, "17132": 3092, "17272": 28677, "17370": 13215, "17468": 1337, "17845": 9676, "17940": 6741, "18076": 16808, "18086": 8467, "18088": 19750, "18413": 1424, "18527": 16040, "18773": 7390, "18943": 30737, "19222": 30278, "19347": 8649, "19445": 4433, "19476": 3090, "19660": 3414, "19704": 40409, "19709": 9496, "19741": 18774, "19822": 2280, "19846": 2542, "19874": 17030, "20108": 4088, "20192": 2984, "20255": 27809, "20323": 44770, "20398": 21818, "20605": 40483, "20785": 1220, "20830": 8902, "20941": 6817, "20949": 14134, "21044": 26435, "21063": 30471, "21140": 823, "21239": 33325, "21258": 27916, "21329": 39077, "21629": 5107, "21957": 14069, "21973": 8801, "22111": 1598, "22142": 6088, "22170": 18980, "22385": 1289, "22423": 6082, "22492": 26617, "22514": 30401, "22536": 4843, "22574": 18294, "22613": 5779, "22665": 14711, "23234": 1609, "23323": 13340, "23426": 10758, "23533": 11365, "23548": 12568, "23549": 11713, "23650": 19403, "23663": 41466, "24170": 28129, "24282": 23391, "24602": 15993, "24640": 46770, "24887": 35174, "24899": 15835, "24930": 40274, "24948": 30851, "25199": 21022, "25407": 15089, "25716": 3319, "25778": 5646, "25831": 4760, "25898": 4343, "26032": 26608, "26088": 1497, "26094": 13712, "26276": 20963, "26307": 32343, "26327": 26501, "26378": 18191, "26409": 34868, "26436": 30707, "26437": 5127, "26472": 36986, "26476": 35530, "26480": 22766, "26583": 34235, "26607": 31196, "26763": 28510, "26815": 44765, "26839": 28517, "26864": 22675, "26881": 16618, "26883": 33766, "27013": 924, "27095": 26699, "27099": 28756, "27113": 19458, "27190": 31229, "27229": 27645, "27239": 26725, "27266": 28245, "27300": 3773, "27320": 13328, "27334": 37430, "27454": 6765, "27476": 26546, "27541": 7272, "27549": 1690, "27576": 32007, "27618": 20757, "27635": 39223, "27661": 31336, "27676": 29681, "27800": 7218, "27840": 15566, "27860": 30841, "27922": 29112, "27960": 2825, "27962": 28044, "28051": 31825, "28109": 7385, "28115": 18312, "28122": 12291, "28204": 31994, "28212": 3649, "28229": 47301, "28315": 47294, "28355": 14084, "28401": 9841, "28485": 26612, "28541": 32483, "28542": 11654, "28546": 28656, "28605": 4752, "28671": 25893, "28701": 18275, "28771": 35975, "28831": 42101, "28874": 43200, "28887": 43678, "28896": 3319, "28945": 20650, "28961": 28334, "29091": 27624, "29119": 33838, "29127": 28609, "29203": 6895, "29211": 23255, "29234": 45482, "29280": 38979, "29411": 34492, "29458": 29374, "29480": 1500, "29497": 31604, "29502": 30785, "29554": 22049, "29624": 30190, "29656": 41605, "29669": 31933, "29773": 26758, "29801": 29000, "29818": 41590, "29819": 44200, "29824": 47288, "29833": 2513, "29836": 37044, "29863": 30210, "29884": 24569, "29929": 34866, "29960": 31814, "29983": 30176, "30005": 17423, "30013": 43960, "30041": 29876, "30069": 6760, "30076": 42109, "30104": 21523, "30119": 28185, "30214": 48562, "30217": 41570, "30230": 47472, "30250": 26742, "30251": 25272, "30254": 34830, "30259": 4462, "30316": 48439, "30349": 28234, "30352": 29345, "30389": 3184, "30456": 18609, "30524": 46970, "30535": 40759, "30553": 47146, "30556": 31940, "30571": 48570, "30587": 2849, "30610": 27104, "30614": 14803, "30647": 26732, "30665": 16909, "30713": 36230, "30721": 44469, "30752": 30751, "30795": 31535, "30798": 34177, "30891": 49024, "30909": 15909, "30930": 46252, "30950": 48176, "30983": 27379, "31032": 32952, "31034": 48782, "31045": 25385, "31050": 19161, "31126": 32300, "31164": 28516, "31247": 41509, "31281": 17030, "31365": 32444, "31406": 41095, "31407": 16333, "31470": 5271, "31477": 6751, "31488": 29107, "31493": 37962, "31508": 38510, "31585": 43086, "31644": 46055, "31696": 46349, "31715": 6952, "31724": 22877, "31742": 39476, "31756": 26531, "31778": 30976, "31837": 22590, "31850": 17030, "31853": 4984, "31883": 42570, "31891": 47295, "31911": 45002, "31938": 11944, "31947": 33726, "31969": 30270, "31970": 1453, "31985": 46554, "32010": 3146, "32012": 45519, "32026": 35880, "32035": 9328, "32043": 26386, "32053": 32440, "32077": 48059, "32090": 27202, "32119": 47988, "32134": 12416, "32160": 16384, "32173": 49056, "32177": 20022, "32181": 28555, "32200": 45493, "32210": 43153, "32231": 3732, "32291": 43842, "32325": 27528, "32350": 36109, "32388": 47886, "32390": 30679, "32391": 31942, "32398": 34994, "32429": 1410, "32455": 48374, "32457": 41523, "32459": 42837, "32499": 32327, "32503": 46487, "32513": 47774, "32538": 43886, "32550": 31955, "32563": 28253, "32572": 32411, "32573": 48098, "32600": 47933, "32605": 31388, "32617": 2522, "32664": 4698, "32741": 2519, "32762": 2146, "32809": 8651, "32844": 2715, "32878": 17564, "32976": 700, "32980": 1118, "33026": 3052, "33043": 41234, "33044": 4827, "33047": 1326, "33064": 8368, "33100": 3146, "33145": 6504, "33179": 7552, "33181": 1205, "33198": 6926, "33263": 15241, "33279": 10150, "33285": 10855, "33311": 23933, "33331": 1936, "33385": 24588, "33398": 4841, "33440": 16387, "33461": 5863, "33502": 15972, "33563": 5680, "33571": 8877, "33581": 1786, "33628": 20698, "33638": 7283, "33671": 5397, "33720": 1143, "33721": 35852, "33736": 16218, "33744": 2660, "33761": 11329, "33777": 4577, "33803": 9017, "33824": 21460, "33855": 32814, "33957": 3485, "33960": 2623, "33994": 33856, "33998": 16758, "34065": 6054, "34081": 7186, "34089": 18435, "34093": 35533, "34104": 35551, "34123": 3061, "34193": 3744, "34198": 35039, "34218": 8669, "34227": 8575, "34278": 35251, "34292": 33126, "34296": 28870, "34306": 13473, "34316": 9572, "34328": 25016, "34339": 16442, "34340": 6084, "34382": 41915, "34401": 35615, "34403": 23921, "34423": 35046, "34443": 8176, "34468": 36337, "34479": 460, "34495": 10683, "34501": 26596, "34518": 10308, "34531": 14221, "34553": 12358, "34576": 25746, "34577": 36747, "34607": 35111, "34643": 26805, "34662": 29644, "34674": 34437, "34678": 34430, "34768": 25848, "34789": 5107, "34799": 44931, "34816": 12475, "34835": 34671, "34869": 14689, "34908": 25343, "34974": 47495, "35053": 30890, "35069": 37980, "35072": 35030, "35079": 6477, "35130": 36363, "35156": 11198, "35157": 11129, "35214": 15988, "35247": 13114, "35267": 19926, "35300": 30350, "35345": 18157, "35354": 32938, "35400": 14692, "35425": 16087, "35429": 21863, "35438": 36187, "35441": 17949, "35452": 7291, "35457": 34636, "35466": 22359, "35492": 36516, "35493": 25221, "35495": 45612, "35500": 8252, "35506": 30488, "35540": 44393, "35543": 41149, "35561": 5708, "35599": 34082, "35611": 16853, "35658": 46583, "35707": 46609, "35726": 29625, "35735": 36004, "35736": 37493, "35737": 36431, "35751": 32017, "35755": 26448, "35763": 579, "35783": 47308, "35794": 27237, "35805": 43548, "35809": 47868, "35812": 32687, "35844": 15400, "35868": 30126, "35876": 605, "35887": 28006, "35894": 26253, "35897": 47408, "35917": 3773, "35923": 10847, "35964": 41734, "35966": 34556, "35970": 14652, "35998": 42736, "36000": 15771, "36006": 842, "36035": 692, "36042": 36465, "36069": 28749, "36086": 48496, "36098": 14514, "36100": 26717, "36128": 29966, "36132": 17400, "36149": 31290, "36150": 8935, "36154": 8902, "36174": 6303, "36199": 35541, "36205": 38049, "36256": 32301, "36275": 31041, "36296": 7440, "36298": 35243, "36303": 35839, "36309": 47406, "36310": 26449, "36319": 4248, "36343": 14505, "36367": 31952, "36372": 44757, "36381": 48847, "36400": 37748, "36407": 32801, "36424": 37041, "36434": 36294, "36503": 7384, "36510": 19592, "36537": 35050, "36550": 24565, "36584": 17030, "36593": 28453, "36603": 16987, "36621": 47698, "36638": 37030, "36641": 48983, "36650": 27139, "36675": 47894, "36693": 20898, "36701": 26411, "36731": 31820, "36771": 36847, "36842": 46855, "36855": 47125, "36864": 45120, "36867": 7272, "36935": 4864, "36944": 1525, "36947": 46681, "36958": 36063, "36959": 39097, "36965": 9287, "36981": 34456, "36989": 25052, "37007": 37194, "37012": 48167, "37032": 40480, "37038": 28030, "37068": 25272, "37077": 1002, "37170": 41162, "37237": 22560, "37258": 36329, "37296": 8171, "37304": 38232, "37365": 7272, "37407": 478, "37431": 17493, "37472": 2588, "37494": 1578, "37597": 28242, "37623": 6303, "37700": 19592, "37728": 37966, "37749": 1127, "37766": 2058, "37854": 8030, "37910": 2542, "37913": 1466, "37941": 27037, "37947": 27987, "37951": 38201, "37996": 38275, "38005": 35467, "38013": 4658, "38033": 38055, "38066": 1609, "38082": 1832, "38104": 4556, "38105": 4229, "38115": 7187, "38208": 35753, "38214": 27002, "38234": 33617, "38248": 2125, "38258": 26858, "38260": 2680, "38300": 2116, "38301": 11420, "38340": 24915, "38344": 9035, "38354": 8787, "38405": 35390, "38424": 13199, "38426": 38850, "38465": 38757, "38486": 31137, "38495": 3442, "38529": 21883, "38590": 15209, "38613": 24890, "38644": 1720, "38667": 9098, "38756": 1670, "38790": 42668, "38835": 9535, "38841": 1639, "38897": 35443, "38903": 24841, "38930": 35720, "38999": 11355, "39016": 34906, "39019": 9198, "39025": 19152, "39029": 39348, "39031": 3458, "39036": 32982, "39042": 28911, "39056": 9486, "39059": 521, "39064": 11001, "39076": 38827, "39088": 29868, "39121": 26575, "39128": 39451, "39162": 27844, "39213": 20410, "39221": 35872, "39279": 14262, "39403": 26736, "39446": 21781, "39466": 43744, "39473": 25272, "39479": 6619, "39492": 21823, "39510": 2916, "39541": 38024, "39548": 25083, "39552": 41315, "39570": 36758, "39574": 16980, "39588": 4097, "39592": 40392, "39652": 40676, "39658": 17293, "39674": 25052, "39700": 3783, "39733": 35687, "39765": 38399, "39775": 27228, "39785": 7537, "39791": 2875, "39805": 28151, "39811": 15521, "39824": 15930, "39903": 5686, "39920": 40851, "39974": 17498, "39978": 1385, "40004": 7766, "40090": 18165, "40101": 1048, "40104": 41029, "40144": 13443, "40148": 40106, "40155": 14861, "40160": 4501, "40168": 39355, "40170": 36395, "40237": 44448, "40247": 44457, "40277": 14588, "40325": 7002, "40333": 5148, "40377": 39905, "40397": 39268, "40413": 34712, "40415": 43481, "40424": 39291, "40474": 31081, "40508": 4243, "40514": 35606, "40548": 36201, "40550": 7674, "40574": 34879, "40582": 3301, "40586": 12337, "40618": 25167, "40651": 36522, "40677": 38954, "40699": 8328, "40711": 38976, "40712": 32489, "40752": 15319, "40834": 39946, "40892": 1183, "40932": 19853, "41020": 45024, "41031": 41062, "41058": 42186, "41059": 16776, "41063": 36856, "41098": 31046, "41125": 34615, "41150": 39388, "41155": 8923, "41156": 43976, "41163": 42042, "41165": 1715, "41167": 40324, "41198": 39945, "41206": 16059, "41231": 4490, "41233": 12931, "41248": 36134, "41286": 42324, "41291": 31286, "41292": 43927, "41308": 45920, "41312": 42352, "41349": 39961, "41350": 42280, "41369": 40165, "41382": 40178, "41501": 47501, "41504": 35618, "41588": 36397, "41627": 37144, "41681": 27086, "41758": 36031, "41781": 19884, "41792": 1209, "41826": 38461, "41832": 41378, "41840": 15676, "41855": 35823, "41865": 30555, "41866": 15048, "41870": 44339, "41887": 42621, "41904": 27468, "41906": 41458, "41931": 41479, "41972": 43182, "42050": 41110, "42054": 43068, "42081": 34802, "42082": 30912, "42130": 35406, "42139": 43992, "42179": 26585, "42197": 44573, "42213": 42918, "42229": 44512, "42242": 40235, "42250": 23082, "42253": 31910, "42287": 42227, "42319": 30687, "42428": 15883, "42443": 4017, "42448": 42555, "42449": 5083, "42450": 40970, "42455": 40257, "42470": 8070, "42473": 8704, "42503": 13982, "42531": 26529, "42534": 20806, "42559": 42770, "42566": 37517, "42581": 31098, "42605": 32395, "42672": 41672, "42731": 43416, "42792": 45866, "42816": 40685, "42822": 39090, "42868": 31900, "42871": 41414, "42901": 9384, "42956": 23438, "42967": 6695, "42991": 39226, "43003": 42044, "43015": 41540, "43031": 42070, "43073": 43925, "43083": 25153, "43110": 26565, "43125": 42994, "43166": 42188, "43190": 44085, "43203": 16245, "43206": 40848, "43348": 37929, "43368": 11262, "43375": 20718, "43378": 26857, "43394": 42711, "43423": 41936, "43436": 44242, "43441": 43795, "43486": 15146, "43498": 44063, "43520": 38019, "43573": 43137, "43634": 38094, "43664": 37810, "43666": 46736, "43738": 42873, "43787": 42403, "43806": 42429, "43819": 7492, "43833": 42463, "43870": 11346, "43883": 13011, "43893": 34818, "43899": 31736, "43947": 40095, "43965": 31630, "44001": 16047, "44016": 37649, "44028": 44460, "44070": 36518, "44125": 44069, "44132": 43282, "44145": 33222, "44157": 44951, "44213": 38480, "44229": 43888, "44231": 44866, "44290": 43618, "44296": 16910, "44304": 39664, "44311": 38568, "44315": 6427, "44329": 18038, "44401": 45390, "44403": 30649, "44417": 45913, "44427": 38724, "44444": 6274, "44453": 43370, "44465": 26827, "44477": 38769, "44483": 18984, "44484": 45206, "44580": 44009, "44583": 38865, "44589": 44091, "44599": 13482, "44610": 12567, "44625": 39933, "44627": 41589, "44633": 38913, "44652": 38931, "44665": 38942, "44672": 42046, "44720": 44909, "44743": 11148, "44747": 44458, "44758": 39347, "44762": 28893, "44763": 39482, "44778": 23064, "44782": 43103, "44785": 39066, "44797": 38523, "44801": 39081, "44821": 43163, "44824": 39106, "44857": 12304, "44859": 39147, "44861": 39150, "44904": 38622, "44917": 39193, "44921": 39208, "44923": 39209, "44928": 39215, "44950": 45072, "44976": 45395, "44999": 44106, "45001": 39296, "45027": 39328, "45028": 43033, "45033": 39339, "45037": 13564, "45053": 39360, "45058": 13238, "45069": 39382, "45071": 39386, "45085": 28184, "45109": 46207, "45135": 24455, "45139": 11898, "45160": 45165, "45163": 42235, "45167": 6158, "45173": 46669, "45183": 44549, "45195": 42434, "45231": 39655, "45248": 43647, "45262": 43491, "45294": 41249, "45320": 47571, "45326": 13051, "45332": 29816, "45340": 34432, "45351": 27744, "45365": 45240, "45376": 15291, "45379": 37553, "45406": 42782, "45408": 45325, "45437": 45874, "45447": 45883, "45476": 48137, "45485": 18237, "45495": 45613, "45502": 45336, "45537": 45624, "45545": 45959, "45556": 2908, "45557": 21944, "45565": 44759, "45570": 22729, "45596": 30369, "45605": 13994, "45622": 36420, "45625": 30775, "45628": 2809, "45645": 42797, "45646": 12027, "45647": 45108, "45661": 45396, "45684": 47124, "45701": 5844, "45714": 45992, "45724": 45961, "45730": 30255, "45747": 38848, "45783": 35505, "45788": 47100, "45791": 43186, "45797": 46157, "45863": 45448, "45869": 32182, "45876": 47119, "45899": 23022, "45925": 30715, "45952": 42498, "45976": 46138, "45980": 9938, "45982": 42980, "45993": 45978, "46027": 42889, "46053": 38872, "46059": 3186, "46060": 42716, "46063": 44733, "46127": 46264, "46129": 30291, "46137": 41164, "46144": 39641, "46170": 4041, "46172": 46968, "46263": 7136, "46268": 47296, "46292": 4111, "46302": 20294, "46316": 36776, "46317": 41077, "46340": 576, "46354": 27385, "46387": 44631, "46389": 46631, "46397": 44321, "46403": 43502, "46410": 34831, "46411": 42666, "46416": 30238, "46445": 34463, "46450": 29873, "46453": 44133, "46456": 8576, "46482": 27418, "46491": 43518, "46503": 44184, "46512": 43124, "46533": 40303, "46559": 43359, "46569": 43398, "46588": 38972, "46618": 43582, "46639": 43323, "46677": 28211, "46697": 36441, "46716": 38782, "46743": 2317, "46748": 44176, "46761": 41775, "46764": 35898, "46789": 47620, "46802": 44374, "46808": 43189, "46836": 45266, "46841": 27926, "46856": 30356, "46869": 20459, "46870": 48454, "46884": 39850, "46902": 41593, "46904": 23857, "46927": 30428, "46932": 43321, "46938": 32484, "46945": 47003, "46971": 8932, "46983": 34222, "47007": 39969, "47008": 46779, "47052": 42556, "47056": 17963, "47072": 26353, "47083": 27668, "47097": 31930, "47105": 7892, "47106": 34346, "47117": 17284, "47122": 45243, "47129": 47404, "47145": 46745, "47151": 28935, "47157": 47182, "47172": 47942, "47180": 30843, "47219": 44135, "47220": 3403, "47223": 47767, "47239": 29510, "47249": 47877, "47261": 46961, "47266": 6092, "47319": 29787, "47324": 46481, "47336": 44799, "47339": 46959, "47354": 45137, "47357": 37056, "47362": 45252, "47363": 45526, "47371": 24269, "47372": 45988, "47374": 45695, "47386": 44967, "47389": 46072, "47400": 36162, "47402": 47187, "47407": 42301, "47422": 46385, "47427": 12859, "47436": 45219, "47445": 42193, "47459": 43333, "47460": 31729, "47465": 47379, "47466": 44301, "47468": 31949, "47477": 16941, "47478": 34958, "47480": 44021, "47485": 14298, "47509": 44979, "47530": 41275, "47531": 31271, "47541": 46565, "47550": 47493, "47553": 40740, "47574": 47015, "47575": 43967, "47577": 47022, "47578": 30606, "47589": 45923, "47600": 18496, "47608": 41790, "47621": 41911, "47640": 41014, "47645": 6414, "47646": 43462, "47648": 29810, "47662": 46668, "47668": 5644, "47677": 46018, "47679": 26923, "47746": 46845, "47749": 46329, "47756": 47796, "47758": 46381, "47768": 29453, "47797": 32252, "47798": 29741, "47806": 47528, "47810": 31783, "47817": 28579, "47838": 46920, "47844": 24498, "47850": 46929, "47853": 46936, "47862": 48047, "47863": 40693, "47871": 46893, "47878": 20918, "47898": 45140, "47909": 41185, "47918": 31304, "47920": 47016, "47921": 46689, "47938": 47523, "47941": 47199, "47944": 41282, "47950": 41464, "47952": 8971, "47965": 45287, "47967": 47067, "47968": 41526, "47980": 41726, "47984": 47090, "48004": 41767, "48007": 46703, "48023": 47721, "48033": 47245, "48053": 29783, "48060": 42327, "48079": 41703, "48080": 42398, "48086": 42456, "48087": 13439, "48093": 3451, "48118": 47232, "48133": 26204, "48140": 42167, "48147": 44358, "48157": 29984, "48163": 46038, "48166": 46025, "48195": 7259, "48205": 47367, "48211": 43431, "48214": 45518, "48216": 43562, "48224": 47131, "48226": 44878, "48228": 43549, "48232": 30897, "48239": 36164, "48247": 47903, "48250": 45014, "48252": 45986, "48272": 43867, "48273": 47395, "48275": 36208, "48276": 47399, "48278": 45626, "48290": 20248, "48295": 44130, "48302": 28316, "48309": 46368, "48313": 45249, "48319": 9593, "48324": 46075, "48327": 47794, "48341": 46530, "48345": 17690, "48352": 44400, "48362": 26548, "48364": 48569, "48369": 29860, "48387": 48484, "48390": 7939, "48403": 22226, "48453": 44833, "48458": 16496, "48461": 47038, "48508": 47690, "48514": 47694, "48519": 47843, "48528": 48127, "48533": 46496, "48554": 28804, "48556": 32601, "48565": 28507, "48571": 47215, "48572": 47807, "48581": 17113, "48583": 46553, "48592": 30985, "48614": 30499, "48620": 27742, "48621": 47607, "48641": 46749, "48643": 48076, "48644": 32545, "48652": 30366, "48682": 46046, "48689": 46854, "48715": 49040, "48719": 48129, "48734": 4969, "48736": 48190, "48746": 48749, "48750": 32505, "48764": 47824, "48766": 48802, "48769": 43159, "48772": 46962, "48778": 47360, "48781": 47654, "48790": 47286, "48794": 46952, "48806": 47405, "48844": 27740, "48866": 48032, "48868": 47388, "48869": 48853, "48873": 28378, "48874": 35264, "48886": 48619, "48893": 47411, "48894": 31703, "48895": 48922, "48896": 44068, "48900": 47625, "48907": 34482, "48916": 32081, "48933": 48097, "48943": 47813, "48950": 47781, "48961": 48356, "48965": 47890, "48972": 43756, "48973": 47949, "48974": 48561, "48978": 16519, "48979": 30948, "48988": 48827, "48990": 48610, "48991": 48888, "48999": 49028, "49002": 36384, "49009": 48198, "49011": 27144, "49013": 36797, "49014": 48742, "49015": 48858, "49030": 49034, "49032": 31784, "49037": 48421, "49043": 48683, "49045": 48713, "49057": 32533, "49063": 48499, "49064": 26075, "49075": 48958, "49078": 48600, "49082": 30663, "49085": 48630, "49086": 36922, "49090": 43635, "49091": 29985, "49094": 29483, "49095": 48707, "49096": 49007, "49102": 48745, "49103": 48774, "49104": 47139, "49107": 48930, "49108": 48336, "49109": 31950, "49114": 48822, "49115": 48902, "49119": 48927, "49132": 48955, "49137": 49138, "49143": 31423, "49145": 32603, "49146": 30977, "49151": 49148}

tokenizer.json

@@ -4815,10 +4815,6 @@
         "El",
         -9.42580795288086
       ],
-      [
-        "tt",
-        -9.42664909362793
-      ],
       [
         "son",
         -9.426764488220217
@@ -24003,6 +23999,10 @@
         "▁mögé",
         -11.24395751953125
       ],
+      [
+        "▁felvett",
+        -11.244065284729004
+      ],
       [
         "▁harag",
         -11.244222640991213
@@ -24215,6 +24215,10 @@
         "ille",
         -11.251391410827637
       ],
+      [
+        "▁művei",
+        -11.251520156860352
+      ],
       [
         "ern",
         -11.251664161682127
@@ -34187,6 +34191,10 @@
         "▁biztosíték",
         -11.594255447387695
       ],
+      [
+        "gott",
+        -11.59428596496582
+      ],
       [
         "▁matematika",
         -11.594395637512209
@@ -34571,6 +34579,10 @@
         "▁equations",
         -11.60633659362793
       ],
+      [
+        "vai",
+        -11.606507301330566
+      ],
       [
         "▁báb",
         -11.606508255004885
@@ -35471,6 +35483,10 @@
         "▁lapos",
         -11.629714012145996
       ],
+      [
+        "▁falvak",
+        -11.629815101623535
+      ],
       [
         "▁stílusú",
         -11.62983512878418
@@ -43291,10 +43307,6 @@
         "▁kór",
         -11.832013130187988
       ],
-      [
-        "▁Sokan",
-        -11.832051277160645
-      ],
       [
         "▁bánat",
         -11.832143783569336
@@ -44575,6 +44587,10 @@
         "ablak",
         -11.864163398742676
       ],
+      [
+        "▁felvette",
+        -11.864184379577637
+      ],
       [
         "papír",
         -11.86447811126709
@@ -48139,6 +48155,10 @@
         "▁merész",
         -11.943978309631348
       ],
+      [
+        "▁kivett",
+        -11.944040298461914
+      ],
       [
         "segít",
         -11.944091796875
@@ -49899,6 +49919,10 @@
         "▁füzet",
         -11.98135471343994
       ],
+      [
+        "vát",
+        -11.981390953063965
+      ],
       [
         "Cache",
         -11.981419563293455
@@ -49991,6 +50015,10 @@
         "▁keresztyén",
         -11.983893394470217
       ],
+      [
+        "▁művet",
+        -11.983941078186035
+      ],
       [
         "dj",
         -11.983988761901855
@@ -63099,10 +63127,6 @@
         "▁Sto",
         -12.2269868850708
       ],
-      [
-        "▁felvet",
-        -12.226998329162598
-      ],
       [
         "▁Újabb",
         -12.227142333984377
@@ -73639,6 +73663,10 @@
         "▁problémákkal",
         -12.390637397766112
       ],
+      [
+        "▁elvitte",
+        -12.39071273803711
+      ],
       [
         "▁Közlönyben",
         -12.390740394592283
@@ -81451,10 +81479,6 @@
         "▁kiadásának",
         -12.500760078430176
       ],
-      [
-        "▁kivet",
-        -12.500771522521973
-      ],
       [
         "tornyot",
         -12.500781059265137
@@ -83303,6 +83327,10 @@
         "▁pajtás",
         -12.52593994140625
       ],
+      [
+        "▁mosolygott",
+        -12.525944709777832
+      ],
       [
         "▁ph",
         -12.526006698608398
@@ -88451,10 +88479,6 @@
         "▁mosó",
         -12.59128189086914
       ],
-      [
-        "tta",
-        -12.591323852539062
-      ],
       [
         "állású",
         -12.591341018676758
@@ -90623,6 +90647,10 @@
         "▁Rezső",
         -12.619868278503418
       ],
+      [
+        "▁felvetette",
+        -12.619959831237791
+      ],
       [
         "▁session",
         -12.619983673095703
@@ -92951,10 +92979,6 @@
         "dátum",
         -12.648262977600098
       ],
-      [
-        "vei",
-        -12.648449420928957
-      ],
       [
         "▁bájos",
         -12.648475646972656
@@ -100511,10 +100535,6 @@
         "juttat",
         -12.739002227783203
       ],
-      [
-        "▁levet",
-        -12.739108085632324
-      ],
       [
         "ettél",
         -12.739171981811523
@@ -102795,6 +102815,10 @@
         "▁fejlődéséhez",
         -12.766057014465332
       ],
+      [
+        "▁falvakban",
+        -12.766210556030272
+      ],
       [
         "▁nad",
         -12.76621913909912
@@ -104299,6 +104323,10 @@
         "átalány",
         -12.784199714660645
       ],
+      [
+        "▁összevetve",
+        -12.784232139587402
+      ],
       [
         "geo",
         -12.78428840637207
@@ -105435,6 +105463,18 @@
         "öblö",
         -12.796637535095217
       ],
+      [
+        "▁megéri",
+        -12.796734809875488
+      ],
+      [
+        "▁looks",
+        -12.796748161315918
+      ],
+      [
+        "▁Zsófi",
+        -12.79676342010498
+      ],
       [
         "▁polgárság",
         -12.796771049499512
@@ -105443,14 +105483,42 @@
         "▁deficit",
         -12.796833038330078
       ],
+      [
+        "ROS",
+        -12.796947479248049
+      ],
+      [
+        "road",
+        -12.796981811523438
+      ],
+      [
+        "panel",
+        -12.797025680541992
+      ],
+      [
+        "hivatkoz",
+        -12.797054290771484
+      ],
+      [
+        "▁fordítását",
+        -12.797069549560549
+      ],
       [
         "▁pleisztocén",
         -12.797074317932127
       ],
+      [
+        "▁experiment",
+        -12.797118186950684
+      ],
       [
         "▁helyesbít",
         -12.797161102294922
       ],
+      [
+        "röhög",
+        -12.797201156616213
+      ],
       [
         "▁ler",
         -12.797283411026
@@ -107579,10 +107647,6 @@
         "▁sch",
         -12.899477243423462
       ],
-      [
-        "Hercegovina",
-        -12.899508476257324
-      ],
       [
         "▁kegyelet",
         -12.899538040161133
@@ -109915,10 +109979,6 @@
         "▁élvező",
         -12.992772102355955
       ],
-      [
-        "▁veszte",
-        -12.992941856384276
-      ],
       [
         "▁kvalitatív",
         -12.993864059448242
@@ -110439,6 +110499,10 @@
         "▁hárs",
         -13.013855934143066
       ],
+      [
+        "Emel",
+        -13.01386260986328
+      ],
       [
         "▁foszt",
         -13.013909339904783
@@ -111795,6 +111859,10 @@
         "▁közalapítvány",
         -13.075096130371094
       ],
+      [
+        "doktori",
+        -13.075101852416992
+      ],
       [
         "▁stimmel",
         -13.075130462646484
@@ -117179,10 +117247,6 @@
         "▁paradox",
         -13.302289009094238
       ],
-      [
-        "ttek",
-        -13.30243682861328
-      ],
       [
         "▁önrendelkezés",
         -13.303096771240234
@@ -119775,10 +119839,6 @@
         "▁önköltség",
         -13.416337013244627
       ],
-      [
-        "Mell",
-        -13.416596412658691
-      ],
       [
         "▁albizottság",
         -13.4166841506958
@@ -125859,10 +125919,6 @@
         "Shaw",
         -13.706841468811035
       ],
-      [
-        "tték",
-        -13.70687198638916
-      ],
       [
         "▁kormánykoalíció",
         -13.706917762756348
@@ -128235,6 +128291,10 @@
         "▁mikroprocesszor",
         -13.830272674560549
       ],
+      [
+        "▁aknavető",
+        -13.830279350280762
+      ],
       [
         "▁gyógyíthatatlan",
         -13.830327987670898
@@ -128447,10 +128507,6 @@
         "▁harcjármű",
         -13.841219902038574
       ],
-      [
-        "ttem",
-        -13.841318130493164
-      ],
       [
         "▁lott",
         -13.841564416885376
@@ -131383,6 +131439,10 @@
         "▁pon",
         -14.188208818435667
       ],
+      [
+        "▁gott",
+        -14.190008401870728
+      ],
       [
         "▁п",
         -14.190243005752563
@@ -132623,6 +132683,10 @@
         "▁Policy",
         -14.479191064834597
       ],
+      [
+        "▁Vet",
+        -14.480298042297363
+      ],
       [
         "▁metszés",
         -14.48103165626526
@@ -140735,10 +140799,6 @@
         "▁packet",
         -15.4236261844635
       ],
-      [
-        "▁Found",
-        -15.424251794815063
-      ],
       [
         "▁Lám",
         -15.42477250099182
@@ -143171,10 +143231,6 @@
         "▁kedvű",
         -15.632192850112917
       ],
-      [
-        "vük",
-        -15.632896900177002
-      ],
       [
         "▁uszodá",
         -15.63322377204895
@@ -148599,10 +148655,6 @@
         "▁légies",
         -16.631510257720947
       ],
-      [
-        "vön",
-        -16.632763385772705
-      ],
       [
         "▁apród",
         -16.63312005996704
@@ -156399,6 +156451,10 @@
         "▁elnöklet",
         -18.661882400512695
       ],
+      [
+        "▁foglalkozn",
+        -18.662598609924316
+      ],
       [
         "▁titkosság",
         -18.664186477661133
@@ -160283,10 +160339,6 @@
         "……",
         -19.37184715270996
       ],
-      [
-        "▁szórakozta",
-        -19.371954917907715
-      ],
       [
         "▁őszül",
         -19.372026443481445
@@ -164375,10 +164427,6 @@
         "▁becsületesség",
         -19.964774131774902
       ],
-      [
-        "▁foglalkozta",
-        -19.9653377532959
-      ],
       [
         "▁szende",
         -19.96594524383545
@@ -167811,10 +167859,6 @@
         "▁vértanúság",
         -20.43210220336914
       ],
-      [
-        "▁öltözte",
-        -20.43240451812744
-      ],
       [
         "▁beltér",
         -20.43252944946289
@@ -169011,10 +169055,6 @@
         "▁kisbetűs",
         -20.602667808532715
       ],
-      [
-        "▁párosodás",
-        -20.60340118408203
-      ],
       [
         "▁etnikumú",
         -20.603835105895996
@@ -175335,10 +175375,6 @@
         "▁Helyzet",
         -21.490195274353027
       ],
-      [
-        "▁kamatozta",
-        -21.49085807800293
-      ],
       [
         "▁modernizál",
         -21.490891456604004
@@ -175815,10 +175851,6 @@
         "▁szezonzáró",
         -21.556296348571777
       ],
-      [
-        "▁káprázta",
-        -21.55659770965576
-      ],
       [
         "▁nyelvtanuló",
         -21.557196617126465
@@ -176659,10 +176691,6 @@
         "▁táncszínház",
         -21.6918363571167
       ],
-      [
-        "▁nyilatkozn",
-        -21.69204354286194
-      ],
       [
         "▁ötletgazda",
         -21.692448616027832
@@ -180171,10 +180199,6 @@
         "▁lévita",
         -22.398947715759277
       ],
-      [
-        "▁emlékkö",
-        -22.3998384475708
-      ],
       [
         "▁gázelosztó",
         -22.40288543701172
@@ -182287,10 +182311,6 @@
         "▁nyugodalm",
         -22.94628143310547
       ],
-      [
-        "▁mérkőzhe",
-        -22.947246551513672
-      ],
       [
         "▁sárgásfehér",
         -22.94734477996826
@@ -184979,10 +184999,6 @@
         "▁zárvatermő",
         -23.895301818847656
       ],
-      [
-        "▁közjav",
-        -23.89534568786621
-      ],
       [
         "▁dumping",
         -23.896888494491577
@@ -185231,10 +185247,6 @@
         "▁klikk",
         -24.020570278167725
       ],
-      [
-        "▁nanocsö",
-        -24.021580696105957
-      ],
       [
         "▁villog",
         -24.02237629890442
@@ -189783,10 +189795,6 @@
         "▁nagyérdemű",
         -27.71691608428955
       ],
-      [
-        "▁számkivet",
-        -27.72173118591309
-      ],
       [
         "▁szemantika",
         -27.72935438156128
@@ -190479,6 +190487,10 @@
         "▁vetkőz",
         -28.36702013015747
       ],
+      [
+        "▁pipetta",
+        -28.368157625198364
+      ],
       [
         "▁brácsá",
         -28.372023582458496
@@ -191331,10 +191343,6 @@
         "▁ágostai",
         -29.238428592681885
       ],
-      [
-        "▁támolyg",
-        -29.2506000995636
-      ],
       [
         "▁pubertás",
         -29.259851455688477
@@ -191711,10 +191719,6 @@
         "▁csurgó",
         -29.53585386276245
       ],
-      [
-        "▁vánszorg",
-        -29.536086320877075
-      ],
       [
         "▁hipermarket",
         -29.539283752441406
@@ -194107,10 +194111,6 @@
         "▁hajmeresztő",
         -32.16242980957031
       ],
-      [
-        "▁ikonosztáz",
-        -32.17224884033203
-      ],
       [
         "▁garmadá",
         -32.17611312866211

tokenizer.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:885d4e1f8a131408d2fe71d73a50bb260b9c873ead6254e238609dd2996f756c
-size 1032598
+oid sha256:6c8bea82e48ee33b2ffcf07c319d4bfdfd2e3e20b82bcc9a5ca3e7773e52257f
+size 1032588