File size: 2,392 Bytes
d4d6921 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | cff-version: 1.2.0
message: "If you use TMP, please cite it as below."
type: software
title: "TMP — Tokenizer Morphological Profile"
abstract: >-
An open diagnostic tool for measuring where a tokenizer places its
boundaries relative to morpheme boundaries in an inflectional language.
Three metrics are read together as a profile: SI (Stem Integrity), the
percentage of forms in which no token boundary falls inside the stem;
ISS (Inflectional Suffix Separation), the percentage of forms with an
overt ending in which a boundary falls exactly at the stem/ending
boundary; and MFL (Mean Fragmentation per Lexeme), the mean number of
tokens per form aggregated per lexeme. Boundaries are read from
character offsets supplied by fast tokenizers, which makes the metrics
robust to byte-level encoding of diacritics. Polish is the validation
material; the metrics are not specific to Polish. The repository
includes a hand-annotated dataset of 10 Polish lexemes in full
inflectional paradigms (121 paradigm cells, 109 unique surface forms),
the reference implementation, and profiles for four tokenizers
(HerBERT, XLM-R, mBERT, GPT-2).
version: 0.2.0
date-released: 2026-07-13
authors:
- family-names: Dawidek
given-names: Elżbieta
orcid: "https://orcid.org/0009-0000-0433-6095"
affiliation: "Uniwersytet DSW Ideis, Wrocław, Poland"
email: plgram.benchmarks@proton.me
repository-code: "https://github.com/lizzy-606/pl-tmp"
license:
- Apache-2.0
- CC-BY-4.0
license-url: "https://github.com/lizzy-606/pl-tmp/blob/main/LICENSE"
keywords:
- tokenization
- subword segmentation
- morphology
- inflectional languages
- morphologically rich languages
- Polish
- language model evaluation
- diagnostic metrics
references:
- type: article
title: >-
Inflectional Paradigms as a Diagnostic Tool for Tokenizers in
Morphologically Rich Languages: A Proposal for a Linguistic Benchmark
authors:
- family-names: Dawidek
given-names: Elżbieta
orcid: "https://orcid.org/0009-0000-0433-6095"
year: 2026
doi: 10.31235/osf.io/tqvuf_v1
url: "https://doi.org/10.31235/osf.io/tqvuf_v1"
notes: >-
Proposes the metrics. The SI implementation reported in that
version is superseded by the corrected implementation in this
repository; see CHANGELOG.md. MFL is unaffected.
|