cff-version: 1.2.0
message: "If you use TMP, please cite it as below."
type: software
title: "TMP — Tokenizer Morphological Profile"
abstract: >-
  An open diagnostic tool for measuring where a tokenizer places its
  boundaries relative to morpheme boundaries in an inflectional language.
  Three metrics are read together as a profile: SI (Stem Integrity), the
  percentage of forms in which no token boundary falls inside the stem;
  ISS (Inflectional Suffix Separation), the percentage of forms with an
  overt ending in which a boundary falls exactly at the stem/ending
  boundary; and MFL (Mean Fragmentation per Lexeme), the mean number of
  tokens per form aggregated per lexeme. Boundaries are read from
  character offsets supplied by fast tokenizers, which makes the metrics
  robust to byte-level encoding of diacritics. Polish is the validation
  material; the metrics are not specific to Polish. The repository
  includes a hand-annotated dataset of 10 Polish lexemes in full
  inflectional paradigms (121 paradigm cells, 109 unique surface forms),
  the reference implementation, and profiles for four tokenizers
  (HerBERT, XLM-R, mBERT, GPT-2).
version: 0.2.0
date-released: 2026-07-13
authors:
  - family-names: Dawidek
    given-names: Elżbieta
    orcid: "https://orcid.org/0009-0000-0433-6095"
    affiliation: "Uniwersytet DSW Ideis, Wrocław, Poland"
    email: plgram.benchmarks@proton.me
repository-code: "https://github.com/lizzy-606/pl-tmp"
license:
  - Apache-2.0
  - CC-BY-4.0
license-url: "https://github.com/lizzy-606/pl-tmp/blob/main/LICENSE"
keywords:
  - tokenization
  - subword segmentation
  - morphology
  - inflectional languages
  - morphologically rich languages
  - Polish
  - language model evaluation
  - diagnostic metrics
references:
  - type: article
    title: >-
      Inflectional Paradigms as a Diagnostic Tool for Tokenizers in
      Morphologically Rich Languages: A Proposal for a Linguistic Benchmark
    authors:
      - family-names: Dawidek
        given-names: Elżbieta
        orcid: "https://orcid.org/0009-0000-0433-6095"
    year: 2026
    doi: 10.31235/osf.io/tqvuf_v1
    url: "https://doi.org/10.31235/osf.io/tqvuf_v1"
    notes: >-
      Proposes the metrics. The SI implementation reported in that
      version is superseded by the corrected implementation in this
      repository; see CHANGELOG.md. MFL is unaffected.