alexbalandi/litlat-bert-lithuanian-morphology-vdu

Lithuanian morphology tagger — accentuation flavor

Token-classification tagger built as the disambiguation engine for Lithuanian accentuation (kirčiavimas) pipelines, notably Kirčiuoklė. It emits UPOS + agreement features in the label space of Lithuanian traditional grammar — the space used by accentuation dictionaries (VDU convention): DET is folded into PRON and AUX into VERB (Lithuanian grammar has neither category; determiners are įvardžiai, auxiliaries are ordinary veiksmažodžiai). If you want strict UD categories, use the -morphology or -morphology-full siblings instead.

Benchmarks (measured)

End-to-end accentuation quality — agreement of homograph stress choices with the production VDU+UDPipe pipeline (each ambiguous word, e.g. galvõs vs gálvos, resolved by this model's morphology):

eval	choices agreeing
766-char sample text (20 ambiguous words)	20/20
2,600-token Wikipedia corpus (370 ambiguous words)	323/370 (86.5%) — equal to an internal UDPipe-taught reference; hand-checked disagreements split both ways

Tag-level (full 684-sentence ALKSNIS gold test, traditional-grammar projection): slots 86.4%, UPOS 90.6%, at 880 tok/s (ONNX INT8, CPU). For treebank-style tagging the -morphology sibling scores higher — this flavor exists for accentuation, where its conventions match the dictionary being scored against.

Not emitted by design: DET/AUX categories, FEATS beyond the ten agreement keys, lemmas (LEMMA = lowercased form; consumers apply a yra→būti rule for the copula homograph).

Usage (verified)

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "alexbalandi/litlat-bert-lithuanian-morphology-vdu"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)

words = ["Tas", "namas", "yra", "gražus"]
enc = tok(words, is_split_into_words=True, return_tensors="pt")
with torch.inference_mode():
    logits = model(**enc).logits[0]
seen = set()
for pos, wid in enumerate(enc.word_ids(0)):
    if wid is not None and wid not in seen:
        seen.add(wid)
        print(words[wid], model.config.id2label[int(logits[pos].argmax())])
# Tas PRON|Case=Nom|Gender=Masc|Number=Sing|Person=3   <- DET folded into PRON
# ...

An onnx/ folder with an INT8 model is included; a UDPipe-REST-protocol serving container lives in the project repo (local/tagger-hf/).

Training data & lineage

Gold corpora only, no NC-licensed model anywhere in the lineage: MATAS v3.0 (CC BY 4.0; UD features reconstructed from its Jablonskis XPOS) + UD_Lithuanian-ALKSNIS (CC BY-SA 4.0), gaps filled by iterated constrained-decoding self-training. Labels normalized to traditional-grammar categories in all splits.

License & attribution

CC BY-SA 4.0 (inherited from litlat-bert and ALKSNIS). Please credit: MATAS v3.0 (Rimkutė, Bielinskienė, Boizou, Dadurkevičius, Kovalevskaitė, Utka; CLARIN-LT), UD_Lithuanian-ALKSNIS (VDU), litlat-bert (Ulčar & Robnik-Šikonja, EMBEDDIA).