--- license: cc-by-nc-4.0 language: - grc base_model: - pranaydeeps/Ancient-Greek-BERT pipeline_tag: token-classification tags: - morphological-tagging - ancient-greek - nlp - universal-dependencies - low-resource-language - digital-humanities metrics: - accuracy - f1 datasets: - proiel --- # DART-FT: Decomposed, Ambiguity-Aware, Risk-Targeted Fine-Tuning **DART-FT** is an Ancient Greek fine-grained morphological tagging model and targeted fine-tuning methodology developed as part of a Doctor of Engineering praxis at The George Washington University. 📄 **Full doctoral praxis:** **[DART-FT: Decomposed, Ambiguity-Aware, Risk-Targeted Fine-Tuning of Transformer Models for Ancient Greek Morphological Tagging](https://scholarspace.library.gwu.edu/etd/4x51hk097)** DART-FT addresses a specific challenge in low-resource, morphologically rich language modeling: strong fine-tuned transformer models can achieve high aggregate performance while retaining systematic errors concentrated in difficult morphological feature-value patterns. Rather than broadly retraining the model, DART-FT identifies an empirically defined **high-risk slice** associated with recurrent baseline tagging errors and performs targeted fine-tuning in which training examples associated with the high-risk patterns are selectively oversampled. Across five random seeds, DART-FT significantly improved fine-grained morphological tagging accuracy on the predefined high-risk slice relative to corresponding standard cross-entropy fine-tuned Ancient-Greek-BERT baselines. The best-performing configuration improved high-risk-slice accuracy by **6.4 percentage points**, from **63.4% to 69.9%** (**p < .001**), while global macro-F1 showed no observed degradation. ## Model Details - **Developed by:** Matthew K. Weber - **Model:** DART-FT - **Model type:** BERT encoder with decomposed morphological classification heads - **Base model:** [`pranaydeeps/Ancient-Greek-BERT`](https://huggingface.co/pranaydeeps/Ancient-Greek-BERT) - **Language:** Ancient Greek (`grc`) - **Primary task:** Fine-grained morphological tagging - **Framework:** PyTorch / Hugging Face Transformers - **Dataset:** UD Ancient Greek PROIEL v2.12 - **Doctoral program:** Doctor of Engineering, The George Washington University - **Praxis:** [Full text in GW ScholarSpace](https://scholarspace.library.gwu.edu/etd/4x51hk097) ## What DART-FT Predicts DART-FT predicts fine-grained morphology using **decomposed classification** rather than treating every complete morphological bundle as a single class. The released schema contains a UPOS head and separate heads for 15 Universal Dependencies morphological attributes: - UPOS - Case - Gender - Number - Definite - PronType - Aspect - Tense - VerbForm - Voice - Degree - Mood - Person - Polarity - Reflex - Poss Individual predictions can then be reconstructed into a complete fine-grained morphological representation such as: ```text NOUN|Case=Acc|Gender=Masc|Number=Sing ``` This decomposition is intended to reduce the data-sparsity problems that arise when every possible combination of morphological features is represented as an independent class. --- # DART-FT Methodology DART-FT combines three principal ideas: 1. **Decomposed morphological classification** 2. **Empirical identification of a high-risk slice** 3. **Risk-targeted fine-tuning through oversampling** ## 1. Decomposed Morphological Classification Instead of representing a complete FEATS bundle as one classification label, DART-FT decomposes each bundle into individual morphological attributes for independent per-feature classification. For example: ```text Case=Nom Gender=Masc Number=Sing ``` are modeled as separate classification decisions. The model additionally predicts UPOS. This architecture reduces sparsity associated with treating the full Cartesian combination of morphological attributes as distinct labels. ## 2. Empirical High-Risk Slice Identification The **high-risk slice is not a predefined dictionary of difficult Ancient Greek words**. Instead, it is identified empirically from recurrent errors made by a standard fine-tuned Ancient-Greek-BERT baseline. The methodology uses a **three-pass process**. ### Pass 1 — Error-Driven Mining The baseline model predicts morphological labels on the validation set. Predicted feature-value pairs are compared with gold labels to identify recurrent error patterns. ### Pass 2 — Feature-Value Filtering and Diversity Constraint Candidate feature-value pairs are filtered to isolate sufficiently frequent and systematic error patterns while preventing a small number of broad categories from dominating the high-risk slice. ### Pass 3 — Slice Freezing and Mask Creation The resulting feature-value definition is frozen. A mask based on this frozen definition is then applied to held-out test data for consistent evaluation across models. The validation set is used to define the high-risk slice so that the test set remains held out from training and model-selection activities. Although morphological ambiguity motivates DART-FT, membership in the high-risk slice is determined by **observed model error behavior at the feature-value level**, rather than lexical identity alone. ## 3. Risk-Targeted Fine-Tuning After the high-risk feature-value patterns are identified, DART-FT performs an additional stage of targeted fine-tuning. Training sentences containing tokens associated with the high-risk patterns are selectively **oversampled**, increasing model exposure to those difficult cases while preserving sentence-level context. The research evaluated: - oversampling factors of **2x, 3x, and 4x** - five random seeds - cross-entropy loss - class-balanced loss - class-balanced + focal loss - multiple class-balanced β values - multiple focal-loss γ values A total of **90 distinct experimental models** were evaluated. --- # Released Model Configuration The model artifact in this repository corresponds to the highest-performing DART-FT configuration evaluated in the research: - **Random seed:** 43 - **Oversampling factor:** 3x - **Loss:** Class-balanced - **β:** 0.993 - **Learning rate:** 3 × 10⁻⁵ - **Batch size:** 8 - **Training epochs:** 5 - **Maximum sequence length:** 512 - **Optimizer:** AdamW - **Weight decay:** 0.01 - **Dropout:** 0.1 - **Mixed precision:** FP16 The repository includes the fine-tuned Ancient-Greek-BERT encoder and separately stored decomposed classification heads. Important files include: ```text model.safetensors heads.pt schema.json meta.json test_pred.conllu ``` ### Repository Components **`model.safetensors`** Fine-tuned transformer encoder weights. **`heads.pt`** Trained decomposed morphological classification heads. **`schema.json`** Label inventory for UPOS and each morphological feature head. **`meta.json`** Metadata describing the released experimental configuration. **`test_pred.conllu`** Model predictions in CoNLL-U-compatible form. --- # Training and Evaluation Data DART-FT was trained and evaluated using the **Universal Dependencies Ancient Greek PROIEL treebank, version 2.12**. The pre-existing PROIEL train, validation, and test splits were preserved. | Split | Sentences | Tokens | UPOS Tags | UFEATS Attributes | |---|---:|---:|---:|---:| | Train | 15,016 | 187,039 | 13 | 15 | | Dev / Validation | 1,019 | 13,652 | 13 | 15 | | Test | 1,047 | 13,314 | 13 | 15 | | **Total** | **17,082** | **214,005** | **13** | **15** | The Ancient Greek PROIEL corpus used in the study includes the **27 books of the Ancient Greek New Testament and portions of Herodotus**. The New Testament portion of PROIEL itself contains approximately **140,000 tokens**. This should not be confused with the full experimental dataset, which contains **214,005 tokens across the train, validation, and test splits**. --- # Tokenization and Label Alignment DART-FT uses the WordPiece tokenization employed by BERT. Because one source word can be divided into multiple subword tokens: - the **first subword** receives the corresponding UPOS and morphological labels; - subsequent subwords are **excluded from the classification loss**. This prevents a word that is split into multiple WordPieces from contributing duplicate morphological training labels and promotes learning at the word level. --- # Evaluation DART-FT was evaluated from two complementary perspectives: 1. **Fine-grained morphological accuracy on the high-risk slice** 2. **Global macro-F1** ## High-Risk Exact-Match Accuracy Fine-grained accuracy uses the complete morphological prediction for a word. The prediction must match the corresponding gold morphological representation for the token; partial corrections do not count as an exact match. ## Macro-F1 Macro-F1 gives classes equal weight rather than allowing highly frequent morphological categories to dominate the evaluation. Global macro-F1 was used specifically to evaluate whether targeted intervention on the high-risk slice caused broader model degradation. ## Statistical Testing Paired baseline and DART-FT predictions were evaluated using the **exact two-sided McNemar test**. Confidence intervals were also calculated for the paired accuracy differences. --- # Results ## High-Risk Slice Performance For the highest-performing DART-FT model associated with each random seed: | Seed | Baseline Accuracy | DART-FT Accuracy | Change | p-value | |---:|---:|---:|---:|---:| | 41 | 59.3% | 63.9% | +4.5 pp | < .001 | | 42 | 65.6% | 70.7% | +5.2 pp | .003 | | 43 | 63.4% | 69.9% | **+6.4 pp** | **< .001** | | 44 | 57.5% | 63.2% | +5.7 pp | < .001 | | 45 | 64.5% | 69.8% | +5.3 pp | < .001 | The largest observed improvement was produced by **seed 43, 3x oversampling, class-balanced loss, β = 0.993**: ```text Baseline high-risk accuracy: 63.4% DART-FT high-risk accuracy: 69.9% Absolute improvement: +6.4 percentage points p-value: < .001 95% confidence interval: 3.38–9.14 pp ``` All five seed comparisons exhibited statistically significant increases in high-risk-slice accuracy. ## Global Model Stability Targeted fine-tuning did **not** produce an observed decrease in global macro-F1 for any of the five random seeds. | Seed | Baseline Global Macro-F1 | DART-FT Global Macro-F1 | Change | |---:|---:|---:|---:| | 41 | 0.921 | 0.939 | +0.018 | | 42 | 0.919 | 0.940 | +0.021 | | 43 | 0.917 | 0.939 | +0.022 | | 44 | 0.919 | 0.938 | +0.019 | | 45 | 0.920 | 0.937 | +0.017 | The mean global macro-F1 improvement was **+0.0194**. These results support the central DART-FT objective: targeted repair of systematic fine-grained morphological tagging errors without observed degradation of overall model performance. ## Error Analysis Error analysis showed that targeted fine-tuning corrected recurring errors across multiple morphological features. A particularly strong effect was observed for the `Gender=Masc,Neut` feature-value pair, where the standard baseline frequently predicted one of the more common individual gender categories rather than the syncretic `Masc,Neut` value. The research also identified localized regressions in some tightly coupled features, demonstrating that targeted fine-tuning can introduce tradeoffs even when aggregate high-risk and global performance improve. --- # Oversampling Sensitivity DART-FT explicitly evaluated whether performance was sensitive to the amount of high-risk oversampling. The tested oversampling factors were: ```text 2x 3x 4x ``` Performance varied across seeds and configurations. Across the experiments, intermediate oversampling generally produced the strongest mean improvement, and the highest-performing individual configuration used **3x oversampling**. This supports the conclusion that targeted fine-tuning effectiveness depends on the strength of the intervention and that simply maximizing oversampling is not necessarily optimal. --- # Intended Uses ## Fine-Grained Ancient Greek Morphological Tagging DART-FT can support prediction of grammatical information including: - part of speech - case - gender - number - tense - mood - voice - person - aspect - degree - pronoun type - definiteness - polarity - reflexivity - possession ## Digital Humanities Potential applications include automated annotation and analysis of Ancient Greek corpora, including historical and biblical texts. ## NLP Pipelines Fine-grained morphological predictions may provide structured linguistic information for downstream tasks such as: - syntactic parsing - corpus analysis - information extraction - search - experimental machine translation workflows Downstream benefits should be independently evaluated for each application. ## Research on Targeted Model Repair DART-FT may also be useful as an example of **error-driven targeted fine-tuning**. The broader methodology identifies a reproducible model weakness through validation-set behavior and selectively increases exposure to corresponding training examples rather than indiscriminately increasing training across the entire corpus. --- # Downstream Translation Pilot The praxis also included a small-scale exploratory Ancient Greek-to-Latin translation pilot. The pilot compared morphological information produced by the DART-FT model with the corresponding baseline model when supplied to a downstream translation system. The results provided limited evidence of downstream improvement: - **chrF++:** +0.24, p = 0.0408 - **TER:** 59.42 → 59.27 - for the subset in which baseline and DART-FT morphological sequences differed (`N = 232`), chrF++ improved by +0.37 and TER decreased by 0.23 This pilot was **secondary and exploratory**. DART-FT should not be interpreted as a production machine-translation system, and the translation results should not be generalized beyond the limited experiment. --- # Limitations ## Corpus Scope DART-FT was trained and evaluated on the Ancient Greek PROIEL treebank, consisting principally of New Testament Greek and portions of Herodotus. Performance should not be assumed to transfer unchanged to: - Homeric Greek - Classical Attic Greek - Septuagint Greek - papyri - inscriptions - patristic Greek - Byzantine Greek - other Ancient Greek treebanks or annotation schemes ## Generalizability Generalizability of the methodology to other languages and domains is plausible but was **not empirically validated** in the praxis. Applying DART-FT to another problem requires, at minimum: - a gold-labeled dataset with token-level annotations; - a pretrained model capable of task-specific fine-tuning; and - a finite, decomposable feature set suitable for identifying systematic feature-level errors. ## High-Risk Slice Definition The high-risk slice is empirically defined and depends on methodological thresholds selected during the research. Different thresholds or filtering rules may result in different high-risk populations. ## Model Dependence The high-risk slice reflects weaknesses exhibited by a particular model and dataset. Different pretrained models, datasets, or annotation systems may produce different concentrations of errors. ## Oversampling Tradeoffs Oversampling increases model exposure to difficult examples but can also increase the risk of overfitting or localized negative transfer. DART-FT therefore treats oversampling strength as a hyperparameter rather than assuming that greater oversampling will always yield greater performance. --- # How to Use > **Important:** The released DART-FT repository is not packaged as a conventional single-head Hugging Face `AutoModelForTokenClassification` checkpoint. DART-FT uses a BERT encoder plus **separately stored decomposed morphological classification heads**. Therefore, a call such as: ```python AutoModelForTokenClassification.from_pretrained("zootsewt/DART-FT") ``` does **not by itself reproduce the complete DART-FT inference architecture**. The transformer encoder and tokenizer can be loaded with: ```python from transformers import AutoTokenizer, AutoModel model_name = "zootsewt/DART-FT" tokenizer = AutoTokenizer.from_pretrained(model_name) encoder = AutoModel.from_pretrained(model_name) ``` The decomposed classification heads stored in: ```text heads.pt ``` must also be loaded and applied according to the label inventory in: ```text schema.json ``` The individual head predictions are then combined to reconstruct a complete fine-grained morphological analysis for each source token. A reference inference wrapper may be provided in a future repository update to automate loading of the encoder, classification heads, schema, token alignment, and output reconstruction. --- # Research Context DART-FT was developed as the principal research contribution of the Doctor of Engineering praxis: ** _DART-FT: Decomposed, Ambiguity-Aware, Risk-Targeted Fine-Tuning of Transformer Models for Ancient Greek Morphological Tagging._ The George Washington University.** ### Read the full praxis **https://scholarspace.library.gwu.edu/etd/4x51hk097** --- # Citation If you use DART-FT or the associated methodology in academic work, please cite the doctoral praxis: ```bibtex @phdthesis{weber2026dartft, author = {Weber, Matthew K.}, title = {DART-FT: Decomposed, Ambiguity-Aware, Risk-Targeted Fine-Tuning of Transformer Models for Ancient Greek Morphological Tagging}, school = {The George Washington University}, year = {2026}, type = {Doctor of Engineering Praxis}, url = {https://scholarspace.library.gwu.edu/etd/4x51hk097} } ``` **Praxis:** https://scholarspace.library.gwu.edu/etd/4x51hk097 --- # Disclaimer DART-FT is a research artifact developed and evaluated within the scope of a Doctor of Engineering praxis. Reported results reflect the specific datasets, baseline model, experimental design, high-risk slice definition, hyperparameters, and evaluation methods used in that research. Results should not be assumed to generalize unchanged to other corpora, models, languages, annotation systems, or downstream applications.