| --- |
| license: apache-2.0 |
| base_model: answerdotai/ModernBERT-Large-Instruct |
| language: en |
| tags: [word-sense-disambiguation, wordnet, modernbert] |
| --- |
| |
| # ModernBERT-Large-Instruct-WSD |
|
|
| Word sense disambiguation as multiple choice over WordNet (omw-en:1.4) definitions with a masked-LM answer |
| slot. Used by https://github.com/sign/word-sense-disambiguation (the `WSDModernBertForMaskedLM` class there |
| adds a 128-way answer-letter decoder; `answer_letters.json` lists the letters). |
|
|
| ## This revision (2026-09-04, "c3") |
|
|
| Fine-tuned from `answerdotai/ModernBERT-Large-Instruct` on: generated sentences for 97k synsets, WordNet's own |
| example sentences (minus a 5,000-example held-out slice), SemCor (222k instances, detokenized) and the |
| Princeton WordNet Gloss Corpus (manual tags). Cosine schedule, label smoothing 0.1, weight decay 0.01, |
| lr 2e-5, 2 epochs, batch 64, fp32 weights with bf16 autocast. Weights are stored in bf16. |
|
|
| Trained with the compact prompt template (the sentence with the target word marked, one definition per option |
| letter, a "none of the above" letter, then `[unused0] [MASK]`; 16% fewer tokens and ~20% faster than the earlier |
| wording). The repository code from PR #49 on builds exactly this prompt; earlier code/models are not compatible. |
|
|
| | Benchmark | previous revision (w4, earlier template) | this revision (c3) | |
| |---|---:|---:| |
| | WordNet held-out slice (5,000 examples, seed 42) | 78.6% | 78.3% | |
| | SemEval "ALL" (Senseval-2/3, SemEval-07/13/15; 7,247 instances, any gold key) | 80.7% | 80.6% | |
| | Throughput, one H100, same pipeline | 3,614 prompts/s | 4,326 prompts/s | |
|
|
| The previous revision over-predicted "none of the above" on long natural sentences; this one almost never does. |
| Full recipe and sweep configs: `training/` in the repository. |
|
|
