---
license: apache-2.0
base_model: answerdotai/ModernBERT-Large-Instruct
language: en
tags: [word-sense-disambiguation, wordnet, modernbert]
---

# ModernBERT-Large-Instruct-WSD

Word sense disambiguation as multiple choice over WordNet (omw-en:1.4) definitions with a masked-LM answer
slot. Used by https://github.com/sign/word-sense-disambiguation (the `WSDModernBertForMaskedLM` class there
adds a 128-way answer-letter decoder; `answer_letters.json` lists the letters).

## This revision (2026-09-04, "c3")

Fine-tuned from `answerdotai/ModernBERT-Large-Instruct` on: generated sentences for 97k synsets, WordNet's own
example sentences (minus a 5,000-example held-out slice), SemCor (222k instances, detokenized) and the
Princeton WordNet Gloss Corpus (manual tags). Cosine schedule, label smoothing 0.1, weight decay 0.01,
lr 2e-5, 2 epochs, batch 64, fp32 weights with bf16 autocast. Weights are stored in bf16.

Trained with the compact prompt template (the sentence with the target word marked, one definition per option
letter, a "none of the above" letter, then `[unused0] [MASK]`; 16% fewer tokens and ~20% faster than the earlier
wording). The repository code from PR #49 on builds exactly this prompt; earlier code/models are not compatible.

| Benchmark | previous revision (w4, earlier template) | this revision (c3) |
|---|---:|---:|
| WordNet held-out slice (5,000 examples, seed 42) | 78.6% | 78.3% |
| SemEval "ALL" (Senseval-2/3, SemEval-07/13/15; 7,247 instances, any gold key) | 80.7% | 80.6% |
| Throughput, one H100, same pipeline | 3,614 prompts/s | 4,326 prompts/s |

The previous revision over-predicted "none of the above" on long natural sentences; this one almost never does.
Full recipe and sweep configs: `training/` in the repository.