thealper2/deberta-v3-base-winogrande-coref

DeBERTa-v3-base span-based coreference resolution (WinoGrande)

End-to-end span-based coreference model (mention detection + antecedent ranking) with a microsoft/deberta-v3-base encoder, fine-tuned on coref-data/winogrande_coref. The coreference layers are implemented from scratch in PyTorch. This is not a sequence classification head.

Architecture

[CLS] document [SEP] -> DeBERTa-v3-base (184M) -> h_t (768-d)
candidate span i (word-level enumeration, mapped to subwords via offset mapping)
g_i   = [h_start ; h_end ; attn_pool(h_start..h_end) ; phi_width]                 (2324-d)
s_m(i) = FFNN_m(g_i)                                                               mention score
prune to k = min(30, ceil(0.4 * #words)) spans, restored to document order
s_a(i,j) = FFNN_a([g_i ; g_j ; g_i * g_j ; phi_dist(i,j)])                         j < i only
s(i,j) = s_m(i) + s_m(j) + s_a(i,j),     s(i,eps) = 0                              dummy antecedent
y_i = argmax_{y in {eps} U {j<i}} s(i,y)  -> links -> union-find clusters

Component	Setting
Candidate spans	width ≤ 10 words, within one sentence, no punctuation-only boundary word
Width / distance embeddings	20-d each; distance buckets 0,1,2,3,4,5–7,8–15,16–31,32–63,64+
FFNNs	1 hidden layer, 1024 units, GELU, dropout 0.3
Task parameters	9.5M
Singleton decoding	a span with no antecedent is a mention iff σ(s_m) > 0.5

Training objective

With Y(i) = {eps} ∪ {j < i} and P(y | i) = softmax_y s(i, y):

L = − 1/B Σ_docs [ Σ_i log Σ_{y ∈ GOLD(i)} P(y | i)
                  + λ Σ_s ( z_s log σ(s_m(s)) + (1 − z_s) log(1 − σ(s_m(s))) ) ]

GOLD(i) is the set of kept spans from i's gold cluster that precede i, or {eps} if there are none (non-mentions, first mentions, singletons). z_s indicates that candidate s is a gold mention, and λ = 1. The marginal likelihood spreads probability mass over all correct antecedents. The auxiliary mention loss is needed because the dataset annotates singletons, which the antecedent loss alone cannot tell apart from non-mentions.

Data

Split	Source	Documents
train	HF train minus dev groups	34,229
dev (model selection)	5% of HF train twin groups (seed 42), grouped by WinoGrande twin id	1,805
test	HF validation	1,117

Dataset revision: e5a194ba9c53da01acfb8c79589811fb87d7b9cf. Every document has one 2-mention chain (antecedent + pronoun) and one singleton (the distractor). Gold spans are [sentence, start, end], with sentence-local word indices and inclusive ends. 99.91% of training gold mentions are reachable as candidate spans. The other 93 have a punctuation boundary word, mostly annotation noise such as ", sneakers". No document is truncated (max 45 subwords).

Training hyper-parameters

Parameter	Value
Optimizer	AdamW (fused), weight decay 0.01
LR encoder / task layers	1e-5 / 2e-4
Schedule	linear, 10% warmup
Batch size	16 documents, no accumulation
Epochs	up to 10, early stopping (patience 3 on dev CoNLL F1)
Gradient clipping	1.0
Precision	bf16 autocast, fp32 master weights
Seed	42
Hardware	1× RTX 5060 Ti 16GB

Evaluation

Corpus-level MUC, B³, CEAF-e, CEAF-m, CoNLL F1 = mean(MUC, B³, CEAF-e). Singletons are kept (the dataset annotates them). CoNLL F1 without singletons is also reported. Mention detection uses exact boundaries. Antecedent accuracy counts gold anaphors whose predicted antecedent belongs to their gold cluster.

Evaluation split: official validation split (1,117 docs)

Metric	P	R	F1
Mention detection	97.44	97.85	97.65
MUC	76.08	76.01	76.04
B³	90.06	90.38	90.22
CEAF-e	89.79	90.40	90.09
CEAF-m	90.22	90.60	90.41
CoNLL F1 (with singletons)			85.45
CoNLL F1 (singletons removed)			81.93

Antecedent metric	Value
Gold anaphors	1117
Anaphor detection rate	99.37
Antecedent accuracy	76.01
Resolution accuracy	76.01

Usage

The model needs the code files in this repo (model.py, config.py, preprocessing.py, dataset.py).

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("thealper2/deberta-v3-base-winogrande-coref")
sys.path.insert(0, path)
from model import CoreferenceModel

model = CoreferenceModel.from_pretrained(path, device="cuda")
result = model.predict(text="Sarah was a much better surgeon than Maria so she always got the harder cases.")
# result = {"mentions": [...], "clusters": [...], "antecedents": [...]}, character offsets, end exclusive

Limitations

• Trained only on short, single-paragraph WinoGrande documents (≤ 3 sentences, 2 candidate entities). It is not evaluated on long documents; inputs are limited to max_seq_length = 256 subwords.
• Gold mentions are only the two candidates and the pronoun. Other entities in the text are labelled as non-mentions, so mention detection is dataset-specific.
• The raw-text inference word splitter is regex-based (PTB-like clitic splitting); training used stanza tokens.