jadermcs/lexsplade-roberta-large-silver-ldv

LexSPLADE (roberta-large, silver, defining vocabulary)

A sparse contextual word encoder whose output is restricted to the Longman Defining Vocabulary (LDV), the ~2,000 words Longman's dictionary writes all its definitions with. Given a sentence with one word marked, it returns a sparse vector describing that word in that context, and every nonzero dimension is one of 2,139 RoBERTa tokens that spell an LDV word. A vector reads as a short definition-style word list.

Usage

from sentence_transformers import SparseEncoder
import torch

model = SparseEncoder("jadermcs/lexsplade-roberta-large-silver-ldv", trust_remote_code=True)

sents = [
    "He sat on the <t>bank</t> of the river and watched the water.",
    "She withdrew the money from her <t>bank</t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()

print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1))  # ~0.16

# Read a vector as words
top = v[0].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['bank', 'river', 'shore', 'forest', 'rough', 'land', 'west', 'dry']

• Wrap the target in <t> … </t> with no spaces inside the markers (<t>bank</t>). RoBERTa's byte-level BPE turns <t> bank </t> into Ġbank plus a stray Ġ, which is not what the model was trained on.
• LDV words are allowed in their word-initial (Ġbank) and bare (bank) spellings where RoBERTa has them; strip Ġ when reading words off a vector.
• trust_remote_code=True is required: the target-span pooling (MLM head on the marked tokens only, logit centring, the vocabulary restriction) is a custom module shipped in this repo as modeling_lexsplade.py.
• Keep the marked word inside max_seq_length. If the markers are truncated away, the pooling falls back to the whole sentence and returns a sentence embedding.

Training

base model	FacebookAI/roberta-large
training data	31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train)
objective	AnglE loss on the sparse target-word vectors, SPLADE FLOPS regularizer (1e-3), Barlow Twins term on the dense span representation
output vocabulary	restricted to the 2,139 tokens spelling LDV words, applied before the loss, so training only ever sees the restricted vectors
logit centring	z = 1.5 (per token, subtract mean + 1.5·std of the logits before the relu)
masked outputs	special tokens, stop words and tokens without a letter or digit
seed	44 (best MCL-WiC dev AP of seeds 42–44)

Evaluation

AP is average precision of the cosine similarity; accuracy is at the best threshold on that split.

set	accuracy	AP
MCL-WiC test (en)	91.10	96.93
MCL-WiC dev (en)	91.50	96.03
crosslemma dev	87.70	94.23

Over three seeds the same recipe gives MCL-WiC test AP 97.02 ± 0.08; the unrestricted RoBERTa recipe gives 97.08 ± 0.13, so the restriction to 4% of the vocabulary costs almost nothing.

Mean nonzero dimensions per usage on MCL-WiC test: 105 of 2,139.

Variants

• jadermcs/lexsplade-xlm-roberta-xl-silver-ldv: XLM-R-XL, defining vocabulary.
• jadermcs/lexsplade-bert-large-silver-ldv: BERT-large, defining vocabulary.
• jadermcs/lexsplade-roberta-large-silver-wic-wnbags: RoBERTa-large, full vocabulary.