jadermcs/lexsplade-xlm-roberta-large-silver

LexSPLADE (xlm-roberta-large, silver)

A sparse contextual word encoder. Given a sentence with one word marked, it returns a sparse vector over XLM-R's 250,004-token vocabulary describing that word in that context: a word embedding, not a sentence embedding. Because the dimensions are vocabulary entries, a vector can be read directly as a list of words.

Trained on English data only. It shares its base with XL-LEXEME (pierluigic/xl-lexeme), so the two can be compared on the same backbone.

Usage

from sentence_transformers import SparseEncoder
import torch

model = SparseEncoder("jadermcs/lexsplade-xlm-roberta-large-silver", trust_remote_code=True)

sents = [
    "He sat on the <t>bank</t> of the river and watched the water.",
    "She withdrew the money from her <t>bank</t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()

print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1))  # ~0.08

# Read a vector as words
top = v[1].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['▁bank', '▁home', '▁account', '▁checking', '▁deposit', '▁banc', '▁online', '▁card']

• Wrap the target in <t> … </t>; <t>bank</t> is the form used in training.
• trust_remote_code=True is required: the target-span pooling, which applies the MLM head only to the marked tokens and centres the logits, is a custom module shipped in this repo as modeling_lexsplade.py. Running the MLM head on the span alone also avoids a [batch, seq, 250k] logit tensor.
• Keep the marked word inside max_seq_length. If the markers are truncated away, the pooling falls back to the whole sentence and returns a sentence embedding.

Training

base model	FacebookAI/xlm-roberta-large
training data	31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train)
objective	AnglE loss on the sparse target-word vectors, SPLADE FLOPS regularizer (1e-3), Barlow Twins term on the dense span representation
logit centring	z = 2.5 (per token, subtract mean + 2.5·std of the logits before the relu; XLM-R's raw logits sit far above zero)
masked outputs	special tokens, stop words and tokens without a letter or digit
seed	44

Evaluation

MCL-WiC, English. AP is average precision of the cosine similarity; accuracy is at the best threshold on that split.

split	accuracy	AP
test	91.70	96.67
dev	91.00	95.07

Mean nonzero dimensions per usage on MCL-WiC test: 246 of 250,004.

Variants

• jadermcs/lexsplade-bert-large-silver: BERT-large, silver only.
• jadermcs/lexsplade-roberta-large-silver-wic-wnbags: RoBERTa-large, silver + WiC, with WordNet sense bags.