LexSPLADE (bert-large-uncased, silver)

A sparse contextual word encoder. Given a sentence with one word marked, it returns a sparse vector over the 30,524-token BERT vocabulary describing that word in that context: a word embedding, not a sentence embedding. Because the dimensions are vocabulary entries, a vector can be read directly as a list of words.

Usage

from sentence_transformers import SparseEncoder
import torch

model = SparseEncoder("jadermcs/lexsplade-bert-large-silver", trust_remote_code=True)

sents = [
    "He sat on the <t>bank</t> of the river and watched the water.",
    "She withdrew the money from her <t>bank</t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()

print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1))  # ~0.19

# Read a vector as words
top = v[1].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['bank', 'money', 'banking', 'cash', 'security', 'financial', 'account', 'atm']
  • Wrap the target in <t> … </t>. Pooling runs over the tokens between the markers only.
  • trust_remote_code=True is required: the target-span pooling, which applies the MLM head only to the marked tokens, is a custom module shipped in this repo as modeling_lexsplade.py.
  • Keep the marked word inside max_seq_length. If the markers are truncated away, the pooling falls back to the whole sentence and returns a sentence embedding.

Training

base model google-bert/bert-large-uncased
training data 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train)
objective AnglE loss on the sparse target-word vectors, SPLADE FLOPS regularizer (1e-3), Barlow Twins term on the dense span representation
masked outputs special tokens, stop words and tokens without a letter or digit
seed 43 (best MCL-WiC dev AP of seeds 42–44)

Evaluation

AP is average precision of the cosine similarity; accuracy is at the best threshold on that split.

set accuracy AP
MCL-WiC test (en) 91.10 96.42
MCL-WiC dev (en) 90.60 95.73
crosslemma dev 88.91 95.79

Over three seeds the same recipe gives MCL-WiC test AP 96.50 ± 0.07, dev 95.49 ± 0.22, crosslemma 95.84 ± 0.15.

Mean nonzero dimensions per usage on MCL-WiC test: 480 of 30,524.

Versions

Updated 2026-11-09 to the current recipe, which masks stop words, punctuation and special tokens out of the output. The first upload (old recipe, seed 42, test AP 96.30, top words could include , and the) is still available as revision 6c03ccb27b9b7642b8223553f04be7999772f540:

SparseEncoder("jadermcs/lexsplade-bert-large-silver", trust_remote_code=True,
              revision="6c03ccb27b9b7642b8223553f04be7999772f540")

Variants

  • jadermcs/lexsplade-bert-large-silver-ldv: output restricted to the Longman Defining Vocabulary.
  • jadermcs/lexsplade-roberta-large-silver-wic-wnbags: RoBERTa-large.
  • jadermcs/lexsplade-xlm-roberta-large-silver: XLM-R-large (multilingual vocabulary).
  • jadermcs/lexsplade-xlm-roberta-xl-silver: XLM-R-XL, the strongest on English MCL-WiC.
Downloads last month
338
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for jadermcs/lexsplade-bert-large-silver

Finetuned
(161)
this model