LexSPLADE (bert-large-uncased, silver, defining vocabulary)

A sparse contextual word encoder whose output is restricted to the Longman Defining Vocabulary (LDV), the ~2,000 words Longman's dictionary writes all its definitions with. Given a sentence with one word marked, it returns a sparse vector describing that word in that context, and every nonzero dimension is one of 1,897 basic English words. A vector reads as a short definition-style word list.

Usage

from sentence_transformers import SparseEncoder
import torch

model = SparseEncoder("jadermcs/lexsplade-bert-large-silver-ldv", trust_remote_code=True)

sents = [
    "He sat on the <t>bank</t> of the river and watched the water.",
    "She withdrew the money from her <t>bank</t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()

print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1))  # ~0.22

top = v[1].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['bank', 'money', 'account', 'pay', 'check', 'trust', 'borrow', 'debt']

trust_remote_code=True is required. The restriction lives in the custom pooling module (modeling_lexsplade.py, ids in 1_TargetWordSpladePooling/config.json). Loading the weights as a plain AutoModelForMaskedLM and pooling yourself will produce unrestricted vectors, which is not what this model was trained to emit.

Keep the marked word inside max_seq_length; if the markers are truncated away the pooling falls back to the whole sentence.

The vocabulary

Built from the Longman Dictionary of Contemporary English defining vocabulary, the ~2,000 basic words in which every LDOCE definition is written:

step dropped remaining
published list, single-word entries 2,188
must be one whole-word token in the BERT vocabulary (advertise โ†’ advert ##ise has no dimension of its own) 63 2,125
no stop words or punctuation 228 1,897

Shipped in this repo as longman_defining_vocabulary.txt (the word list) and ldv_vocabulary.json (the 1,897 words plus the token ids that can be nonzero). The word list is reproduced from the published LDOCE defining vocabulary for research use; the dictionary and its defining vocabulary are the work of Longman/Pearson.

Training

base model google-bert/bert-large-uncased
training data 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train)
objective AnglE loss on the sparse target-word vectors, SPLADE FLOPS regularizer (1e-3), Barlow Twins term on the dense span representation
output space 1,897 LDV dimensions, masked throughout training
seed 43 (best MCL-WiC dev AP of seeds 42โ€“44)

Evaluation

AP is average precision of the cosine similarity; accuracy is at the best threshold on that split.

set accuracy AP
MCL-WiC test (en) 91.30 96.13
MCL-WiC dev (en) 90.50 95.21
crosslemma dev 87.90 94.92

Over three seeds the same recipe gives MCL-WiC test AP 95.97 ยฑ 0.16, against 96.50 ยฑ 0.07 for the unrestricted BERT recipe.

Mean nonzero dimensions per usage on MCL-WiC test: 218 of 1,897.

Versions

Updated 2026-11-09 to the current recipe (stop words, punctuation and special tokens masked out of the output during training too). The first upload (seed 42, test AP 96.16) is still available as revision 8dda346a27808bdb3f2e44a1611393a9ded42e2f.

Variants

  • jadermcs/lexsplade-bert-large-silver: full 30,524-dim output.
  • jadermcs/lexsplade-roberta-large-silver-ldv: RoBERTa-large, defining vocabulary.
  • jadermcs/lexsplade-xlm-roberta-xl-silver-ldv: XLM-R-XL, defining vocabulary.
Downloads last month
75
Safetensors
Model size
0.3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for jadermcs/lexsplade-bert-large-silver-ldv

Finetuned
(161)
this model