Instructions to use jadermcs/lexsplade-bert-large-silver-ldv with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use jadermcs/lexsplade-bert-large-silver-ldv with sentence-transformers:
from sentence_transformers import SparseEncoder model = SparseEncoder("jadermcs/lexsplade-bert-large-silver-ldv") queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
LexSPLADE (bert-large-uncased, silver, defining vocabulary)
A sparse contextual word encoder whose output is restricted to the Longman Defining Vocabulary (LDV), the ~2,000 words Longman's dictionary writes all its definitions with. Given a sentence with one word marked, it returns a sparse vector describing that word in that context, and every nonzero dimension is one of 1,897 basic English words. A vector reads as a short definition-style word list.
Usage
from sentence_transformers import SparseEncoder
import torch
model = SparseEncoder("jadermcs/lexsplade-bert-large-silver-ldv", trust_remote_code=True)
sents = [
"He sat on the <t>bank</t> of the river and watched the water.",
"She withdrew the money from her <t>bank</t> on Monday morning.",
]
v = model.encode(sents, convert_to_tensor=True).to_dense().float()
print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1)) # ~0.22
top = v[1].topk(8).indices
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
# ['bank', 'money', 'account', 'pay', 'check', 'trust', 'borrow', 'debt']
trust_remote_code=True is required. The restriction lives in the custom
pooling module (modeling_lexsplade.py, ids in
1_TargetWordSpladePooling/config.json). Loading the weights as a plain
AutoModelForMaskedLM and pooling yourself will produce unrestricted vectors,
which is not what this model was trained to emit.
Keep the marked word inside max_seq_length; if the markers are truncated away
the pooling falls back to the whole sentence.
The vocabulary
Built from the Longman Dictionary of Contemporary English defining vocabulary, the ~2,000 basic words in which every LDOCE definition is written:
| step | dropped | remaining |
|---|---|---|
| published list, single-word entries | 2,188 | |
must be one whole-word token in the BERT vocabulary (advertise โ advert ##ise has no dimension of its own) |
63 | 2,125 |
| no stop words or punctuation | 228 | 1,897 |
Shipped in this repo as longman_defining_vocabulary.txt (the word list) and
ldv_vocabulary.json (the 1,897 words plus the token ids that can be nonzero).
The word list is reproduced from the published LDOCE defining vocabulary for
research use; the dictionary and its defining vocabulary are the work of
Longman/Pearson.
Training
| base model | google-bert/bert-large-uncased |
| training data | 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train) |
| objective | AnglE loss on the sparse target-word vectors, SPLADE FLOPS regularizer (1e-3), Barlow Twins term on the dense span representation |
| output space | 1,897 LDV dimensions, masked throughout training |
| seed | 43 (best MCL-WiC dev AP of seeds 42โ44) |
Evaluation
AP is average precision of the cosine similarity; accuracy is at the best threshold on that split.
| set | accuracy | AP |
|---|---|---|
| MCL-WiC test (en) | 91.30 | 96.13 |
| MCL-WiC dev (en) | 90.50 | 95.21 |
| crosslemma dev | 87.90 | 94.92 |
Over three seeds the same recipe gives MCL-WiC test AP 95.97 ยฑ 0.16, against 96.50 ยฑ 0.07 for the unrestricted BERT recipe.
Mean nonzero dimensions per usage on MCL-WiC test: 218 of 1,897.
Versions
Updated 2026-11-09 to the current recipe (stop words, punctuation and special
tokens masked out of the output during training too). The first upload (seed 42,
test AP 96.16) is still available as revision
8dda346a27808bdb3f2e44a1611393a9ded42e2f.
Variants
jadermcs/lexsplade-bert-large-silver: full 30,524-dim output.jadermcs/lexsplade-roberta-large-silver-ldv: RoBERTa-large, defining vocabulary.jadermcs/lexsplade-xlm-roberta-xl-silver-ldv: XLM-R-XL, defining vocabulary.
- Downloads last month
- 75
Model tree for jadermcs/lexsplade-bert-large-silver-ldv
Base model
google-bert/bert-large-uncased