Feature Extraction
sentence-transformers
Safetensors
English
bert
splade
sparse-encoder
lexical-semantics
word-in-context
word-sense-disambiguation
text-embeddings-inference
Instructions to use jadermcs/lexsplade-bert-large-silver with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use jadermcs/lexsplade-bert-large-silver with sentence-transformers:
from sentence_transformers import SparseEncoder model = SparseEncoder("jadermcs/lexsplade-bert-large-silver") queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
LexSPLADE bert-large-uncased, silver pool, seed 42
Browse files- 1_TargetWordSpladePooling/config.json +7 -0
- README.md +80 -0
- config.json +31 -0
- config_sentence_transformers.json +14 -0
- model.safetensors +3 -0
- modeling_lexsplade.py +118 -0
- modules.json +14 -0
- sentence_bert_config.json +10 -0
- tokenizer.json +0 -0
- tokenizer_config.json +26 -0
1_TargetWordSpladePooling/config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"pooling_strategy": "sum",
|
| 3 |
+
"activation_function": "relu",
|
| 4 |
+
"embedding_dimension": 30524,
|
| 5 |
+
"m1_tok_id": 30522,
|
| 6 |
+
"m2_tok_id": 30523
|
| 7 |
+
}
|
README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language: en
|
| 4 |
+
library_name: sentence-transformers
|
| 5 |
+
base_model: google-bert/bert-large-uncased
|
| 6 |
+
pipeline_tag: feature-extraction
|
| 7 |
+
tags:
|
| 8 |
+
- splade
|
| 9 |
+
- sparse-encoder
|
| 10 |
+
- lexical-semantics
|
| 11 |
+
- word-in-context
|
| 12 |
+
- word-sense-disambiguation
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# LexSPLADE (bert-large-uncased, silver)
|
| 16 |
+
|
| 17 |
+
A **sparse contextual word encoder**. Given a sentence with one word marked, it
|
| 18 |
+
returns a sparse vector over the 30,524-token BERT vocabulary describing *that
|
| 19 |
+
word in that context* — a word embedding, not a sentence embedding. Because the
|
| 20 |
+
dimensions are vocabulary entries, a vector can be read directly as a list of
|
| 21 |
+
words.
|
| 22 |
+
|
| 23 |
+
The target word must be wrapped in `<t>` … `</t>`. Pooling runs over the tokens
|
| 24 |
+
between those markers only.
|
| 25 |
+
|
| 26 |
+
## Usage
|
| 27 |
+
|
| 28 |
+
```python
|
| 29 |
+
from sentence_transformers import SparseEncoder
|
| 30 |
+
import torch
|
| 31 |
+
|
| 32 |
+
model = SparseEncoder("jadermcs/lexsplade-bert-large-silver", trust_remote_code=True)
|
| 33 |
+
|
| 34 |
+
sents = [
|
| 35 |
+
"He sat on the <t> bank </t> of the river and watched the water.",
|
| 36 |
+
"She withdrew the money from her <t> bank </t> on Monday morning.",
|
| 37 |
+
]
|
| 38 |
+
v = model.encode(sents, convert_to_tensor=True).to_dense().float()
|
| 39 |
+
|
| 40 |
+
print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1)) # ~0.15
|
| 41 |
+
|
| 42 |
+
# Read a vector as words
|
| 43 |
+
top = v[0].topk(8).indices
|
| 44 |
+
print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
|
| 45 |
+
# ['bank', 'banks', 'side', 'water', ',', 'bed', 'shore', 'edge']
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
`trust_remote_code=True` is required: the target-span pooling is a custom module
|
| 49 |
+
shipped in this repo as `modeling_lexsplade.py`.
|
| 50 |
+
|
| 51 |
+
Keep the marked word inside `max_seq_length`. If the markers are truncated away
|
| 52 |
+
the pooling falls back to the whole sentence and silently returns a *sentence*
|
| 53 |
+
embedding.
|
| 54 |
+
|
| 55 |
+
## Training
|
| 56 |
+
|
| 57 |
+
| | |
|
| 58 |
+
|---|---|
|
| 59 |
+
| base model | `google-bert/bert-large-uncased` |
|
| 60 |
+
| training data | 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train) |
|
| 61 |
+
| objective | AnglE loss on the sparse target-word vectors, with a SPLADE FLOPS regularizer and a Barlow Twins term on the dense span representation |
|
| 62 |
+
| seed | 42 |
|
| 63 |
+
|
| 64 |
+
## Evaluation
|
| 65 |
+
|
| 66 |
+
MCL-WiC, English. AP is average precision; accuracy is at the best threshold.
|
| 67 |
+
|
| 68 |
+
| split | accuracy | AP |
|
| 69 |
+
|---|---|---|
|
| 70 |
+
| test | 91.20 | 96.30 |
|
| 71 |
+
| dev | 90.40 | 95.24 |
|
| 72 |
+
|
| 73 |
+
Mean nonzero dimensions per usage: **307** of 30,524.
|
| 74 |
+
|
| 75 |
+
## Variants
|
| 76 |
+
|
| 77 |
+
- **`jadermcs/lexsplade-bert-large-silver`** (this model) — full 30,524-dim output.
|
| 78 |
+
- **`jadermcs/lexsplade-bert-large-silver-ldv`** — identical training, but the
|
| 79 |
+
output is restricted to 1,897 dimensions drawn from the Longman Defining
|
| 80 |
+
Vocabulary, so every vector reads as a definition in basic English.
|
config.json
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_cross_attention": false,
|
| 3 |
+
"architectures": [
|
| 4 |
+
"BertForMaskedLM"
|
| 5 |
+
],
|
| 6 |
+
"attention_probs_dropout_prob": 0.1,
|
| 7 |
+
"bos_token_id": null,
|
| 8 |
+
"classifier_dropout": null,
|
| 9 |
+
"dtype": "float32",
|
| 10 |
+
"eos_token_id": null,
|
| 11 |
+
"gradient_checkpointing": false,
|
| 12 |
+
"hidden_act": "gelu",
|
| 13 |
+
"hidden_dropout_prob": 0.1,
|
| 14 |
+
"hidden_size": 1024,
|
| 15 |
+
"initializer_range": 0.02,
|
| 16 |
+
"intermediate_size": 4096,
|
| 17 |
+
"is_decoder": false,
|
| 18 |
+
"layer_norm_eps": 1e-12,
|
| 19 |
+
"max_position_embeddings": 512,
|
| 20 |
+
"model_type": "bert",
|
| 21 |
+
"num_attention_heads": 16,
|
| 22 |
+
"num_hidden_layers": 24,
|
| 23 |
+
"output_hidden_states": true,
|
| 24 |
+
"pad_token_id": 0,
|
| 25 |
+
"position_embedding_type": "absolute",
|
| 26 |
+
"tie_word_embeddings": true,
|
| 27 |
+
"transformers_version": "5.14.1",
|
| 28 |
+
"type_vocab_size": 2,
|
| 29 |
+
"use_cache": true,
|
| 30 |
+
"vocab_size": 30524
|
| 31 |
+
}
|
config_sentence_transformers.json
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"__version__": {
|
| 3 |
+
"pytorch": "2.13.0+cu130",
|
| 4 |
+
"sentence_transformers": "5.6.1",
|
| 5 |
+
"transformers": "5.14.1"
|
| 6 |
+
},
|
| 7 |
+
"default_prompt_name": null,
|
| 8 |
+
"model_type": "SparseEncoder",
|
| 9 |
+
"prompts": {
|
| 10 |
+
"document": "",
|
| 11 |
+
"query": ""
|
| 12 |
+
},
|
| 13 |
+
"similarity_fn_name": "cosine"
|
| 14 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8fdab7508faa9b57350a67638620f9721b4c75b051f74bb6160d42425700bc74
|
| 3 |
+
size 1340753168
|
modeling_lexsplade.py
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""LexSPLADE: target-word SPLADE pooling, loadable outside the training script.
|
| 2 |
+
|
| 3 |
+
The training script saved ``modules.json`` referring to
|
| 4 |
+
``__main__.DenseHiddenStatesTransformer`` / ``__main__.TargetWordSpladePooling``,
|
| 5 |
+
classes that only exist while that script is running -- so the checkpoint could
|
| 6 |
+
not be loaded by anyone else. These are the same two modules, in an importable
|
| 7 |
+
file, so ``SparseEncoder(repo, trust_remote_code=True)`` works.
|
| 8 |
+
|
| 9 |
+
What the pooling does: a sentence carries its target word wrapped in ``<t>`` /
|
| 10 |
+
``</t>``, and the SPLADE activations are pooled over **only the tokens between
|
| 11 |
+
those markers** rather than the whole sentence. The result is a sparse vector
|
| 12 |
+
over the MLM vocabulary describing *that word in that context* -- a word
|
| 13 |
+
embedding, not a sentence embedding.
|
| 14 |
+
|
| 15 |
+
``allowed_token_ids``, when present, is a hard restriction of the output space:
|
| 16 |
+
every dimension outside the set is zeroed before the vector is returned. The
|
| 17 |
+
LDV variant of this model is trained that way and is **only correct with the
|
| 18 |
+
restriction applied**, which is why the ids live in the pooling config rather
|
| 19 |
+
than in a README instruction.
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
|
| 24 |
+
import torch
|
| 25 |
+
from sentence_transformers.sparse_encoder import modules
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
class DenseHiddenStatesTransformer(modules.Transformer):
|
| 29 |
+
"""Fill-mask transformer that also exposes the last hidden layer.
|
| 30 |
+
|
| 31 |
+
The SPLADE head surfaces only the vocabulary-sized MLM logits as
|
| 32 |
+
``token_embeddings``. Training additionally needed the dense hidden states
|
| 33 |
+
for a Barlow Twins term; they are kept here so the module signature matches
|
| 34 |
+
the saved checkpoint. Inference does not depend on them.
|
| 35 |
+
"""
|
| 36 |
+
|
| 37 |
+
def forward(self, features: dict[str, torch.Tensor], **kwargs) -> dict[str, torch.Tensor]:
|
| 38 |
+
features = super().forward(features, **kwargs)
|
| 39 |
+
all_layers = features.pop("all_layer_embeddings", None)
|
| 40 |
+
if all_layers is not None:
|
| 41 |
+
features["dense_token_embeddings"] = all_layers[-1]
|
| 42 |
+
return features
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
class TargetWordSpladePooling(modules.SpladePooling):
|
| 46 |
+
"""SPLADE pooling restricted to the span between ``<t>`` and ``</t>``."""
|
| 47 |
+
|
| 48 |
+
config_keys = modules.SpladePooling.config_keys + [
|
| 49 |
+
"m1_tok_id", "m2_tok_id", "allowed_token_ids",
|
| 50 |
+
]
|
| 51 |
+
|
| 52 |
+
def __init__(
|
| 53 |
+
self,
|
| 54 |
+
*args,
|
| 55 |
+
m1_tok_id: int = 30522,
|
| 56 |
+
m2_tok_id: int = 30523,
|
| 57 |
+
allowed_token_ids: list[int] | None = None,
|
| 58 |
+
**kwargs,
|
| 59 |
+
):
|
| 60 |
+
super().__init__(*args, **kwargs)
|
| 61 |
+
self.m1_tok_id = m1_tok_id
|
| 62 |
+
self.m2_tok_id = m2_tok_id
|
| 63 |
+
self.allowed_token_ids = (
|
| 64 |
+
sorted(set(allowed_token_ids)) if allowed_token_ids else None
|
| 65 |
+
)
|
| 66 |
+
self._vocab_mask = None
|
| 67 |
+
|
| 68 |
+
def _vocab_mask_for(self, embeddings: torch.Tensor) -> torch.Tensor:
|
| 69 |
+
if (
|
| 70 |
+
self._vocab_mask is None
|
| 71 |
+
or self._vocab_mask.shape[0] != embeddings.shape[1]
|
| 72 |
+
or self._vocab_mask.device != embeddings.device
|
| 73 |
+
):
|
| 74 |
+
mask = torch.zeros(
|
| 75 |
+
embeddings.shape[1], dtype=torch.bool, device=embeddings.device
|
| 76 |
+
)
|
| 77 |
+
mask[torch.tensor(self.allowed_token_ids, device=embeddings.device)] = True
|
| 78 |
+
self._vocab_mask = mask
|
| 79 |
+
return self._vocab_mask
|
| 80 |
+
|
| 81 |
+
def forward(self, features: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:
|
| 82 |
+
input_ids = features["input_ids"]
|
| 83 |
+
attention_mask = features["attention_mask"]
|
| 84 |
+
|
| 85 |
+
is_open = input_ids == self.m1_tok_id
|
| 86 |
+
is_close = input_ids == self.m2_tok_id
|
| 87 |
+
after_open = is_open.cumsum(dim=1) > 0
|
| 88 |
+
before_close = is_close.cumsum(dim=1) == 0
|
| 89 |
+
target_mask = after_open & before_close & ~is_open & attention_mask.bool()
|
| 90 |
+
|
| 91 |
+
# Markers truncated away: pool the whole sentence rather than return an
|
| 92 |
+
# empty vector. Note this silently yields a *sentence* embedding, so
|
| 93 |
+
# keep the target inside max_seq_length.
|
| 94 |
+
empty = target_mask.sum(dim=1) == 0
|
| 95 |
+
if empty.any():
|
| 96 |
+
target_mask[empty] = attention_mask[empty].bool()
|
| 97 |
+
|
| 98 |
+
original_mask = features["attention_mask"]
|
| 99 |
+
features["attention_mask"] = target_mask.to(attention_mask.dtype)
|
| 100 |
+
|
| 101 |
+
dense_tokens = features.get("dense_token_embeddings")
|
| 102 |
+
if dense_tokens is not None:
|
| 103 |
+
m = target_mask.to(dense_tokens.dtype).unsqueeze(-1)
|
| 104 |
+
features["dense_sentence_embedding"] = (
|
| 105 |
+
(dense_tokens * m).sum(dim=1) / m.sum(dim=1).clamp(min=1.0)
|
| 106 |
+
)
|
| 107 |
+
|
| 108 |
+
features = super().forward(features)
|
| 109 |
+
features["attention_mask"] = original_mask
|
| 110 |
+
|
| 111 |
+
# masked_fill rather than a multiply: `max` pooling starts at -inf and
|
| 112 |
+
# -inf * 0 is NaN.
|
| 113 |
+
if self.allowed_token_ids is not None:
|
| 114 |
+
emb = features["sentence_embedding"]
|
| 115 |
+
features["sentence_embedding"] = emb.masked_fill(
|
| 116 |
+
~self._vocab_mask_for(emb), 0.0
|
| 117 |
+
)
|
| 118 |
+
return features
|
modules.json
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"idx": 0,
|
| 4 |
+
"name": "0",
|
| 5 |
+
"path": "",
|
| 6 |
+
"type": "modeling_lexsplade.DenseHiddenStatesTransformer"
|
| 7 |
+
},
|
| 8 |
+
{
|
| 9 |
+
"idx": 1,
|
| 10 |
+
"name": "1",
|
| 11 |
+
"path": "1_TargetWordSpladePooling",
|
| 12 |
+
"type": "modeling_lexsplade.TargetWordSpladePooling"
|
| 13 |
+
}
|
| 14 |
+
]
|
sentence_bert_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"transformer_task": "fill-mask",
|
| 3 |
+
"modality_config": {
|
| 4 |
+
"text": {
|
| 5 |
+
"method": "forward",
|
| 6 |
+
"method_output_name": "logits"
|
| 7 |
+
}
|
| 8 |
+
},
|
| 9 |
+
"module_output_name": "token_embeddings"
|
| 10 |
+
}
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"backend": "tokenizers",
|
| 3 |
+
"cls_token": "[CLS]",
|
| 4 |
+
"do_lower_case": true,
|
| 5 |
+
"extra_special_tokens": [
|
| 6 |
+
"<t>",
|
| 7 |
+
"</t>"
|
| 8 |
+
],
|
| 9 |
+
"is_local": true,
|
| 10 |
+
"local_files_only": false,
|
| 11 |
+
"mask_token": "[MASK]",
|
| 12 |
+
"max_length": 64,
|
| 13 |
+
"model_max_length": 64,
|
| 14 |
+
"pad_to_multiple_of": null,
|
| 15 |
+
"pad_token": "[PAD]",
|
| 16 |
+
"pad_token_type_id": 0,
|
| 17 |
+
"padding_side": "right",
|
| 18 |
+
"sep_token": "[SEP]",
|
| 19 |
+
"stride": 0,
|
| 20 |
+
"strip_accents": null,
|
| 21 |
+
"tokenize_chinese_chars": true,
|
| 22 |
+
"tokenizer_class": "BertTokenizer",
|
| 23 |
+
"truncation_side": "right",
|
| 24 |
+
"truncation_strategy": "longest_first",
|
| 25 |
+
"unk_token": "[UNK]"
|
| 26 |
+
}
|