jadermcs commited on
Commit
6c03ccb
·
verified ·
1 Parent(s): c65a167

LexSPLADE bert-large-uncased, silver pool, seed 42

Browse files
1_TargetWordSpladePooling/config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "pooling_strategy": "sum",
3
+ "activation_function": "relu",
4
+ "embedding_dimension": 30524,
5
+ "m1_tok_id": 30522,
6
+ "m2_tok_id": 30523
7
+ }
README.md ADDED
@@ -0,0 +1,80 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language: en
4
+ library_name: sentence-transformers
5
+ base_model: google-bert/bert-large-uncased
6
+ pipeline_tag: feature-extraction
7
+ tags:
8
+ - splade
9
+ - sparse-encoder
10
+ - lexical-semantics
11
+ - word-in-context
12
+ - word-sense-disambiguation
13
+ ---
14
+
15
+ # LexSPLADE (bert-large-uncased, silver)
16
+
17
+ A **sparse contextual word encoder**. Given a sentence with one word marked, it
18
+ returns a sparse vector over the 30,524-token BERT vocabulary describing *that
19
+ word in that context* — a word embedding, not a sentence embedding. Because the
20
+ dimensions are vocabulary entries, a vector can be read directly as a list of
21
+ words.
22
+
23
+ The target word must be wrapped in `<t>` … `</t>`. Pooling runs over the tokens
24
+ between those markers only.
25
+
26
+ ## Usage
27
+
28
+ ```python
29
+ from sentence_transformers import SparseEncoder
30
+ import torch
31
+
32
+ model = SparseEncoder("jadermcs/lexsplade-bert-large-silver", trust_remote_code=True)
33
+
34
+ sents = [
35
+ "He sat on the <t> bank </t> of the river and watched the water.",
36
+ "She withdrew the money from her <t> bank </t> on Monday morning.",
37
+ ]
38
+ v = model.encode(sents, convert_to_tensor=True).to_dense().float()
39
+
40
+ print(torch.nn.functional.cosine_similarity(v[0:1], v[1:2], dim=1)) # ~0.15
41
+
42
+ # Read a vector as words
43
+ top = v[0].topk(8).indices
44
+ print([model.tokenizer.convert_ids_to_tokens(int(i)) for i in top])
45
+ # ['bank', 'banks', 'side', 'water', ',', 'bed', 'shore', 'edge']
46
+ ```
47
+
48
+ `trust_remote_code=True` is required: the target-span pooling is a custom module
49
+ shipped in this repo as `modeling_lexsplade.py`.
50
+
51
+ Keep the marked word inside `max_seq_length`. If the markers are truncated away
52
+ the pooling falls back to the whole sentence and silently returns a *sentence*
53
+ embedding.
54
+
55
+ ## Training
56
+
57
+ | | |
58
+ |---|---|
59
+ | base model | `google-bert/bert-large-uncased` |
60
+ | training data | 31,914 same-lemma word-in-context pairs (LLM-verified silver + MCL-WiC train) |
61
+ | objective | AnglE loss on the sparse target-word vectors, with a SPLADE FLOPS regularizer and a Barlow Twins term on the dense span representation |
62
+ | seed | 42 |
63
+
64
+ ## Evaluation
65
+
66
+ MCL-WiC, English. AP is average precision; accuracy is at the best threshold.
67
+
68
+ | split | accuracy | AP |
69
+ |---|---|---|
70
+ | test | 91.20 | 96.30 |
71
+ | dev | 90.40 | 95.24 |
72
+
73
+ Mean nonzero dimensions per usage: **307** of 30,524.
74
+
75
+ ## Variants
76
+
77
+ - **`jadermcs/lexsplade-bert-large-silver`** (this model) — full 30,524-dim output.
78
+ - **`jadermcs/lexsplade-bert-large-silver-ldv`** — identical training, but the
79
+ output is restricted to 1,897 dimensions drawn from the Longman Defining
80
+ Vocabulary, so every vector reads as a definition in basic English.
config.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_cross_attention": false,
3
+ "architectures": [
4
+ "BertForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": null,
8
+ "classifier_dropout": null,
9
+ "dtype": "float32",
10
+ "eos_token_id": null,
11
+ "gradient_checkpointing": false,
12
+ "hidden_act": "gelu",
13
+ "hidden_dropout_prob": 0.1,
14
+ "hidden_size": 1024,
15
+ "initializer_range": 0.02,
16
+ "intermediate_size": 4096,
17
+ "is_decoder": false,
18
+ "layer_norm_eps": 1e-12,
19
+ "max_position_embeddings": 512,
20
+ "model_type": "bert",
21
+ "num_attention_heads": 16,
22
+ "num_hidden_layers": 24,
23
+ "output_hidden_states": true,
24
+ "pad_token_id": 0,
25
+ "position_embedding_type": "absolute",
26
+ "tie_word_embeddings": true,
27
+ "transformers_version": "5.14.1",
28
+ "type_vocab_size": 2,
29
+ "use_cache": true,
30
+ "vocab_size": 30524
31
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "pytorch": "2.13.0+cu130",
4
+ "sentence_transformers": "5.6.1",
5
+ "transformers": "5.14.1"
6
+ },
7
+ "default_prompt_name": null,
8
+ "model_type": "SparseEncoder",
9
+ "prompts": {
10
+ "document": "",
11
+ "query": ""
12
+ },
13
+ "similarity_fn_name": "cosine"
14
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8fdab7508faa9b57350a67638620f9721b4c75b051f74bb6160d42425700bc74
3
+ size 1340753168
modeling_lexsplade.py ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """LexSPLADE: target-word SPLADE pooling, loadable outside the training script.
2
+
3
+ The training script saved ``modules.json`` referring to
4
+ ``__main__.DenseHiddenStatesTransformer`` / ``__main__.TargetWordSpladePooling``,
5
+ classes that only exist while that script is running -- so the checkpoint could
6
+ not be loaded by anyone else. These are the same two modules, in an importable
7
+ file, so ``SparseEncoder(repo, trust_remote_code=True)`` works.
8
+
9
+ What the pooling does: a sentence carries its target word wrapped in ``<t>`` /
10
+ ``</t>``, and the SPLADE activations are pooled over **only the tokens between
11
+ those markers** rather than the whole sentence. The result is a sparse vector
12
+ over the MLM vocabulary describing *that word in that context* -- a word
13
+ embedding, not a sentence embedding.
14
+
15
+ ``allowed_token_ids``, when present, is a hard restriction of the output space:
16
+ every dimension outside the set is zeroed before the vector is returned. The
17
+ LDV variant of this model is trained that way and is **only correct with the
18
+ restriction applied**, which is why the ids live in the pooling config rather
19
+ than in a README instruction.
20
+ """
21
+
22
+ from __future__ import annotations
23
+
24
+ import torch
25
+ from sentence_transformers.sparse_encoder import modules
26
+
27
+
28
+ class DenseHiddenStatesTransformer(modules.Transformer):
29
+ """Fill-mask transformer that also exposes the last hidden layer.
30
+
31
+ The SPLADE head surfaces only the vocabulary-sized MLM logits as
32
+ ``token_embeddings``. Training additionally needed the dense hidden states
33
+ for a Barlow Twins term; they are kept here so the module signature matches
34
+ the saved checkpoint. Inference does not depend on them.
35
+ """
36
+
37
+ def forward(self, features: dict[str, torch.Tensor], **kwargs) -> dict[str, torch.Tensor]:
38
+ features = super().forward(features, **kwargs)
39
+ all_layers = features.pop("all_layer_embeddings", None)
40
+ if all_layers is not None:
41
+ features["dense_token_embeddings"] = all_layers[-1]
42
+ return features
43
+
44
+
45
+ class TargetWordSpladePooling(modules.SpladePooling):
46
+ """SPLADE pooling restricted to the span between ``<t>`` and ``</t>``."""
47
+
48
+ config_keys = modules.SpladePooling.config_keys + [
49
+ "m1_tok_id", "m2_tok_id", "allowed_token_ids",
50
+ ]
51
+
52
+ def __init__(
53
+ self,
54
+ *args,
55
+ m1_tok_id: int = 30522,
56
+ m2_tok_id: int = 30523,
57
+ allowed_token_ids: list[int] | None = None,
58
+ **kwargs,
59
+ ):
60
+ super().__init__(*args, **kwargs)
61
+ self.m1_tok_id = m1_tok_id
62
+ self.m2_tok_id = m2_tok_id
63
+ self.allowed_token_ids = (
64
+ sorted(set(allowed_token_ids)) if allowed_token_ids else None
65
+ )
66
+ self._vocab_mask = None
67
+
68
+ def _vocab_mask_for(self, embeddings: torch.Tensor) -> torch.Tensor:
69
+ if (
70
+ self._vocab_mask is None
71
+ or self._vocab_mask.shape[0] != embeddings.shape[1]
72
+ or self._vocab_mask.device != embeddings.device
73
+ ):
74
+ mask = torch.zeros(
75
+ embeddings.shape[1], dtype=torch.bool, device=embeddings.device
76
+ )
77
+ mask[torch.tensor(self.allowed_token_ids, device=embeddings.device)] = True
78
+ self._vocab_mask = mask
79
+ return self._vocab_mask
80
+
81
+ def forward(self, features: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:
82
+ input_ids = features["input_ids"]
83
+ attention_mask = features["attention_mask"]
84
+
85
+ is_open = input_ids == self.m1_tok_id
86
+ is_close = input_ids == self.m2_tok_id
87
+ after_open = is_open.cumsum(dim=1) > 0
88
+ before_close = is_close.cumsum(dim=1) == 0
89
+ target_mask = after_open & before_close & ~is_open & attention_mask.bool()
90
+
91
+ # Markers truncated away: pool the whole sentence rather than return an
92
+ # empty vector. Note this silently yields a *sentence* embedding, so
93
+ # keep the target inside max_seq_length.
94
+ empty = target_mask.sum(dim=1) == 0
95
+ if empty.any():
96
+ target_mask[empty] = attention_mask[empty].bool()
97
+
98
+ original_mask = features["attention_mask"]
99
+ features["attention_mask"] = target_mask.to(attention_mask.dtype)
100
+
101
+ dense_tokens = features.get("dense_token_embeddings")
102
+ if dense_tokens is not None:
103
+ m = target_mask.to(dense_tokens.dtype).unsqueeze(-1)
104
+ features["dense_sentence_embedding"] = (
105
+ (dense_tokens * m).sum(dim=1) / m.sum(dim=1).clamp(min=1.0)
106
+ )
107
+
108
+ features = super().forward(features)
109
+ features["attention_mask"] = original_mask
110
+
111
+ # masked_fill rather than a multiply: `max` pooling starts at -inf and
112
+ # -inf * 0 is NaN.
113
+ if self.allowed_token_ids is not None:
114
+ emb = features["sentence_embedding"]
115
+ features["sentence_embedding"] = emb.masked_fill(
116
+ ~self._vocab_mask_for(emb), 0.0
117
+ )
118
+ return features
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "modeling_lexsplade.DenseHiddenStatesTransformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_TargetWordSpladePooling",
12
+ "type": "modeling_lexsplade.TargetWordSpladePooling"
13
+ }
14
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "transformer_task": "fill-mask",
3
+ "modality_config": {
4
+ "text": {
5
+ "method": "forward",
6
+ "method_output_name": "logits"
7
+ }
8
+ },
9
+ "module_output_name": "token_embeddings"
10
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "cls_token": "[CLS]",
4
+ "do_lower_case": true,
5
+ "extra_special_tokens": [
6
+ "<t>",
7
+ "</t>"
8
+ ],
9
+ "is_local": true,
10
+ "local_files_only": false,
11
+ "mask_token": "[MASK]",
12
+ "max_length": 64,
13
+ "model_max_length": 64,
14
+ "pad_to_multiple_of": null,
15
+ "pad_token": "[PAD]",
16
+ "pad_token_type_id": 0,
17
+ "padding_side": "right",
18
+ "sep_token": "[SEP]",
19
+ "stride": 0,
20
+ "strip_accents": null,
21
+ "tokenize_chinese_chars": true,
22
+ "tokenizer_class": "BertTokenizer",
23
+ "truncation_side": "right",
24
+ "truncation_strategy": "longest_first",
25
+ "unk_token": "[UNK]"
26
+ }