tomaarsen HF Staff commited on
Commit
78bae81
·
verified ·
1 Parent(s): d87a048

Upload 50/50 merged retrieval model and model card

Browse files
1_SpladePooling/config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "pooling_strategy": "max",
3
+ "activation_function": "relu",
4
+ "embedding_dimension": 30522
5
+ }
README.md ADDED
@@ -0,0 +1,169 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - sentence-transformers
4
+ - sparse-encoder
5
+ - sparse
6
+ - splade
7
+ pipeline_tag: feature-extraction
8
+ library_name: sentence-transformers
9
+ language: en
10
+ license: cc-by-nc-sa-4.0
11
+ base_model:
12
+ - naver/splade-v3
13
+ - naver/splade-cocondenser-selfdistil
14
+ base_model_relation: merge
15
+ ---
16
+
17
+ # merged-SPLADE-v3-selfdistil
18
+
19
+ This is a [Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model created by a 50/50 linear merge of [SPLADE-v3](https://huggingface.co/naver/splade-v3) and [SPLADE CoCondenser SelfDistil](https://huggingface.co/naver/splade-cocondenser-selfdistil), using the [`SparseEncoder.merge`](https://www.sbert.net/docs/package_reference/sparse_encoder/model.html#sentence_transformers.sparse_encoder.model.SparseEncoder.merge) method from the [sentence-transformers](https://www.SBERT.net) library. It maps inputs to 30,522-dimensional sparse lexical vectors using SPLADE max pooling, for semantic search with learned query and document expansion.
20
+
21
+ The merged model scores above the average of its two parents on **9 of the 13 NanoBEIR datasets**, including **8 where it beats both**. It achieves a mean nDCG@10 of **0.6401**, compared with **0.6336** for SPLADE-v3 and **0.6270** for SPLADE CoCondenser SelfDistil, using the same inference settings for all three models.
22
+
23
+ Excluding MS MARCO, the merged model achieves a mean nDCG@10 of **0.6372**, compared with **0.6281** for SPLADE-v3.
24
+
25
+ See [Model Creation](#model-creation) for the full generation script.
26
+
27
+ ## Model Details
28
+
29
+ ### Model Description
30
+
31
+ - **Model Type:** Sparse Encoder
32
+ - **Base models:** [SPLADE-v3](https://huggingface.co/naver/splade-v3) and [SPLADE CoCondenser SelfDistil](https://huggingface.co/naver/splade-cocondenser-selfdistil)
33
+ - **Maximum Sequence Length:** 512 tokens
34
+ - **Output Dimensionality:** 30522 dimensions
35
+ - **Similarity Function:** Dot Product
36
+ - **Supported Modality:** Text
37
+ - **Language:** English
38
+ - **License:** CC-BY-NC-SA-4.0
39
+
40
+ ### Model Sources
41
+
42
+ - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
43
+ - **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
44
+ - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
45
+ - **Parent model:** [SPLADE-v3](https://huggingface.co/naver/splade-v3)
46
+ - **Parent model:** [SPLADE CoCondenser SelfDistil](https://huggingface.co/naver/splade-cocondenser-selfdistil)
47
+
48
+ ### Full Model Architecture
49
+
50
+ ```
51
+ SparseEncoder(
52
+ (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertForMaskedLM'})
53
+ (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 30522})
54
+ )
55
+ ```
56
+
57
+ ## Usage
58
+
59
+ ### Direct Usage (Sentence Transformers)
60
+
61
+ First install the Sentence Transformers library:
62
+
63
+ ```bash
64
+ pip install -U sentence-transformers
65
+ ```
66
+
67
+ Then you can load this model and run inference.
68
+
69
+ ```python
70
+ from sentence_transformers import SparseEncoder
71
+
72
+ model = SparseEncoder("sparse-encoder/merged-SPLADE-v3-selfdistil")
73
+ queries = ['Which planet is known as the Red Planet?']
74
+ documents = [
75
+ "Venus is often called Earth's twin because of its similar size and proximity.",
76
+ 'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
77
+ 'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
78
+ ]
79
+ query_embeddings = model.encode_query(queries)
80
+ document_embeddings = model.encode_document(documents)
81
+ print(query_embeddings.shape, document_embeddings.shape)
82
+ # torch.Size([1, 30522]) torch.Size([3, 30522])
83
+
84
+ similarities = model.similarity(query_embeddings, document_embeddings)
85
+ print(similarities)
86
+ # tensor([[ 9.5819, 23.5003, 21.2222]])
87
+ ```
88
+
89
+ ## Evaluation
90
+
91
+ ### Metrics
92
+
93
+ #### NanoBEIR
94
+
95
+ * Dataset: [NanoBEIR-en](https://huggingface.co/datasets/sentence-transformers/NanoBEIR-en)
96
+ * Evaluated with `SparseNanoBEIREvaluator` on all 13 datasets.
97
+ * Maximum input length: 512 tokens. Query and document prompts: empty.
98
+ * Similarity: dot product. SPLADE max pooling with log(1 + ReLU(logits)). FP32 inference with eager attention.
99
+ * Metric: nDCG@10. The mean is unweighted across datasets.
100
+
101
+ The parent models were evaluated with the same prompts and length limits.
102
+
103
+ | Dataset | merged-SPLADE-v3-selfdistil | SPLADE-v3 | SPLADE CoCondenser SelfDistil |
104
+ | --- | --- | --- | --- |
105
+ | msmarco | 0.674565 | <u>0.700087</u> | 0.659201 |
106
+ | nq | <u>0.731771</u> | 0.727694 | 0.694880 |
107
+ | fiqa2018 | <u>0.518832</u> | 0.514431 | 0.515037 |
108
+ | nfcorpus | 0.373541 | <u>0.386574</u> | 0.373691 |
109
+ | scidocs | <u>0.363320</u> | 0.362581 | 0.345038 |
110
+ | arguana | 0.476651 | <u>0.488462</u> | 0.452402 |
111
+ | dbpedia | 0.680222 | <u>0.687173</u> | 0.680651 |
112
+ | quoraretrieval | <u>0.964132</u> | 0.907928 | 0.954648 |
113
+ | touche2020 | <u>0.625124</u> | 0.612912 | 0.583962 |
114
+ | climatefever | 0.326903 | 0.310448 | <u>0.346635</u> |
115
+ | fever | <u>0.938595</u> | 0.913058 | 0.927825 |
116
+ | hotpotqa | <u>0.842994</u> | 0.828082 | 0.838331 |
117
+ | scifact | <u>0.804576</u> | 0.797861 | 0.778772 |
118
+ | **mean** | **<u>0.640094</u>** | **0.633638** | **0.627006** |
119
+
120
+ ## Model Creation
121
+
122
+ The complete BERT backbone and masked-language-model prediction head were merged in fp32 with equal weights, without any additional training. Both parents use the same 30,522-token vocabulary, SPLADE max pooling, and 512-token input limit.
123
+
124
+ This is all it took:
125
+
126
+ ```python
127
+ # pip install "sentence-transformers[merge]"
128
+ from sentence_transformers import SparseEncoder
129
+
130
+ models = [
131
+ SparseEncoder("naver/splade-v3"),
132
+ SparseEncoder("naver/splade-cocondenser-selfdistil"),
133
+ ]
134
+ model = SparseEncoder.merge(models, method="linear", parameters={"weight": [0.5, 0.5]}, dtype="float32")
135
+ model[1].embedding_dimension = model[0].model.config.vocab_size
136
+ model.save_pretrained("merged-SPLADE-v3-selfdistil")
137
+ ```
138
+
139
+ ### Framework Versions
140
+
141
+ - Python: 3.12.3
142
+ - Sentence Transformers: 6.2.0
143
+ - Transformers: 5.4.0
144
+ - PyTorch: 2.11.0+cu128
145
+ - Mergekit: 0.1.5
146
+ - Datasets: 4.8.5
147
+ - Tokenizers: 0.22.2
148
+
149
+ ## Citation
150
+
151
+ ### BibTeX
152
+
153
+ ### Mergekit
154
+
155
+ ```bibtex
156
+ @inproceedings{goddard-etal-2024-arcees,
157
+ title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",
158
+ author = "Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vladimir and Benedict, Brian and McQuade, Mark and Solawetz, Jacob",
159
+ editor = "Dernoncourt, Franck and Preo{\c{t}}iuc-Pietro, Daniel and Shimorina, Anastasia",
160
+ booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
161
+ month = nov,
162
+ year = "2024",
163
+ address = "Miami, Florida, US",
164
+ publisher = "Association for Computational Linguistics",
165
+ url = "https://aclanthology.org/2024.emnlp-industry.36",
166
+ doi = "10.18653/v1/2024.emnlp-industry.36",
167
+ pages = "477--485",
168
+ }
169
+ ```
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_cross_attention": false,
3
+ "architectures": [
4
+ "BertForMaskedLM"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": null,
8
+ "classifier_dropout": null,
9
+ "dtype": "float32",
10
+ "eos_token_id": null,
11
+ "gradient_checkpointing": false,
12
+ "hidden_act": "gelu",
13
+ "hidden_dropout_prob": 0.1,
14
+ "hidden_size": 768,
15
+ "initializer_range": 0.02,
16
+ "intermediate_size": 3072,
17
+ "is_decoder": false,
18
+ "layer_norm_eps": 1e-12,
19
+ "max_position_embeddings": 512,
20
+ "model_type": "bert",
21
+ "num_attention_heads": 12,
22
+ "num_hidden_layers": 12,
23
+ "pad_token_id": 0,
24
+ "position_embedding_type": "absolute",
25
+ "tie_word_embeddings": true,
26
+ "transformers_version": "5.4.0",
27
+ "type_vocab_size": 2,
28
+ "use_cache": false,
29
+ "vocab_size": 30522
30
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "pytorch": "2.11.0+cu128",
4
+ "sentence_transformers": "6.2.0",
5
+ "transformers": "5.4.0"
6
+ },
7
+ "default_prompt_name": null,
8
+ "model_type": "SparseEncoder",
9
+ "prompts": {
10
+ "document": "",
11
+ "query": ""
12
+ },
13
+ "similarity_fn_name": "dot"
14
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:77b4e0b569f54fdc7bf151bb6c32950b4db5977902adf69f380102e2ec70724f
3
+ size 438080872
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.sparse_encoder.modules.mlm_transformer.MLMTransformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_SpladePooling",
12
+ "type": "sentence_transformers.sparse_encoder.modules.splade_pooling.SpladePooling"
13
+ }
14
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "transformer_task": "fill-mask",
3
+ "modality_config": {
4
+ "text": {
5
+ "method": "forward",
6
+ "method_output_name": "logits"
7
+ }
8
+ },
9
+ "module_output_name": "token_embeddings"
10
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "clean_up_tokenization_spaces": true,
4
+ "cls_token": "[CLS]",
5
+ "do_basic_tokenize": true,
6
+ "do_lower_case": true,
7
+ "is_local": true,
8
+ "mask_token": "[MASK]",
9
+ "max_length": 128,
10
+ "model_max_length": 512,
11
+ "never_split": null,
12
+ "pad_to_multiple_of": null,
13
+ "pad_token": "[PAD]",
14
+ "pad_token_type_id": 0,
15
+ "padding_side": "right",
16
+ "sep_token": "[SEP]",
17
+ "stride": 0,
18
+ "strip_accents": null,
19
+ "tokenize_chinese_chars": true,
20
+ "tokenizer_class": "BertTokenizer",
21
+ "truncation_side": "right",
22
+ "truncation_strategy": "longest_first",
23
+ "unk_token": "[UNK]"
24
+ }