Feature Extraction
Model2Vec
Safetensors
sentence-transformers
English
embeddings
static-embeddings
sentence-similarity
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ figures/mteb_vs_beir.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -9,9 +9,11 @@ tags:
9
  language: en
10
  ---
11
 
12
- # blink-8M
 
 
13
 
14
- blink-8M is a fast English static embedding model: a single 30,000 × 256 token embedding table (7.7M parameters, 15MB in float16) with mean pooling. It has no transformer layers, so it runs on any CPU.
15
 
16
  ## Usage
17
 
@@ -24,6 +26,29 @@ model = StaticModel.from_pretrained("minishlab/blink-8M")
24
  embeddings = model.encode(["a first sentence", "a second sentence"])
25
  ```
26
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
  With [Sentence Transformers](https://www.sbert.net):
28
 
29
  ```python
@@ -35,7 +60,22 @@ embeddings = model.encode(["a first sentence", "a second sentence"])
35
 
36
  ## Results
37
 
38
- MTEB(eng, v2), all 41 tasks (mteb 2.27.1), every model at 256 dimensions. blink-8M runs through mteb's model2vec loader; the other models use their first 256 dimensions with normalized embeddings:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
 
40
  | Model | Mean (task) | Retrieval | Reranking | PairCls | Cls | Clust | STS | Summ |
41
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
@@ -47,17 +87,250 @@ MTEB(eng, v2), all 41 tasks (mteb 2.27.1), every model at 256 dimensions. blink-
47
  | potion-base-8M | 53.30 | 31.22 | 41.79 | 76.62 | 70.41 | 39.88 | 72.45 | **25.14** |
48
  | static-retrieval-mrl-en-v1 | 52.00 | 33.87 | 41.57 | 74.68 | 63.16 | 39.98 | 71.35 | 18.95 |
49
 
50
- Retrieval on LightOn's decontaminated BEIR (12 datasets, nDCG@10), every model at 256 dimensions:
51
-
52
- | Model | BEIR |
53
- |---|---:|
54
- | **blink-8M** | **46.72** |
55
- | lattice-retrieval | 46.26 |
56
- | bekko-static-embedding-v1 | 45.47 |
57
- | potion-retrieval-32M | 43.67 |
58
- | static-retrieval-mrl-en-v1 | 42.14 |
59
- | potion-base-32M | 40.35 |
60
- | potion-base-8M | 39.96 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
61
 
62
  ## Training
63
 
 
9
  language: en
10
  ---
11
 
12
+ <div align="center">
13
+ <img src="figures/sparkle_2_blink.svg" alt="blink logo" width="325">
14
+ </div>
15
 
16
+ blink-8M is a fast English static embedding model by [Minish](minish.ai): a single 30,000 × 256 token embedding table (7.7M parameters, 15MB in float16) with mean pooling. It has no transformer layers, so it runs on any CPU.
17
 
18
  ## Usage
19
 
 
26
  embeddings = model.encode(["a first sentence", "a second sentence"])
27
  ```
28
 
29
+ With [model2vec-rs](https://github.com/MinishLab/model2vec-rs)
30
+
31
+ ```rust
32
+ use anyhow::Result;
33
+ use model2vec_rs::model::StaticModel;
34
+
35
+ fn main() -> Result<()> {
36
+ let model = StaticModel::from_pretrained(
37
+ "minishlab/blink-8M",
38
+ )?;
39
+
40
+ let sentences = vec![
41
+ "Hello world".to_string(),
42
+ "Rust is awesome".to_string(),
43
+ ];
44
+
45
+ let embeddings = model.encode(&sentences);
46
+ println!("Generated {} embeddings.", embeddings.len());
47
+
48
+ Ok(())
49
+ }
50
+ ```
51
+
52
  With [Sentence Transformers](https://www.sbert.net):
53
 
54
  ```python
 
60
 
61
  ## Results
62
 
63
+ We compare blink-8M to six other static embedding models:
64
+
65
+ - [potion-base-8M](https://huggingface.co/minishlab/potion-base-8M): MinishLab's general-purpose model2vec model, 30k vocab, 256 dimensions.
66
+ - [potion-base-32M](https://huggingface.co/minishlab/potion-base-32M): a larger version of potion-base-8M, 65k vocab, 512 dimensions.
67
+ - [potion-retrieval-32M](https://huggingface.co/minishlab/potion-retrieval-32M): potion-base-32M fine-tuned for retrieval, 65k vocab, 512 dimensions.
68
+ - [static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1): Sentence Transformers' static retrieval model, 30k vocab, 1024 dimensions.
69
+ - [bekko-static-embedding-v1](hotchpotch/bekko-static-embedding-v1): A multilingual model trained on retrieval data, 200k vocab, 1024 dimensions.
70
+ - [lattice-retrieval](erikkaum/lattice-retrieval): A monolingual model, trained on retrieval data, 30k vocab, 1024 dimensions.
71
+
72
+ Every model is evaluated at 256 dimensions. potion-base-32M, potion-retrieval-32M and static-retrieval-mrl-en-v1 are truncated to their first 256 dimensions and use L2 normalized embeddings, even if the original model did not have normalization. Applying normalization uniformly improved performance across all models.
73
+
74
+ ![MTEB mean against BEIR nDCG@10 for all models](figures/mteb_vs_beir.png)
75
+
76
+ ### MTEB
77
+
78
+ MTEB(eng, v2), all 41 tasks (mteb 2.27.1).
79
 
80
  | Model | Mean (task) | Retrieval | Reranking | PairCls | Cls | Clust | STS | Summ |
81
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
 
87
  | potion-base-8M | 53.30 | 31.22 | 41.79 | 76.62 | 70.41 | 39.88 | 72.45 | **25.14** |
88
  | static-retrieval-mrl-en-v1 | 52.00 | 33.87 | 41.57 | 74.68 | 63.16 | 39.98 | 71.35 | 18.95 |
89
 
90
+ #### Summary
91
+
92
+ - **Mean:** blink-8M has the highest mean (55.34), just ahead of lattice (55.24) and about 1 point ahead of bekko (54.38).
93
+ - **Retrieval-type tasks:** blink-8M is best on retrieval, reranking and pair classification. On retrieval it scores 38.86, against 38.01 for lattice and 31.22 for potion-base-8M. Note that this task has significant overlap with BEIR, see below.
94
+ - **Other tasks:** the potion-base models are best on classification (70.48 and 70.41) and potion-base-8M on summarization (25.14), with blink-8M at 67.88 and 22.44. Bekko is best on STS and lattice on clustering.
95
+ - **Against potion-base-8M:** at roughly the same size, blink-8M is 2.04 points higher on the mean and 7.64 points higher on retrieval, but 2.53 points lower on classification.
96
+
97
+ ### RTEB
98
+
99
+ [RTEB](https://huggingface.co/blog/rteb) (`RTEB(eng, beta)` in MTEB): retrieval across specialized domains, namely legal, finance, healthcare and code. We use the 15 public English tasks, including only the English subset of CUREv1. The private tasks are left out because their data is not available. All scores are nDCG@10, and the last row is the mean over the 15 tasks.
100
+
101
+ | Task | Domain | blink-8M | bekko-static-embedding-v1 | lattice-retrieval | potion-retrieval-32M | static-retrieval-mrl-en-v1 | potion-base-32M | potion-base-8M |
102
+ |---|---|---:|---:|---:|---:|---:|---:|---:|
103
+ | AILACasedocs | Legal | 26.81 | 27.03 | **27.56** | 21.97 | 24.21 | 25.02 | 27.17 |
104
+ | AILAStatutes | Legal | **25.10** | 22.67 | 20.02 | 18.76 | 20.42 | 15.62 | 15.06 |
105
+ | LegalSummarization | Legal | **58.55** | 58.19 | 57.99 | 54.95 | 52.91 | 57.02 | 55.35 |
106
+ | FinanceBench | Finance | 34.76 | **37.21** | 35.55 | 31.96 | 35.51 | 27.30 | 27.88 |
107
+ | HC3Finance | Finance | **33.21** | 27.08 | 32.96 | 26.96 | 25.11 | 25.67 | 24.76 |
108
+ | FinQA | Finance | **50.80** | 48.22 | 45.96 | 45.16 | 44.41 | 38.77 | 38.09 |
109
+ | ChatDoctor | Healthcare | 25.12 | **26.88** | 26.02 | 23.54 | 23.18 | 18.52 | 17.16 |
110
+ | CUREv1 (en) | Healthcare | **33.18** | 32.80 | 32.77 | 29.71 | 27.56 | 28.17 | 26.93 |
111
+ | Apps | Code | 3.60 | **8.54** | 4.16 | 3.90 | 4.08 | 3.22 | 2.86 |
112
+ | DS1000 | Code | 23.95 | **23.97** | 22.96 | 21.70 | 21.75 | 18.66 | 17.36 |
113
+ | HumanEval | Code | 45.67 | **49.14** | 42.69 | 40.73 | 40.29 | 38.68 | 39.13 |
114
+ | MBPP | Code | 37.26 | **41.84** | 34.00 | 24.85 | 25.19 | 25.51 | 27.95 |
115
+ | WikiSQL | Code | **37.29** | 25.84 | 31.62 | 32.18 | 29.90 | 21.94 | 21.81 |
116
+ | FreshStack | Code | **22.73** | 21.45 | 21.32 | 19.22 | 18.88 | 18.13 | 18.20 |
117
+ | SWE-bench | Code | **4.57** | 3.89 | 4.04 | 2.94 | 2.76 | 2.73 | 2.74 |
118
+ | **Mean** | | **30.84** | 30.32 | 29.31 | 26.57 | 26.41 | 24.33 | 24.16 |
119
+
120
+ Note that FreshStack is also present as a separate eval, see below, although there it uses Recall@50 and α-nDCG@10.
121
+
122
+ #### Summary
123
+
124
+ - **Mean:** blink-8M has the highest mean (30.84), but bekko is close (30.32). Blink-8M is best on 8 of 15 tasks, bekko on 6 and lattice on 1 (AILACasedocs).
125
+ - **By domain:** blink-8M leads on legal (36.82 against 35.96 for bekko) and finance (39.59 against 37.50). Bekko leads on healthcare (29.84 against 29.15), and code is a tie (25.01 against 24.95).
126
+ - **Code:** bekko is best on the code-generation tasks (Apps, HumanEval, MBPP and DS1000, the last by 0.02), while blink-8M has the largest win of the table on WikiSQL (37.29 against at most 32.18).
127
+ - **Overall:** Apps and SWE-bench are hard for every model, with nDCG@10 below 9.
128
+
129
+ ### BrowseComp-Plus
130
+
131
+ [BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus): 830 hard, multi-hop BrowseComp questions over a fixed corpus of 100,195 web documents. Queries are the original questions, and documents are embedded in full, without truncation. We report two sets of relevance labels: *evidence* documents (all documents needed to answer the question) and *gold* documents (the documents that contain the answer).
132
+
133
+ **Evidence**
134
+
135
+ | Model | nDCG@10 | Recall@5 | Recall@100 | Recall@1000 |
136
+ |---|---:|---:|---:|---:|
137
+ | **blink-8M** | **2.13** | 1.47 | 9.30 | 32.31 |
138
+ | bekko-static-embedding-v1 | 2.09 | **1.57** | **11.13** | **36.46** |
139
+ | lattice-retrieval | 1.88 | 1.44 | 9.05 | 31.44 |
140
+ | potion-retrieval-32M | 1.73 | 1.20 | 9.32 | 32.96 |
141
+ | static-retrieval-mrl-en-v1 | 1.42 | 1.03 | 8.09 | 29.80 |
142
+ | potion-base-32M | 1.32 | 0.86 | 7.43 | 25.49 |
143
+ | potion-base-8M | 1.25 | 0.98 | 6.93 | 24.58 |
144
+
145
+ **Gold**
146
+
147
+ | Model | nDCG@10 | Recall@5 | Recall@100 | Recall@1000 |
148
+ |---|---:|---:|---:|---:|
149
+ | **blink-8M** | **1.78** | **1.89** | 10.16 | 34.65 |
150
+ | bekko-static-embedding-v1 | 1.74 | 1.82 | **12.11** | **37.93** |
151
+ | lattice-retrieval | 1.58 | 1.81 | 9.82 | 33.23 |
152
+ | potion-retrieval-32M | 1.44 | 1.29 | 9.16 | 34.38 |
153
+ | static-retrieval-mrl-en-v1 | 1.23 | 1.34 | 8.62 | 29.86 |
154
+ | potion-base-32M | 1.01 | 0.89 | 7.80 | 24.33 |
155
+ | potion-base-8M | 0.88 | 0.78 | 7.06 | 23.79 |
156
+
157
+ Blink has the best nDCG@10 on both label sets and the best gold Recall@5. Bekko has the best evidence Recall@5 and the best Recall@100 and Recall@1000 on both label sets.
158
+
159
+ ### BEIR
160
+
161
+ We evaluate on LightOn's decontaminated version of BEIR (`lightonai/<dataset>-decontaminated`). These are 12 BEIR datasets with every query and document found in the [mgte-en](https://huggingface.co/datasets/Alibaba-NLP/mgte-en) pre-training data removed. Contamination was found in two passes: exact matches on normalized text hashes, and 13-gram containment ≥ 0.5. Relevance judgments that point to a removed query or document were dropped as well.
162
+
163
+ We use the test split and only keep queries that have at least one relevant document left. Documents are encoded as `title + text`, and retrieval uses cosine similarity over the full corpus. We report nDCG@10 (ranking quality in the top 10) and Recall@100 (fraction of relevant documents found in the top 100). Following LightOn and [Lattice](https://github.com/ErikKaum/lattice/blob/main/evals.md), we leave out FEVER and ClimateFEVER.
164
+
165
+ | Dataset | Queries | Documents |
166
+ |---|---:|---:|
167
+ | ArguAna | 1,373 | 8,546 |
168
+ | DBPedia | 264 | 1,678,309 |
169
+ | FiQA | 61 | 47,617 |
170
+ | HotpotQA | 2,092 | 2,314,813 |
171
+ | MS MARCO | 41 | 4,036,967 |
172
+ | NFCorpus | 169 | 912 |
173
+ | NQ | 26 | 305,674 |
174
+ | Quora | 2,788 | 413,157 |
175
+ | SCIDOCS | 130 | 5,833 |
176
+ | SciFact | 41 | 858 |
177
+ | TREC-COVID | 50 | 99,522 |
178
+ | Touché-2020 | 30 | 378,223 |
179
+
180
+ #### nDCG@10
181
+
182
+ | Dataset | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-retrieval-32M | static-retrieval-mrl-en-v1 | potion-base-32M | potion-base-8M |
183
+ |---|---:|---:|---:|---:|---:|---:|---:|
184
+ | ArguAna | 33.19 | 33.56 | **33.73** | 32.72 | 32.12 | 31.19 | 31.24 |
185
+ | DBPedia | 25.87 | **28.80** | 27.45 | 27.10 | 26.10 | 25.77 | 24.67 |
186
+ | FiQA | **31.58** | 28.35 | 27.60 | 27.04 | 27.84 | 25.28 | 24.14 |
187
+ | HotpotQA | **52.68** | 50.71 | 49.55 | 47.44 | 47.45 | 42.08 | 40.54 |
188
+ | MS MARCO | **51.75** | 48.63 | 48.00 | 50.32 | 47.64 | 43.68 | 40.62 |
189
+ | NFCorpus | **21.13** | 20.33 | 21.12 | 20.68 | 17.56 | 17.32 | 15.43 |
190
+ | NQ | 67.40 | **70.31** | 64.15 | 70.24 | 63.05 | 60.36 | 63.97 |
191
+ | Quora | 85.58 | 84.99 | **86.03** | 81.69 | 80.42 | 84.39 | 83.70 |
192
+ | SCIDOCS | 17.66 | **18.45** | 17.10 | 14.65 | 12.62 | 13.65 | 13.27 |
193
+ | SciFact | **83.39** | 80.67 | 81.29 | 78.61 | 79.25 | 74.76 | 72.68 |
194
+ | TREC-COVID | **68.04** | 66.96 | 66.14 | 50.71 | 47.74 | 45.16 | 48.85 |
195
+ | Touché-2020 | 22.07 | 23.35 | 23.49 | 22.99 | **23.85** | 20.51 | 20.34 |
196
+ | **Mean** | **46.70** | 46.26 | 45.47 | 43.68 | 42.14 | 40.35 | 39.95 |
197
+
198
+ #### Recall@100
199
+
200
+ | Dataset | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-retrieval-32M | static-retrieval-mrl-en-v1 | potion-base-32M | potion-base-8M |
201
+ |---|---:|---:|---:|---:|---:|---:|---:|
202
+ | ArguAna | 95.41 | 95.48 | **95.92** | 95.70 | 95.27 | 92.94 | 92.43 |
203
+ | DBPedia | 47.21 | 49.88 | **50.15** | 48.21 | 48.09 | 44.70 | 42.06 |
204
+ | FiQA | 65.57 | 60.40 | **66.86** | 61.51 | 59.35 | 57.13 | 58.84 |
205
+ | HotpotQA | **74.07** | 72.94 | 73.37 | 70.24 | 70.72 | 65.46 | 63.15 |
206
+ | MS MARCO | 48.58 | 48.92 | 48.61 | 50.41 | **50.78** | 42.56 | 37.95 |
207
+ | NFCorpus | 43.83 | 43.34 | **44.78** | 40.85 | 38.89 | 38.64 | 35.59 |
208
+ | NQ | **96.15** | 92.31 | 88.46 | **96.15** | 92.31 | 88.46 | 88.46 |
209
+ | Quora | 98.47 | 98.43 | **98.65** | 97.58 | 97.03 | 98.15 | 97.94 |
210
+ | SCIDOCS | **48.59** | 47.37 | 42.63 | 44.94 | 39.23 | 42.63 | 41.09 |
211
+ | SciFact | 95.12 | **97.56** | 95.12 | 95.12 | 95.12 | 95.12 | 95.12 |
212
+ | TREC-COVID | 14.76 | 15.44 | **15.92** | 10.05 | 9.67 | 9.91 | 9.49 |
213
+ | Touché-2020 | 51.70 | **54.47** | 53.29 | 53.42 | 51.01 | 50.78 | 48.53 |
214
+ | **Mean** | **64.96** | 64.71 | 64.48 | 63.68 | 62.29 | 60.54 | 59.22 |
215
+
216
+ #### Summary
217
+
218
+ - **nDCG@10:** blink-8M has the highest mean (46.70) and is best on 6 of 12 datasets: FiQA, HotpotQA, MS MARCO, NFCorpus, SciFact and TREC-COVID. Lattice is best on 3 (DBPedia, NQ, SCIDOCS), bekko on 2 (ArguAna, Quora) and static-retrieval-mrl on 1 (Touché-2020).
219
+ - **Recall@100:** the top three are close: blink-8M 64.96, lattice 64.71, bekko 64.48. Bekko is best on 6 datasets, blink-8M on 3 (one shared with potion-retrieval-32M).
220
+ - **TREC-COVID:** blink-8M, lattice and bekko score 66–68 nDCG@10, while the potion models and static-retrieval-mrl score 45–51.
221
+
222
+ ### BRIGHT
223
+
224
+ [BRIGHT](https://huggingface.co/datasets/xlangai/BRIGHT) (Su et al., 2024) is a retrieval benchmark that needs reasoning: relevant documents often share few keywords or little surface topic with the query. For example, a query may describe a coding problem, and the relevant document is another problem that uses the same algorithm. BRIGHT has 12 tasks in three groups:
225
+
226
+ - **StackExchange** (7): Biology, Earth Science, Economics, Psychology, Robotics, Stack Overflow and Sustainable Living. Queries are user posts, and relevant documents are web pages linked from accepted answers.
227
+ - **Coding** (2): LeetCode (problems that share a solution technique) and Pony (documentation for a low-resource programming language).
228
+ - **Math** (3): AoPS (competition problems with a similar solution), TheoremQA-Q (problems that use the same theorem) and TheoremQA-T (the theorem statement needed to solve a problem).
229
+
230
+ We use the original queries, not the LLM-generated reasoning traces, and the standard (short) document split. For each query we drop the documents BRIGHT marks as excluded, then retrieve the top 1000 documents by cosine similarity. Metrics are computed with `pytrec_eval`: nDCG@10 (ranking quality in the top 10, BRIGHT's main metric) and Recall@100 (fraction of relevant documents found in the top 100).
231
+
232
+ Because these need reasoning, we expect low scores: these are all static models and can't reason. Nevertheless, it's an interesting held-out task to try.
233
+
234
+ | Task | Group | Queries | Documents |
235
+ |---|---|---:|---:|
236
+ | Biology | StackExchange | 103 | 57,359 |
237
+ | Earth Science | StackExchange | 116 | 121,249 |
238
+ | Economics | StackExchange | 103 | 50,220 |
239
+ | Psychology | StackExchange | 101 | 52,835 |
240
+ | Robotics | StackExchange | 101 | 61,961 |
241
+ | Stack Overflow | StackExchange | 117 | 107,081 |
242
+ | Sustainable Living | StackExchange | 108 | 60,792 |
243
+ | LeetCode | Coding | 142 | 413,932 |
244
+ | Pony | Coding | 112 | 7,894 |
245
+ | AoPS | Math | 111 | 188,002 |
246
+ | TheoremQA-Q | Math | 194 | 188,002 |
247
+ | TheoremQA-T | Math | 76 | 23,839 |
248
+
249
+ #### nDCG@10
250
+
251
+ | Task | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-base-32M | potion-retrieval-32M | potion-base-8M | static-retrieval-mrl-en-v1 |
252
+ |---|---:|---:|---:|---:|---:|---:|---:|
253
+ | Biology | **14.68** | 13.50 | 11.19 | 11.57 | 10.57 | 9.60 | 9.18 |
254
+ | Earth Science | **25.58** | 24.01 | 21.24 | 19.35 | 19.61 | 17.11 | 18.82 |
255
+ | Economics | **11.22** | 11.04 | 10.94 | 11.17 | 9.76 | 10.79 | 9.72 |
256
+ | Psychology | **11.93** | 11.06 | 10.11 | 8.78 | 8.78 | 7.61 | 6.24 |
257
+ | Robotics | 7.26 | 6.51 | 6.27 | 7.17 | **7.63** | 5.49 | 7.44 |
258
+ | Stack Overflow | **12.94** | 11.00 | 10.08 | 9.12 | 7.21 | 8.39 | 6.65 |
259
+ | Sustainable Living | 7.55 | 7.16 | 8.04 | **8.90** | 6.62 | 8.11 | 5.91 |
260
+ | LeetCode | **15.98** | 15.70 | 13.41 | 13.79 | 13.14 | 12.78 | 14.10 |
261
+ | Pony | **3.52** | 3.44 | 1.18 | 0.77 | 2.49 | 1.09 | 1.33 |
262
+ | AoPS | 5.78 | **6.15** | 4.49 | 4.61 | 5.59 | 4.68 | 5.14 |
263
+ | TheoremQA-Q | **7.11** | 6.76 | 5.54 | 6.23 | 6.21 | 5.97 | 5.67 |
264
+ | TheoremQA-T | **2.59** | 1.37 | 1.30 | 1.53 | 1.18 | 1.70 | 0.89 |
265
+ | **Mean** | **10.51** | 9.81 | 8.65 | 8.58 | 8.23 | 7.77 | 7.59 |
266
+
267
+ #### Recall@100
268
+
269
+ | Task | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-base-32M | potion-retrieval-32M | potion-base-8M | static-retrieval-mrl-en-v1 |
270
+ |---|---:|---:|---:|---:|---:|---:|---:|
271
+ | Biology | **50.99** | 45.07 | 44.26 | 40.37 | 39.12 | 37.50 | 37.80 |
272
+ | Earth Science | **58.10** | 55.71 | 53.50 | 47.19 | 50.62 | 47.05 | 48.91 |
273
+ | Economics | **37.89** | 36.92 | 37.18 | 34.47 | 33.88 | 33.93 | 30.39 |
274
+ | Psychology | 40.42 | 38.59 | **44.50** | 32.77 | 36.74 | 30.87 | 30.74 |
275
+ | Robotics | **31.66** | 26.84 | 26.90 | 30.27 | 27.20 | 27.33 | 23.42 |
276
+ | Stack Overflow | **40.20** | 33.54 | 34.06 | 32.45 | 29.53 | 29.58 | 25.34 |
277
+ | Sustainable Living | **45.27** | 43.28 | 42.70 | 35.89 | 42.81 | 36.64 | 36.96 |
278
+ | LeetCode | 38.73 | 40.79 | 34.08 | **42.05** | 39.38 | 41.94 | 36.29 |
279
+ | Pony | **17.03** | 10.85 | 8.81 | 9.11 | 14.30 | 8.94 | 10.81 |
280
+ | AoPS | **20.24** | 19.86 | 15.59 | 13.45 | 14.36 | 12.74 | 14.09 |
281
+ | TheoremQA-Q | 16.28 | **16.49** | 15.75 | 13.09 | 15.49 | 14.60 | 14.60 |
282
+ | TheoremQA-T | 7.68 | 7.46 | **10.53** | 6.80 | 7.24 | 6.36 | 3.73 |
283
+ | **Mean** | **33.71** | 31.28 | 30.65 | 28.16 | 29.22 | 27.29 | 26.09 |
284
+
285
+ #### Summary
286
+
287
+ - **nDCG@10:** blink-8M has the highest mean (10.51, against 9.81 for lattice) and is best on 9 of 12 tasks. The other three go to potion-retrieval-32M (Robotics), potion-base-32M (Sustainable Living) and lattice (AoPS).
288
+ - **Recall@100:** blink-8M has the highest mean (33.71, against 31.28 for lattice) and is best on 8 of 12 tasks. The largest gaps are on Pony (17.03 against at most 14.30 for the others) and Stack Overflow (40.20 against at most 34.06).
289
+
290
+ ### FreshStack
291
+
292
+ [FreshStack](https://fresh-stack.github.io/) (Thakur et al., 2025) tests retrieval on niche, recent technical topics. The queries are Stack Overflow questions, and the corpus is code and documentation chunks from the GitHub repositories of each topic, taken from an October 2024 snapshot. Each question's accepted answer is split into atomic facts ("nuggets"), and corpus chunks are labeled with the nuggets they support. A good ranking therefore has to cover all the facts needed to answer the question, not just find one relevant chunk.
293
+
294
+ We evaluate on the test split of the 5 topics. Documents are encoded as `title + text`. Queries and documents are truncated to 2048 tokens, and we retrieve the top 50 documents by cosine similarity. Metrics come from the official `freshstack` evaluation package:
295
+
296
+ - **α-nDCG@10**: nDCG over the top 10 where each document only gets credit for nuggets that higher-ranked documents have not already covered (α = 0.5). It rewards rankings that cover many different nuggets.
297
+ - **Recall@50**: the fraction of a query's relevant documents found in the top 50.
298
+
299
+ | Topic | Queries | Documents | Nuggets |
300
+ |---|---:|---:|---:|
301
+ | Angular | 129 | 117,286 | 417 |
302
+ | Godot | 99 | 25,477 | 325 |
303
+ | LangChain | 203 | 49,505 | 639 |
304
+ | Laravel | 184 | 52,341 | 552 |
305
+ | YOLO | 57 | 27,207 | 198 |
306
+
307
+ #### α-nDCG@10
308
+
309
+ | Topic | blink-8M | bekko-static-embedding-v1 | lattice-retrieval | potion-base-8M | potion-retrieval-32M | potion-base-32M | static-retrieval-mrl-en-v1 |
310
+ |---|---:|---:|---:|---:|---:|---:|---:|
311
+ | Angular | **16.89** | 16.11 | 12.88 | 15.67 | 13.00 | 12.91 | 12.90 |
312
+ | Godot | 12.60 | **14.42** | 12.01 | 10.10 | 10.55 | 9.97 | 8.91 |
313
+ | LangChain | **15.49** | 14.83 | 13.42 | 12.85 | 12.43 | 12.29 | 13.50 |
314
+ | Laravel | **21.27** | 17.11 | 15.68 | 16.61 | 16.75 | 16.36 | 15.80 |
315
+ | YOLO | **12.36** | 11.56 | 12.22 | 9.61 | 8.50 | 9.59 | 6.43 |
316
+ | **Mean** | **15.72** | 14.81 | 13.24 | 12.97 | 12.25 | 12.22 | 11.51 |
317
+
318
+ #### Recall@50
319
+
320
+ | Topic | blink-8M | bekko-static-embedding-v1 | lattice-retrieval | potion-base-8M | potion-retrieval-32M | potion-base-32M | static-retrieval-mrl-en-v1 |
321
+ |---|---:|---:|---:|---:|---:|---:|---:|
322
+ | Angular | 21.45 | 20.06 | 17.45 | **22.77** | 20.03 | 19.45 | 18.60 |
323
+ | Godot | **25.41** | 25.40 | 23.27 | 19.50 | 21.07 | 18.09 | 17.97 |
324
+ | LangChain | 21.11 | **21.34** | 20.70 | 18.78 | 18.51 | 17.52 | 18.47 |
325
+ | Laravel | **36.71** | 27.73 | 31.73 | 31.70 | 25.82 | 28.60 | 27.98 |
326
+ | YOLO | 22.28 | **23.21** | 20.88 | 21.42 | 16.92 | 21.18 | 20.65 |
327
+ | **Mean** | **25.39** | 23.55 | 22.81 | 22.83 | 20.47 | 20.97 | 20.73 |
328
+
329
+ #### Summary
330
+
331
+ - **α-nDCG@10:** blink-8M has the highest mean (15.72, against 14.81 for bekko) and is best on 4 of 5 topics. Bekko is best on Godot. The largest gap is on Laravel: 21.27, against 17.11 for the next best model.
332
+ - **Recall@50:** blink-8M has the highest mean (25.39) but is best on only 2 of 5 topics, Godot (by 0.01) and Laravel. Bekko is best on LangChain and YOLO, and potion-base-8M on Angular.
333
+ - **potion-base-8M:** it is the smallest model here, yet it is competitive. It is 4th on mean α-nDCG@10, and on mean Recall@50 it is roughly tied with lattice (22.83 vs 22.81).
334
 
335
  ## Training
336
 
figures/mteb_vs_beir.png ADDED

Git LFS Details

  • SHA256: af0d31197f60dbb4e4d828d72461d9c1e9f31b6aba5878525011601ad969279f
  • Pointer size: 131 Bytes
  • Size of remote file: 181 kB
figures/sparkle_2_blink.svg ADDED