Feature Extraction
Model2Vec
Safetensors
sentence-transformers
English
embeddings
static-embeddings
sentence-similarity
Instructions to use minishlab/blink-8M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Model2Vec
How to use minishlab/blink-8M with Model2Vec:
from model2vec import StaticModel model = StaticModel.from_pretrained("minishlab/blink-8M") embeddings = model.encode(["It's dangerous to go alone!", "It's a secret to everybody."]) print(embeddings.shape) - sentence-transformers
How to use minishlab/blink-8M with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("minishlab/blink-8M") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
Readme fixes
#1
by stephantulkens - opened
- .gitattributes +1 -0
- README.md +287 -14
- figures/mteb_vs_beir.png +3 -0
- figures/sparkle_2_blink.svg +15 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
figures/mteb_vs_beir.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -9,9 +9,11 @@ tags:
|
|
| 9 |
language: en
|
| 10 |
---
|
| 11 |
|
| 12 |
-
|
|
|
|
|
|
|
| 13 |
|
| 14 |
-
blink-8M is a fast English static embedding model: a single 30,000 × 256 token embedding table (7.7M parameters, 15MB in float16) with mean pooling. It has no transformer layers, so it runs on any CPU.
|
| 15 |
|
| 16 |
## Usage
|
| 17 |
|
|
@@ -24,6 +26,29 @@ model = StaticModel.from_pretrained("minishlab/blink-8M")
|
|
| 24 |
embeddings = model.encode(["a first sentence", "a second sentence"])
|
| 25 |
```
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
With [Sentence Transformers](https://www.sbert.net):
|
| 28 |
|
| 29 |
```python
|
|
@@ -35,7 +60,22 @@ embeddings = model.encode(["a first sentence", "a second sentence"])
|
|
| 35 |
|
| 36 |
## Results
|
| 37 |
|
| 38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
| Model | Mean (task) | Retrieval | Reranking | PairCls | Cls | Clust | STS | Summ |
|
| 41 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
@@ -47,17 +87,250 @@ MTEB(eng, v2), all 41 tasks (mteb 2.27.1), every model at 256 dimensions. blink-
|
|
| 47 |
| potion-base-8M | 53.30 | 31.22 | 41.79 | 76.62 | 70.41 | 39.88 | 72.45 | **25.14** |
|
| 48 |
| static-retrieval-mrl-en-v1 | 52.00 | 33.87 | 41.57 | 74.68 | 63.16 | 39.98 | 71.35 | 18.95 |
|
| 49 |
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
## Training
|
| 63 |
|
|
|
|
| 9 |
language: en
|
| 10 |
---
|
| 11 |
|
| 12 |
+
<div align="center">
|
| 13 |
+
<img src="figures/sparkle_2_blink.svg" alt="blink logo" width="325">
|
| 14 |
+
</div>
|
| 15 |
|
| 16 |
+
blink-8M is a fast English static embedding model by [Minish](minish.ai): a single 30,000 × 256 token embedding table (7.7M parameters, 15MB in float16) with mean pooling. It has no transformer layers, so it runs on any CPU.
|
| 17 |
|
| 18 |
## Usage
|
| 19 |
|
|
|
|
| 26 |
embeddings = model.encode(["a first sentence", "a second sentence"])
|
| 27 |
```
|
| 28 |
|
| 29 |
+
With [model2vec-rs](https://github.com/MinishLab/model2vec-rs)
|
| 30 |
+
|
| 31 |
+
```rust
|
| 32 |
+
use anyhow::Result;
|
| 33 |
+
use model2vec_rs::model::StaticModel;
|
| 34 |
+
|
| 35 |
+
fn main() -> Result<()> {
|
| 36 |
+
let model = StaticModel::from_pretrained(
|
| 37 |
+
"minishlab/blink-8M",
|
| 38 |
+
)?;
|
| 39 |
+
|
| 40 |
+
let sentences = vec![
|
| 41 |
+
"Hello world".to_string(),
|
| 42 |
+
"Rust is awesome".to_string(),
|
| 43 |
+
];
|
| 44 |
+
|
| 45 |
+
let embeddings = model.encode(&sentences);
|
| 46 |
+
println!("Generated {} embeddings.", embeddings.len());
|
| 47 |
+
|
| 48 |
+
Ok(())
|
| 49 |
+
}
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
With [Sentence Transformers](https://www.sbert.net):
|
| 53 |
|
| 54 |
```python
|
|
|
|
| 60 |
|
| 61 |
## Results
|
| 62 |
|
| 63 |
+
We compare blink-8M to six other static embedding models:
|
| 64 |
+
|
| 65 |
+
- [potion-base-8M](https://huggingface.co/minishlab/potion-base-8M): MinishLab's general-purpose model2vec model, 30k vocab, 256 dimensions.
|
| 66 |
+
- [potion-base-32M](https://huggingface.co/minishlab/potion-base-32M): a larger version of potion-base-8M, 65k vocab, 512 dimensions.
|
| 67 |
+
- [potion-retrieval-32M](https://huggingface.co/minishlab/potion-retrieval-32M): potion-base-32M fine-tuned for retrieval, 65k vocab, 512 dimensions.
|
| 68 |
+
- [static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1): Sentence Transformers' static retrieval model, 30k vocab, 1024 dimensions.
|
| 69 |
+
- [bekko-static-embedding-v1](hotchpotch/bekko-static-embedding-v1): A multilingual model trained on retrieval data, 200k vocab, 1024 dimensions.
|
| 70 |
+
- [lattice-retrieval](erikkaum/lattice-retrieval): A monolingual model, trained on retrieval data, 30k vocab, 1024 dimensions.
|
| 71 |
+
|
| 72 |
+
Every model is evaluated at 256 dimensions. potion-base-32M, potion-retrieval-32M and static-retrieval-mrl-en-v1 are truncated to their first 256 dimensions and use L2 normalized embeddings, even if the original model did not have normalization. Applying normalization uniformly improved performance across all models.
|
| 73 |
+
|
| 74 |
+

|
| 75 |
+
|
| 76 |
+
### MTEB
|
| 77 |
+
|
| 78 |
+
MTEB(eng, v2), all 41 tasks (mteb 2.27.1).
|
| 79 |
|
| 80 |
| Model | Mean (task) | Retrieval | Reranking | PairCls | Cls | Clust | STS | Summ |
|
| 81 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
|
|
| 87 |
| potion-base-8M | 53.30 | 31.22 | 41.79 | 76.62 | 70.41 | 39.88 | 72.45 | **25.14** |
|
| 88 |
| static-retrieval-mrl-en-v1 | 52.00 | 33.87 | 41.57 | 74.68 | 63.16 | 39.98 | 71.35 | 18.95 |
|
| 89 |
|
| 90 |
+
#### Summary
|
| 91 |
+
|
| 92 |
+
- **Mean:** blink-8M has the highest mean (55.34), just ahead of lattice (55.24) and about 1 point ahead of bekko (54.38).
|
| 93 |
+
- **Retrieval-type tasks:** blink-8M is best on retrieval, reranking and pair classification. On retrieval it scores 38.86, against 38.01 for lattice and 31.22 for potion-base-8M. Note that this task has significant overlap with BEIR, see below.
|
| 94 |
+
- **Other tasks:** the potion-base models are best on classification (70.48 and 70.41) and potion-base-8M on summarization (25.14), with blink-8M at 67.88 and 22.44. Bekko is best on STS and lattice on clustering.
|
| 95 |
+
- **Against potion-base-8M:** at roughly the same size, blink-8M is 2.04 points higher on the mean and 7.64 points higher on retrieval, but 2.53 points lower on classification.
|
| 96 |
+
|
| 97 |
+
### RTEB
|
| 98 |
+
|
| 99 |
+
[RTEB](https://huggingface.co/blog/rteb) (`RTEB(eng, beta)` in MTEB): retrieval across specialized domains, namely legal, finance, healthcare and code. We use the 15 public English tasks, including only the English subset of CUREv1. The private tasks are left out because their data is not available. All scores are nDCG@10, and the last row is the mean over the 15 tasks.
|
| 100 |
+
|
| 101 |
+
| Task | Domain | blink-8M | bekko-static-embedding-v1 | lattice-retrieval | potion-retrieval-32M | static-retrieval-mrl-en-v1 | potion-base-32M | potion-base-8M |
|
| 102 |
+
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
| 103 |
+
| AILACasedocs | Legal | 26.81 | 27.03 | **27.56** | 21.97 | 24.21 | 25.02 | 27.17 |
|
| 104 |
+
| AILAStatutes | Legal | **25.10** | 22.67 | 20.02 | 18.76 | 20.42 | 15.62 | 15.06 |
|
| 105 |
+
| LegalSummarization | Legal | **58.55** | 58.19 | 57.99 | 54.95 | 52.91 | 57.02 | 55.35 |
|
| 106 |
+
| FinanceBench | Finance | 34.76 | **37.21** | 35.55 | 31.96 | 35.51 | 27.30 | 27.88 |
|
| 107 |
+
| HC3Finance | Finance | **33.21** | 27.08 | 32.96 | 26.96 | 25.11 | 25.67 | 24.76 |
|
| 108 |
+
| FinQA | Finance | **50.80** | 48.22 | 45.96 | 45.16 | 44.41 | 38.77 | 38.09 |
|
| 109 |
+
| ChatDoctor | Healthcare | 25.12 | **26.88** | 26.02 | 23.54 | 23.18 | 18.52 | 17.16 |
|
| 110 |
+
| CUREv1 (en) | Healthcare | **33.18** | 32.80 | 32.77 | 29.71 | 27.56 | 28.17 | 26.93 |
|
| 111 |
+
| Apps | Code | 3.60 | **8.54** | 4.16 | 3.90 | 4.08 | 3.22 | 2.86 |
|
| 112 |
+
| DS1000 | Code | 23.95 | **23.97** | 22.96 | 21.70 | 21.75 | 18.66 | 17.36 |
|
| 113 |
+
| HumanEval | Code | 45.67 | **49.14** | 42.69 | 40.73 | 40.29 | 38.68 | 39.13 |
|
| 114 |
+
| MBPP | Code | 37.26 | **41.84** | 34.00 | 24.85 | 25.19 | 25.51 | 27.95 |
|
| 115 |
+
| WikiSQL | Code | **37.29** | 25.84 | 31.62 | 32.18 | 29.90 | 21.94 | 21.81 |
|
| 116 |
+
| FreshStack | Code | **22.73** | 21.45 | 21.32 | 19.22 | 18.88 | 18.13 | 18.20 |
|
| 117 |
+
| SWE-bench | Code | **4.57** | 3.89 | 4.04 | 2.94 | 2.76 | 2.73 | 2.74 |
|
| 118 |
+
| **Mean** | | **30.84** | 30.32 | 29.31 | 26.57 | 26.41 | 24.33 | 24.16 |
|
| 119 |
+
|
| 120 |
+
Note that FreshStack is also present as a separate eval, see below, although there it uses Recall@50 and α-nDCG@10.
|
| 121 |
+
|
| 122 |
+
#### Summary
|
| 123 |
+
|
| 124 |
+
- **Mean:** blink-8M has the highest mean (30.84), but bekko is close (30.32). Blink-8M is best on 8 of 15 tasks, bekko on 6 and lattice on 1 (AILACasedocs).
|
| 125 |
+
- **By domain:** blink-8M leads on legal (36.82 against 35.96 for bekko) and finance (39.59 against 37.50). Bekko leads on healthcare (29.84 against 29.15), and code is a tie (25.01 against 24.95).
|
| 126 |
+
- **Code:** bekko is best on the code-generation tasks (Apps, HumanEval, MBPP and DS1000, the last by 0.02), while blink-8M has the largest win of the table on WikiSQL (37.29 against at most 32.18).
|
| 127 |
+
- **Overall:** Apps and SWE-bench are hard for every model, with nDCG@10 below 9.
|
| 128 |
+
|
| 129 |
+
### BrowseComp-Plus
|
| 130 |
+
|
| 131 |
+
[BrowseComp-Plus](https://github.com/texttron/BrowseComp-Plus): 830 hard, multi-hop BrowseComp questions over a fixed corpus of 100,195 web documents. Queries are the original questions, and documents are embedded in full, without truncation. We report two sets of relevance labels: *evidence* documents (all documents needed to answer the question) and *gold* documents (the documents that contain the answer).
|
| 132 |
+
|
| 133 |
+
**Evidence**
|
| 134 |
+
|
| 135 |
+
| Model | nDCG@10 | Recall@5 | Recall@100 | Recall@1000 |
|
| 136 |
+
|---|---:|---:|---:|---:|
|
| 137 |
+
| **blink-8M** | **2.13** | 1.47 | 9.30 | 32.31 |
|
| 138 |
+
| bekko-static-embedding-v1 | 2.09 | **1.57** | **11.13** | **36.46** |
|
| 139 |
+
| lattice-retrieval | 1.88 | 1.44 | 9.05 | 31.44 |
|
| 140 |
+
| potion-retrieval-32M | 1.73 | 1.20 | 9.32 | 32.96 |
|
| 141 |
+
| static-retrieval-mrl-en-v1 | 1.42 | 1.03 | 8.09 | 29.80 |
|
| 142 |
+
| potion-base-32M | 1.32 | 0.86 | 7.43 | 25.49 |
|
| 143 |
+
| potion-base-8M | 1.25 | 0.98 | 6.93 | 24.58 |
|
| 144 |
+
|
| 145 |
+
**Gold**
|
| 146 |
+
|
| 147 |
+
| Model | nDCG@10 | Recall@5 | Recall@100 | Recall@1000 |
|
| 148 |
+
|---|---:|---:|---:|---:|
|
| 149 |
+
| **blink-8M** | **1.78** | **1.89** | 10.16 | 34.65 |
|
| 150 |
+
| bekko-static-embedding-v1 | 1.74 | 1.82 | **12.11** | **37.93** |
|
| 151 |
+
| lattice-retrieval | 1.58 | 1.81 | 9.82 | 33.23 |
|
| 152 |
+
| potion-retrieval-32M | 1.44 | 1.29 | 9.16 | 34.38 |
|
| 153 |
+
| static-retrieval-mrl-en-v1 | 1.23 | 1.34 | 8.62 | 29.86 |
|
| 154 |
+
| potion-base-32M | 1.01 | 0.89 | 7.80 | 24.33 |
|
| 155 |
+
| potion-base-8M | 0.88 | 0.78 | 7.06 | 23.79 |
|
| 156 |
+
|
| 157 |
+
Blink has the best nDCG@10 on both label sets and the best gold Recall@5. Bekko has the best evidence Recall@5 and the best Recall@100 and Recall@1000 on both label sets.
|
| 158 |
+
|
| 159 |
+
### BEIR
|
| 160 |
+
|
| 161 |
+
We evaluate on LightOn's decontaminated version of BEIR (`lightonai/<dataset>-decontaminated`). These are 12 BEIR datasets with every query and document found in the [mgte-en](https://huggingface.co/datasets/Alibaba-NLP/mgte-en) pre-training data removed. Contamination was found in two passes: exact matches on normalized text hashes, and 13-gram containment ≥ 0.5. Relevance judgments that point to a removed query or document were dropped as well.
|
| 162 |
+
|
| 163 |
+
We use the test split and only keep queries that have at least one relevant document left. Documents are encoded as `title + text`, and retrieval uses cosine similarity over the full corpus. We report nDCG@10 (ranking quality in the top 10) and Recall@100 (fraction of relevant documents found in the top 100). Following LightOn and [Lattice](https://github.com/ErikKaum/lattice/blob/main/evals.md), we leave out FEVER and ClimateFEVER.
|
| 164 |
+
|
| 165 |
+
| Dataset | Queries | Documents |
|
| 166 |
+
|---|---:|---:|
|
| 167 |
+
| ArguAna | 1,373 | 8,546 |
|
| 168 |
+
| DBPedia | 264 | 1,678,309 |
|
| 169 |
+
| FiQA | 61 | 47,617 |
|
| 170 |
+
| HotpotQA | 2,092 | 2,314,813 |
|
| 171 |
+
| MS MARCO | 41 | 4,036,967 |
|
| 172 |
+
| NFCorpus | 169 | 912 |
|
| 173 |
+
| NQ | 26 | 305,674 |
|
| 174 |
+
| Quora | 2,788 | 413,157 |
|
| 175 |
+
| SCIDOCS | 130 | 5,833 |
|
| 176 |
+
| SciFact | 41 | 858 |
|
| 177 |
+
| TREC-COVID | 50 | 99,522 |
|
| 178 |
+
| Touché-2020 | 30 | 378,223 |
|
| 179 |
+
|
| 180 |
+
#### nDCG@10
|
| 181 |
+
|
| 182 |
+
| Dataset | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-retrieval-32M | static-retrieval-mrl-en-v1 | potion-base-32M | potion-base-8M |
|
| 183 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 184 |
+
| ArguAna | 33.19 | 33.56 | **33.73** | 32.72 | 32.12 | 31.19 | 31.24 |
|
| 185 |
+
| DBPedia | 25.87 | **28.80** | 27.45 | 27.10 | 26.10 | 25.77 | 24.67 |
|
| 186 |
+
| FiQA | **31.58** | 28.35 | 27.60 | 27.04 | 27.84 | 25.28 | 24.14 |
|
| 187 |
+
| HotpotQA | **52.68** | 50.71 | 49.55 | 47.44 | 47.45 | 42.08 | 40.54 |
|
| 188 |
+
| MS MARCO | **51.75** | 48.63 | 48.00 | 50.32 | 47.64 | 43.68 | 40.62 |
|
| 189 |
+
| NFCorpus | **21.13** | 20.33 | 21.12 | 20.68 | 17.56 | 17.32 | 15.43 |
|
| 190 |
+
| NQ | 67.40 | **70.31** | 64.15 | 70.24 | 63.05 | 60.36 | 63.97 |
|
| 191 |
+
| Quora | 85.58 | 84.99 | **86.03** | 81.69 | 80.42 | 84.39 | 83.70 |
|
| 192 |
+
| SCIDOCS | 17.66 | **18.45** | 17.10 | 14.65 | 12.62 | 13.65 | 13.27 |
|
| 193 |
+
| SciFact | **83.39** | 80.67 | 81.29 | 78.61 | 79.25 | 74.76 | 72.68 |
|
| 194 |
+
| TREC-COVID | **68.04** | 66.96 | 66.14 | 50.71 | 47.74 | 45.16 | 48.85 |
|
| 195 |
+
| Touché-2020 | 22.07 | 23.35 | 23.49 | 22.99 | **23.85** | 20.51 | 20.34 |
|
| 196 |
+
| **Mean** | **46.70** | 46.26 | 45.47 | 43.68 | 42.14 | 40.35 | 39.95 |
|
| 197 |
+
|
| 198 |
+
#### Recall@100
|
| 199 |
+
|
| 200 |
+
| Dataset | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-retrieval-32M | static-retrieval-mrl-en-v1 | potion-base-32M | potion-base-8M |
|
| 201 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 202 |
+
| ArguAna | 95.41 | 95.48 | **95.92** | 95.70 | 95.27 | 92.94 | 92.43 |
|
| 203 |
+
| DBPedia | 47.21 | 49.88 | **50.15** | 48.21 | 48.09 | 44.70 | 42.06 |
|
| 204 |
+
| FiQA | 65.57 | 60.40 | **66.86** | 61.51 | 59.35 | 57.13 | 58.84 |
|
| 205 |
+
| HotpotQA | **74.07** | 72.94 | 73.37 | 70.24 | 70.72 | 65.46 | 63.15 |
|
| 206 |
+
| MS MARCO | 48.58 | 48.92 | 48.61 | 50.41 | **50.78** | 42.56 | 37.95 |
|
| 207 |
+
| NFCorpus | 43.83 | 43.34 | **44.78** | 40.85 | 38.89 | 38.64 | 35.59 |
|
| 208 |
+
| NQ | **96.15** | 92.31 | 88.46 | **96.15** | 92.31 | 88.46 | 88.46 |
|
| 209 |
+
| Quora | 98.47 | 98.43 | **98.65** | 97.58 | 97.03 | 98.15 | 97.94 |
|
| 210 |
+
| SCIDOCS | **48.59** | 47.37 | 42.63 | 44.94 | 39.23 | 42.63 | 41.09 |
|
| 211 |
+
| SciFact | 95.12 | **97.56** | 95.12 | 95.12 | 95.12 | 95.12 | 95.12 |
|
| 212 |
+
| TREC-COVID | 14.76 | 15.44 | **15.92** | 10.05 | 9.67 | 9.91 | 9.49 |
|
| 213 |
+
| Touché-2020 | 51.70 | **54.47** | 53.29 | 53.42 | 51.01 | 50.78 | 48.53 |
|
| 214 |
+
| **Mean** | **64.96** | 64.71 | 64.48 | 63.68 | 62.29 | 60.54 | 59.22 |
|
| 215 |
+
|
| 216 |
+
#### Summary
|
| 217 |
+
|
| 218 |
+
- **nDCG@10:** blink-8M has the highest mean (46.70) and is best on 6 of 12 datasets: FiQA, HotpotQA, MS MARCO, NFCorpus, SciFact and TREC-COVID. Lattice is best on 3 (DBPedia, NQ, SCIDOCS), bekko on 2 (ArguAna, Quora) and static-retrieval-mrl on 1 (Touché-2020).
|
| 219 |
+
- **Recall@100:** the top three are close: blink-8M 64.96, lattice 64.71, bekko 64.48. Bekko is best on 6 datasets, blink-8M on 3 (one shared with potion-retrieval-32M).
|
| 220 |
+
- **TREC-COVID:** blink-8M, lattice and bekko score 66–68 nDCG@10, while the potion models and static-retrieval-mrl score 45–51.
|
| 221 |
+
|
| 222 |
+
### BRIGHT
|
| 223 |
+
|
| 224 |
+
[BRIGHT](https://huggingface.co/datasets/xlangai/BRIGHT) (Su et al., 2024) is a retrieval benchmark that needs reasoning: relevant documents often share few keywords or little surface topic with the query. For example, a query may describe a coding problem, and the relevant document is another problem that uses the same algorithm. BRIGHT has 12 tasks in three groups:
|
| 225 |
+
|
| 226 |
+
- **StackExchange** (7): Biology, Earth Science, Economics, Psychology, Robotics, Stack Overflow and Sustainable Living. Queries are user posts, and relevant documents are web pages linked from accepted answers.
|
| 227 |
+
- **Coding** (2): LeetCode (problems that share a solution technique) and Pony (documentation for a low-resource programming language).
|
| 228 |
+
- **Math** (3): AoPS (competition problems with a similar solution), TheoremQA-Q (problems that use the same theorem) and TheoremQA-T (the theorem statement needed to solve a problem).
|
| 229 |
+
|
| 230 |
+
We use the original queries, not the LLM-generated reasoning traces, and the standard (short) document split. For each query we drop the documents BRIGHT marks as excluded, then retrieve the top 1000 documents by cosine similarity. Metrics are computed with `pytrec_eval`: nDCG@10 (ranking quality in the top 10, BRIGHT's main metric) and Recall@100 (fraction of relevant documents found in the top 100).
|
| 231 |
+
|
| 232 |
+
Because these need reasoning, we expect low scores: these are all static models and can't reason. Nevertheless, it's an interesting held-out task to try.
|
| 233 |
+
|
| 234 |
+
| Task | Group | Queries | Documents |
|
| 235 |
+
|---|---|---:|---:|
|
| 236 |
+
| Biology | StackExchange | 103 | 57,359 |
|
| 237 |
+
| Earth Science | StackExchange | 116 | 121,249 |
|
| 238 |
+
| Economics | StackExchange | 103 | 50,220 |
|
| 239 |
+
| Psychology | StackExchange | 101 | 52,835 |
|
| 240 |
+
| Robotics | StackExchange | 101 | 61,961 |
|
| 241 |
+
| Stack Overflow | StackExchange | 117 | 107,081 |
|
| 242 |
+
| Sustainable Living | StackExchange | 108 | 60,792 |
|
| 243 |
+
| LeetCode | Coding | 142 | 413,932 |
|
| 244 |
+
| Pony | Coding | 112 | 7,894 |
|
| 245 |
+
| AoPS | Math | 111 | 188,002 |
|
| 246 |
+
| TheoremQA-Q | Math | 194 | 188,002 |
|
| 247 |
+
| TheoremQA-T | Math | 76 | 23,839 |
|
| 248 |
+
|
| 249 |
+
#### nDCG@10
|
| 250 |
+
|
| 251 |
+
| Task | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-base-32M | potion-retrieval-32M | potion-base-8M | static-retrieval-mrl-en-v1 |
|
| 252 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 253 |
+
| Biology | **14.68** | 13.50 | 11.19 | 11.57 | 10.57 | 9.60 | 9.18 |
|
| 254 |
+
| Earth Science | **25.58** | 24.01 | 21.24 | 19.35 | 19.61 | 17.11 | 18.82 |
|
| 255 |
+
| Economics | **11.22** | 11.04 | 10.94 | 11.17 | 9.76 | 10.79 | 9.72 |
|
| 256 |
+
| Psychology | **11.93** | 11.06 | 10.11 | 8.78 | 8.78 | 7.61 | 6.24 |
|
| 257 |
+
| Robotics | 7.26 | 6.51 | 6.27 | 7.17 | **7.63** | 5.49 | 7.44 |
|
| 258 |
+
| Stack Overflow | **12.94** | 11.00 | 10.08 | 9.12 | 7.21 | 8.39 | 6.65 |
|
| 259 |
+
| Sustainable Living | 7.55 | 7.16 | 8.04 | **8.90** | 6.62 | 8.11 | 5.91 |
|
| 260 |
+
| LeetCode | **15.98** | 15.70 | 13.41 | 13.79 | 13.14 | 12.78 | 14.10 |
|
| 261 |
+
| Pony | **3.52** | 3.44 | 1.18 | 0.77 | 2.49 | 1.09 | 1.33 |
|
| 262 |
+
| AoPS | 5.78 | **6.15** | 4.49 | 4.61 | 5.59 | 4.68 | 5.14 |
|
| 263 |
+
| TheoremQA-Q | **7.11** | 6.76 | 5.54 | 6.23 | 6.21 | 5.97 | 5.67 |
|
| 264 |
+
| TheoremQA-T | **2.59** | 1.37 | 1.30 | 1.53 | 1.18 | 1.70 | 0.89 |
|
| 265 |
+
| **Mean** | **10.51** | 9.81 | 8.65 | 8.58 | 8.23 | 7.77 | 7.59 |
|
| 266 |
+
|
| 267 |
+
#### Recall@100
|
| 268 |
+
|
| 269 |
+
| Task | blink-8M | lattice-retrieval | bekko-static-embedding-v1 | potion-base-32M | potion-retrieval-32M | potion-base-8M | static-retrieval-mrl-en-v1 |
|
| 270 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 271 |
+
| Biology | **50.99** | 45.07 | 44.26 | 40.37 | 39.12 | 37.50 | 37.80 |
|
| 272 |
+
| Earth Science | **58.10** | 55.71 | 53.50 | 47.19 | 50.62 | 47.05 | 48.91 |
|
| 273 |
+
| Economics | **37.89** | 36.92 | 37.18 | 34.47 | 33.88 | 33.93 | 30.39 |
|
| 274 |
+
| Psychology | 40.42 | 38.59 | **44.50** | 32.77 | 36.74 | 30.87 | 30.74 |
|
| 275 |
+
| Robotics | **31.66** | 26.84 | 26.90 | 30.27 | 27.20 | 27.33 | 23.42 |
|
| 276 |
+
| Stack Overflow | **40.20** | 33.54 | 34.06 | 32.45 | 29.53 | 29.58 | 25.34 |
|
| 277 |
+
| Sustainable Living | **45.27** | 43.28 | 42.70 | 35.89 | 42.81 | 36.64 | 36.96 |
|
| 278 |
+
| LeetCode | 38.73 | 40.79 | 34.08 | **42.05** | 39.38 | 41.94 | 36.29 |
|
| 279 |
+
| Pony | **17.03** | 10.85 | 8.81 | 9.11 | 14.30 | 8.94 | 10.81 |
|
| 280 |
+
| AoPS | **20.24** | 19.86 | 15.59 | 13.45 | 14.36 | 12.74 | 14.09 |
|
| 281 |
+
| TheoremQA-Q | 16.28 | **16.49** | 15.75 | 13.09 | 15.49 | 14.60 | 14.60 |
|
| 282 |
+
| TheoremQA-T | 7.68 | 7.46 | **10.53** | 6.80 | 7.24 | 6.36 | 3.73 |
|
| 283 |
+
| **Mean** | **33.71** | 31.28 | 30.65 | 28.16 | 29.22 | 27.29 | 26.09 |
|
| 284 |
+
|
| 285 |
+
#### Summary
|
| 286 |
+
|
| 287 |
+
- **nDCG@10:** blink-8M has the highest mean (10.51, against 9.81 for lattice) and is best on 9 of 12 tasks. The other three go to potion-retrieval-32M (Robotics), potion-base-32M (Sustainable Living) and lattice (AoPS).
|
| 288 |
+
- **Recall@100:** blink-8M has the highest mean (33.71, against 31.28 for lattice) and is best on 8 of 12 tasks. The largest gaps are on Pony (17.03 against at most 14.30 for the others) and Stack Overflow (40.20 against at most 34.06).
|
| 289 |
+
|
| 290 |
+
### FreshStack
|
| 291 |
+
|
| 292 |
+
[FreshStack](https://fresh-stack.github.io/) (Thakur et al., 2025) tests retrieval on niche, recent technical topics. The queries are Stack Overflow questions, and the corpus is code and documentation chunks from the GitHub repositories of each topic, taken from an October 2024 snapshot. Each question's accepted answer is split into atomic facts ("nuggets"), and corpus chunks are labeled with the nuggets they support. A good ranking therefore has to cover all the facts needed to answer the question, not just find one relevant chunk.
|
| 293 |
+
|
| 294 |
+
We evaluate on the test split of the 5 topics. Documents are encoded as `title + text`. Queries and documents are truncated to 2048 tokens, and we retrieve the top 50 documents by cosine similarity. Metrics come from the official `freshstack` evaluation package:
|
| 295 |
+
|
| 296 |
+
- **α-nDCG@10**: nDCG over the top 10 where each document only gets credit for nuggets that higher-ranked documents have not already covered (α = 0.5). It rewards rankings that cover many different nuggets.
|
| 297 |
+
- **Recall@50**: the fraction of a query's relevant documents found in the top 50.
|
| 298 |
+
|
| 299 |
+
| Topic | Queries | Documents | Nuggets |
|
| 300 |
+
|---|---:|---:|---:|
|
| 301 |
+
| Angular | 129 | 117,286 | 417 |
|
| 302 |
+
| Godot | 99 | 25,477 | 325 |
|
| 303 |
+
| LangChain | 203 | 49,505 | 639 |
|
| 304 |
+
| Laravel | 184 | 52,341 | 552 |
|
| 305 |
+
| YOLO | 57 | 27,207 | 198 |
|
| 306 |
+
|
| 307 |
+
#### α-nDCG@10
|
| 308 |
+
|
| 309 |
+
| Topic | blink-8M | bekko-static-embedding-v1 | lattice-retrieval | potion-base-8M | potion-retrieval-32M | potion-base-32M | static-retrieval-mrl-en-v1 |
|
| 310 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 311 |
+
| Angular | **16.89** | 16.11 | 12.88 | 15.67 | 13.00 | 12.91 | 12.90 |
|
| 312 |
+
| Godot | 12.60 | **14.42** | 12.01 | 10.10 | 10.55 | 9.97 | 8.91 |
|
| 313 |
+
| LangChain | **15.49** | 14.83 | 13.42 | 12.85 | 12.43 | 12.29 | 13.50 |
|
| 314 |
+
| Laravel | **21.27** | 17.11 | 15.68 | 16.61 | 16.75 | 16.36 | 15.80 |
|
| 315 |
+
| YOLO | **12.36** | 11.56 | 12.22 | 9.61 | 8.50 | 9.59 | 6.43 |
|
| 316 |
+
| **Mean** | **15.72** | 14.81 | 13.24 | 12.97 | 12.25 | 12.22 | 11.51 |
|
| 317 |
+
|
| 318 |
+
#### Recall@50
|
| 319 |
+
|
| 320 |
+
| Topic | blink-8M | bekko-static-embedding-v1 | lattice-retrieval | potion-base-8M | potion-retrieval-32M | potion-base-32M | static-retrieval-mrl-en-v1 |
|
| 321 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 322 |
+
| Angular | 21.45 | 20.06 | 17.45 | **22.77** | 20.03 | 19.45 | 18.60 |
|
| 323 |
+
| Godot | **25.41** | 25.40 | 23.27 | 19.50 | 21.07 | 18.09 | 17.97 |
|
| 324 |
+
| LangChain | 21.11 | **21.34** | 20.70 | 18.78 | 18.51 | 17.52 | 18.47 |
|
| 325 |
+
| Laravel | **36.71** | 27.73 | 31.73 | 31.70 | 25.82 | 28.60 | 27.98 |
|
| 326 |
+
| YOLO | 22.28 | **23.21** | 20.88 | 21.42 | 16.92 | 21.18 | 20.65 |
|
| 327 |
+
| **Mean** | **25.39** | 23.55 | 22.81 | 22.83 | 20.47 | 20.97 | 20.73 |
|
| 328 |
+
|
| 329 |
+
#### Summary
|
| 330 |
+
|
| 331 |
+
- **α-nDCG@10:** blink-8M has the highest mean (15.72, against 14.81 for bekko) and is best on 4 of 5 topics. Bekko is best on Godot. The largest gap is on Laravel: 21.27, against 17.11 for the next best model.
|
| 332 |
+
- **Recall@50:** blink-8M has the highest mean (25.39) but is best on only 2 of 5 topics, Godot (by 0.01) and Laravel. Bekko is best on LangChain and YOLO, and potion-base-8M on Angular.
|
| 333 |
+
- **potion-base-8M:** it is the smallest model here, yet it is competitive. It is 4th on mean α-nDCG@10, and on mean Recall@50 it is roughly tied with lattice (22.83 vs 22.81).
|
| 334 |
|
| 335 |
## Training
|
| 336 |
|
figures/mteb_vs_beir.png
ADDED
|
Git LFS Details
|
figures/sparkle_2_blink.svg
ADDED
|
|