Hugging Face's logo

minishlab
/
blink-8M

Feature Extraction
Model2Vec
Safetensors
sentence-transformers
English
embeddings
static-embeddings
sentence-similarity
Model card Files
xet
 Community
2
blink logo

blink-8M is a fast English static embedding model by Minish: a single 30,000 × 256 token embedding table (7.7M parameters, 31MB in float32) with mean pooling. It outperforms every static model of the same size, and, depending on the task, often much larger ones. It fully supersedes potion-base-8m, being the same size but better on every task. Static models are models without any active parameters; all model parameters are just embedding parameters. Because of this, they are many orders of magnitude faster than transformers. Although this comes at a cost, this cost is much lower than you would expect.

Usage

With model2vec:

from model2vec import StaticModel

model = StaticModel.from_pretrained("minishlab/blink-8M")
embeddings = model.encode(["a first sentence", "a second sentence"])

With model2vec-rs

use anyhow::Result;
use model2vec_rs::model::StaticModel;

fn main() -> Result<()> {
    let model = StaticModel::from_pretrained(
        "minishlab/blink-8M",
    )?;

    let sentences = vec![
        "Hello world".to_string(),
        "Rust is awesome".to_string(),
    ];

    let embeddings = model.encode(&sentences);
    println!("Generated {} embeddings.", embeddings.len());

    Ok(())
}

With Sentence Transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("minishlab/blink-8M")
embeddings = model.encode(["a first sentence", "a second sentence"])

Results

MTEB mean against Decontaminated BEIR nDCG@10 for all models

MTEB

MTEB(eng, v2), all 41 tasks (mteb 2.27.1). Every model is evaluated at 256 dimensions, see Compared models.

Model Mean (task) Retrieval Reranking PairCls Cls Clust STS Summ
blink-8M 55.35 38.86 43.12 78.42 67.88 41.49 73.56 22.48
lattice-retrieval 55.24 38.01 42.96 77.51 68.25 42.11 73.41 22.56
bekko-static-embedding-v1 55.21 37.62 42.54 78.32 68.42 40.77 74.60 22.24
potion-base-32M 54.02 31.94 42.52 77.33 70.48 41.49 73.17 24.14
potion-retrieval-32M 53.70 34.64 41.95 75.94 66.67 41.76 72.89 19.95
potion-base-8M 53.30 31.22 41.79 76.62 70.41 39.88 72.45 25.14
static-retrieval-mrl-en-v1 52.00 33.87 41.57 74.68 63.16 39.98 71.35 18.95

Summary

  • Mean: blink-8M has the highest mean (55.35), just ahead of lattice (55.24) and bekko (55.21).
  • Retrieval-type tasks: blink-8M is best on retrieval, reranking and pair classification. On retrieval it scores 38.86, against 38.01 for lattice and 31.22 for potion-base-8M. Note that this task has significant overlap with BEIR, see below.
  • Other tasks: the potion-base models are best on classification (70.48 and 70.41) and potion-base-8M on summarization (25.14), with blink-8M at 67.88 and 22.48. Bekko is best on STS and lattice on clustering.
  • Against potion-base-8M: at roughly the same size, blink-8M is 2.05 points higher on the mean and 7.64 points higher on retrieval, but 2.53 points lower on classification.

Compared models

We compare blink-8M to six other static embedding models:

Every model is evaluated at 256 dimensions. potion-base-32M, potion-retrieval-32M and static-retrieval-mrl-en-v1 are truncated to their first 256 dimensions and use L2 normalized embeddings, even if the original model did not have normalization. Applying normalization uniformly improved performance across all models.

We also evaluate many other datasets and tasks below.

RTEB: blink-8M 30.84 mean nDCG@10 (best)

RTEB (RTEB(eng, beta) in MTEB): retrieval across specialized domains, namely legal, finance, healthcare and code. We use the 15 public English tasks, including only the English subset of CUREv1. The private tasks are left out because their data is not available. All scores are nDCG@10, and the last row is the mean over the 15 tasks.

Task Domain blink-8M bekko-static-embedding-v1 lattice-retrieval potion-retrieval-32M static-retrieval-mrl-en-v1 potion-base-32M potion-base-8M
AILACasedocs Legal 26.81 27.03 27.56 21.97 24.21 25.02 27.17
AILAStatutes Legal 25.10 22.67 20.02 18.76 20.42 15.62 15.06
LegalSummarization Legal 58.55 58.19 57.99 54.95 52.91 57.02 55.35
FinanceBench Finance 34.76 37.21 35.55 31.96 35.51 27.30 27.88
HC3Finance Finance 33.21 27.08 32.96 26.96 25.11 25.67 24.76
FinQA Finance 50.80 48.22 45.96 45.16 44.41 38.77 38.09
ChatDoctor Healthcare 25.12 26.88 26.02 23.54 23.18 18.52 17.16
CUREv1 (en) Healthcare 33.18 32.80 32.77 29.71 27.56 28.17 26.93
Apps Code 3.60 8.54 4.16 3.90 4.08 3.22 2.86
DS1000 Code 23.95 23.97 22.96 21.70 21.75 18.66 17.36
HumanEval Code 45.67 49.14 42.69 40.73 40.29 38.68 39.13
MBPP Code 37.26 41.84 34.00 24.85 25.19 25.51 27.95
WikiSQL Code 37.29 25.84 31.62 32.18 29.90 21.94 21.81
FreshStack Code 22.73 21.45 21.32 19.22 18.88 18.13 18.20
SWE-bench Code 4.57 3.89 4.04 2.94 2.76 2.73 2.74
Mean 30.84 30.32 29.31 26.57 26.41 24.33 24.16

Note that FreshStack is also present as a separate eval, see below, although there it uses Recall@50 and α-nDCG@10.

Summary

  • Mean: blink-8M has the highest mean (30.84), but bekko is close (30.32). Blink-8M is best on 8 of 15 tasks, bekko on 6 and lattice on 1 (AILACasedocs).
  • By domain: blink-8M leads on legal (36.82 against 35.96 for bekko) and finance (39.59 against 37.50). Bekko leads on healthcare (29.84 against 29.15), and code is a tie (25.01 against 24.95).
  • Code: bekko is best on the code-generation tasks (Apps, HumanEval, MBPP and DS1000, the last by 0.02), while blink-8M has the largest win of the table on WikiSQL (37.29 against at most 32.18).
  • Overall: Apps and SWE-bench are hard for every model, with nDCG@10 below 9.
BrowseComp-Plus: blink-8M 2.13 evidence nDCG@10 (best)

BrowseComp-Plus: 830 hard, multi-hop BrowseComp questions over a fixed corpus of 100,195 web documents. Queries are the original questions, and documents are embedded in full, without truncation. We report two sets of relevance labels: evidence documents (all documents needed to answer the question) and gold documents (the documents that contain the answer).

Evidence

Model nDCG@10 Recall@5 Recall@100 Recall@1000
blink-8M 2.13 1.47 9.30 32.31
bekko-static-embedding-v1 2.09 1.57 11.13 36.46
lattice-retrieval 1.88 1.44 9.05 31.44
potion-retrieval-32M 1.73 1.20 9.32 32.96
static-retrieval-mrl-en-v1 1.42 1.03 8.09 29.80
potion-base-32M 1.32 0.86 7.43 25.49
potion-base-8M 1.25 0.98 6.93 24.58

Gold

Model nDCG@10 Recall@5 Recall@100 Recall@1000
blink-8M 1.78 1.89 10.16 34.65
bekko-static-embedding-v1 1.74 1.82 12.11 37.93
lattice-retrieval 1.58 1.81 9.82 33.23
potion-retrieval-32M 1.44 1.29 9.16 34.38
static-retrieval-mrl-en-v1 1.23 1.34 8.62 29.86
potion-base-32M 1.01 0.89 7.80 24.33
potion-base-8M 0.88 0.78 7.06 23.79

Blink has the best nDCG@10 on both label sets and the best gold Recall@5. Bekko has the best evidence Recall@5 and the best Recall@100 and Recall@1000 on both label sets.

Decontaminated BEIR: blink-8M 46.70 mean nDCG@10 (best)

We evaluate on LightOn's decontaminated version of BEIR (lightonai/<dataset>-decontaminated). These are 12 BEIR datasets with every query and document found in the mgte-en pre-training data removed. Contamination was found in two passes: exact matches on normalized text hashes, and 13-gram containment ≥ 0.5. Relevance judgments that point to a removed query or document were dropped as well.

We use the test split and only keep queries that have at least one relevant document left. Documents are encoded as title + text, and retrieval uses cosine similarity over the full corpus. We report nDCG@10 (ranking quality in the top 10) and Recall@100 (fraction of relevant documents found in the top 100). Following LightOn and Lattice, we leave out FEVER and ClimateFEVER.

Dataset Queries Documents
ArguAna 1,373 8,546
DBPedia 264 1,678,309
FiQA 61 47,617
HotpotQA 2,092 2,314,813
MS MARCO 41 4,036,967
NFCorpus 169 912
NQ 26 305,674
Quora 2,788 413,157
SCIDOCS 130 5,833
SciFact 41 858
TREC-COVID 50 99,522
Touché-2020 30 378,223

nDCG@10

Dataset blink-8M lattice-retrieval bekko-static-embedding-v1 potion-retrieval-32M static-retrieval-mrl-en-v1 potion-base-32M potion-base-8M
ArguAna 33.19 33.56 33.73 32.72 32.12 31.19 31.24
DBPedia 25.87 28.80 27.45 27.10 26.10 25.77 24.67
FiQA 31.58 28.35 27.60 27.04 27.84 25.28 24.14
HotpotQA 52.68 50.71 49.55 47.44 47.45 42.08 40.54
MS MARCO 51.75 48.63 48.00 50.32 47.64 43.68 40.62
NFCorpus 21.13 20.33 21.12 20.68 17.56 17.32 15.43
NQ 67.40 70.31 64.15 70.24 63.05 60.36 63.97
Quora 85.58 84.99 86.03 81.69 80.42 84.39 83.70
SCIDOCS 17.66 18.45 17.10 14.65 12.62 13.65 13.27
SciFact 83.39 80.67 81.29 78.61 79.25 74.76 72.68
TREC-COVID 68.04 66.96 66.14 50.71 47.74 45.16 48.85
Touché-2020 22.07 23.35 23.49 22.99 23.85 20.51 20.34
Mean 46.70 46.26 45.47 43.68 42.14 40.35 39.95

Recall@100

Dataset blink-8M lattice-retrieval bekko-static-embedding-v1 potion-retrieval-32M static-retrieval-mrl-en-v1 potion-base-32M potion-base-8M
ArguAna 95.41 95.48 95.92 95.70 95.27 92.94 92.43
DBPedia 47.21 49.88 50.15 48.21 48.09 44.70 42.06
FiQA 65.57 60.40 66.86 61.51 59.35 57.13 58.84
HotpotQA 74.07 72.94 73.37 70.24 70.72 65.46 63.15
MS MARCO 48.58 48.92 48.61 50.41 50.78 42.56 37.95
NFCorpus 43.83 43.34 44.78 40.85 38.89 38.64 35.59
NQ 96.15 92.31 88.46 96.15 92.31 88.46 88.46
Quora 98.47 98.43 98.65 97.58 97.03 98.15 97.94
SCIDOCS 48.59 47.37 42.63 44.94 39.23 42.63 41.09
SciFact 95.12 97.56 95.12 95.12 95.12 95.12 95.12
TREC-COVID 14.76 15.44 15.92 10.05 9.67 9.91 9.49
Touché-2020 51.70 54.47 53.29 53.42 51.01 50.78 48.53
Mean 64.96 64.71 64.48 63.68 62.29 60.54 59.22

Summary

  • nDCG@10: blink-8M has the highest mean (46.70) and is best on 6 of 12 datasets: FiQA, HotpotQA, MS MARCO, NFCorpus, SciFact and TREC-COVID. Lattice is best on 3 (DBPedia, NQ, SCIDOCS), bekko on 2 (ArguAna, Quora) and static-retrieval-mrl on 1 (Touché-2020).
  • Recall@100: the top three are close: blink-8M 64.96, lattice 64.71, bekko 64.48. Bekko is best on 6 datasets, blink-8M on 3 (one shared with potion-retrieval-32M).
  • TREC-COVID: blink-8M, lattice and bekko score 66–68 nDCG@10, while the potion models and static-retrieval-mrl score 45–51.
BRIGHT: blink-8M 10.51 mean nDCG@10 (best)

BRIGHT (Su et al., 2024) is a retrieval benchmark that needs reasoning: relevant documents often share few keywords or little surface topic with the query. For example, a query may describe a coding problem, and the relevant document is another problem that uses the same algorithm. BRIGHT has 12 tasks in three groups:

  • StackExchange (7): Biology, Earth Science, Economics, Psychology, Robotics, Stack Overflow and Sustainable Living. Queries are user posts, and relevant documents are web pages linked from accepted answers.
  • Coding (2): LeetCode (problems that share a solution technique) and Pony (documentation for a low-resource programming language).
  • Math (3): AoPS (competition problems with a similar solution), TheoremQA-Q (problems that use the same theorem) and TheoremQA-T (the theorem statement needed to solve a problem).

We use the original queries, not the LLM-generated reasoning traces, and the standard (short) document split. For each query we drop the documents BRIGHT marks as excluded, then retrieve the top 1000 documents by cosine similarity. Metrics are computed with pytrec_eval: nDCG@10 (ranking quality in the top 10, BRIGHT's main metric) and Recall@100 (fraction of relevant documents found in the top 100).

Because these need reasoning, we expect low scores: these are all static models and can't reason. Nevertheless, it's an interesting held-out task to try.

Task Group Queries Documents
Biology StackExchange 103 57,359
Earth Science StackExchange 116 121,249
Economics StackExchange 103 50,220
Psychology StackExchange 101 52,835
Robotics StackExchange 101 61,961
Stack Overflow StackExchange 117 107,081
Sustainable Living StackExchange 108 60,792
LeetCode Coding 142 413,932
Pony Coding 112 7,894
AoPS Math 111 188,002
TheoremQA-Q Math 194 188,002
TheoremQA-T Math 76 23,839

nDCG@10

Task blink-8M lattice-retrieval bekko-static-embedding-v1 potion-base-32M potion-retrieval-32M potion-base-8M static-retrieval-mrl-en-v1
Biology 14.68 13.50 11.19 11.57 10.57 9.60 9.18
Earth Science 25.58 24.01 21.24 19.35 19.61 17.11 18.82
Economics 11.22 11.04 10.94 11.17 9.76 10.79 9.72
Psychology 11.93 11.06 10.11 8.78 8.78 7.61 6.24
Robotics 7.26 6.51 6.27 7.17 7.63 5.49 7.44
Stack Overflow 12.94 11.00 10.08 9.12 7.21 8.39 6.65
Sustainable Living 7.55 7.16 8.04 8.90 6.62 8.11 5.91
LeetCode 15.98 15.70 13.41 13.79 13.14 12.78 14.10
Pony 3.52 3.44 1.18 0.77 2.49 1.09 1.33
AoPS 5.78 6.15 4.49 4.61 5.59 4.68 5.14
TheoremQA-Q 7.11 6.76 5.54 6.23 6.21 5.97 5.67
TheoremQA-T 2.59 1.37 1.30 1.53 1.18 1.70 0.89
Mean 10.51 9.81 8.65 8.58 8.23 7.77 7.59

Recall@100

Task blink-8M lattice-retrieval bekko-static-embedding-v1 potion-base-32M potion-retrieval-32M potion-base-8M static-retrieval-mrl-en-v1
Biology 50.99 45.07 44.26 40.37 39.12 37.50 37.80
Earth Science 58.10 55.71 53.50 47.19 50.62 47.05 48.91
Economics 37.89 36.92 37.18 34.47 33.88 33.93 30.39
Psychology 40.42 38.59 44.50 32.77 36.74 30.87 30.74
Robotics 31.66 26.84 26.90 30.27 27.20 27.33 23.42
Stack Overflow 40.20 33.54 34.06 32.45 29.53 29.58 25.34
Sustainable Living 45.27 43.28 42.70 35.89 42.81 36.64 36.96
LeetCode 38.73 40.79 34.08 42.05 39.38 41.94 36.29
Pony 17.03 10.85 8.81 9.11 14.30 8.94 10.81
AoPS 20.24 19.86 15.59 13.45 14.36 12.74 14.09
TheoremQA-Q 16.28 16.49 15.75 13.09 15.49 14.60 14.60
TheoremQA-T 7.68 7.46 10.53 6.80 7.24 6.36 3.73
Mean 33.71 31.28 30.65 28.16 29.22 27.29 26.09

Summary

  • nDCG@10: blink-8M has the highest mean (10.51, against 9.81 for lattice) and is best on 9 of 12 tasks. The other three go to potion-retrieval-32M (Robotics), potion-base-32M (Sustainable Living) and lattice (AoPS).
  • Recall@100: blink-8M has the highest mean (33.71, against 31.28 for lattice) and is best on 8 of 12 tasks. The largest gaps are on Pony (17.03 against at most 14.30 for the others) and Stack Overflow (40.20 against at most 34.06).
FreshStack: blink-8M 15.72 mean α-nDCG@10 (best)

FreshStack (Thakur et al., 2025) tests retrieval on niche, recent technical topics. The queries are Stack Overflow questions, and the corpus is code and documentation chunks from the GitHub repositories of each topic, taken from an October 2024 snapshot. Each question's accepted answer is split into atomic facts ("nuggets"), and corpus chunks are labeled with the nuggets they support. A good ranking therefore has to cover all the facts needed to answer the question, not just find one relevant chunk.

We evaluate on the test split of the 5 topics. Documents are encoded as title + text. Queries and documents are truncated to 2048 tokens, and we retrieve the top 50 documents by cosine similarity. Metrics come from the official freshstack evaluation package:

  • α-nDCG@10: nDCG over the top 10 where each document only gets credit for nuggets that higher-ranked documents have not already covered (α = 0.5). It rewards rankings that cover many different nuggets.
  • Recall@50: the fraction of a query's relevant documents found in the top 50.
Topic Queries Documents Nuggets
Angular 129 117,286 417
Godot 99 25,477 325
LangChain 203 49,505 639
Laravel 184 52,341 552
YOLO 57 27,207 198

α-nDCG@10

Topic blink-8M bekko-static-embedding-v1 lattice-retrieval potion-base-8M potion-retrieval-32M potion-base-32M static-retrieval-mrl-en-v1
Angular 16.89 16.11 12.88 15.67 13.00 12.91 12.90
Godot 12.60 14.42 12.01 10.10 10.55 9.97 8.91
LangChain 15.49 14.83 13.42 12.85 12.43 12.29 13.50
Laravel 21.27 17.11 15.68 16.61 16.75 16.36 15.80
YOLO 12.36 11.56 12.22 9.61 8.50 9.59 6.43
Mean 15.72 14.81 13.24 12.97 12.25 12.22 11.51

Recall@50

Topic blink-8M bekko-static-embedding-v1 lattice-retrieval potion-base-8M potion-retrieval-32M potion-base-32M static-retrieval-mrl-en-v1
Angular 21.45 20.06 17.45 22.77 20.03 19.45 18.60
Godot 25.41 25.40 23.27 19.50 21.07 18.09 17.97
LangChain 21.11 21.34 20.70 18.78 18.51 17.52 18.47
Laravel 36.71 27.73 31.73 31.70 25.82 28.60 27.98
YOLO 22.28 23.21 20.88 21.42 16.92 21.18 20.65
Mean 25.39 23.55 22.81 22.83 20.47 20.97 20.73

Summary

  • α-nDCG@10: blink-8M has the highest mean (15.72, against 14.81 for bekko) and is best on 4 of 5 topics. Bekko is best on Godot. The largest gap is on Laravel: 21.27, against 17.11 for the next best model.
  • Recall@50: blink-8M has the highest mean (25.39) but is best on only 2 of 5 topics, Godot (by 0.01) and Laravel. Bekko is best on LangChain and YOLO, and potion-base-8M on Angular.
  • potion-base-8M: it is the smallest model here, yet it is competitive. It is 4th on mean α-nDCG@10, and on mean Recall@50 it is roughly tied with lattice (22.83 vs 22.81).

Training

We followed a four-step procedure.

1. Tokenizer and potion initialization

We first train a 30k WordPiece tokenizer on LightOn's English pre-training data, and a distillation stage (potion) from bge-base-en-v1.5 on 1.1M samples of the Tokenlearn C4 data. This led to results comparable to potion-base-8m on MTEB and the retrieval tasks.

2. Contrastive training

Following, Erik Kaunismäki's work on training Lattice, we train on 96M text pairs from lightonai/embeddings-pre-training-curated, using three epochs instead of 1. We ablated hyperparameters in this phase on NanoBEIR. This led us to disable MRL, which had a slightly negative effect, but also did not lead to better performance at lower dimensionalities. Similarly, and unlike previous static models, we also found that training with a learning rate scheduler is unnecessary. However, like Hotchpotch's bekko recipe, we found that drastically lowering temperature works much better. Other hyperparameters and regularization schemes, e.g., dropping tokens, had little effect or were negative.

3. Hard negative fine tuning

Again following Erik's work, we then perform hard-negative fine-tuning on lightonai/embeddings-fine-tuning-filtered-en (1.57M queries with 10 mined hard negatives each), training for 3 epochs, again with a 0.2 constant learning rate.

4. Weight averaging

We found that weight averaging then caused the model to outperform either of the stages separately: the final model is 30% of the contrastive model from step 2 and 70% of the hard-negative model from step 3, giving us a boost on MTEB without sacrificing retrieval quality.

Library Authors

Model2Vec was developed by the Minish team consisting of Stephan Tulkens and Thomas van Dongen.

Citation

Please cite the Model2Vec repository if you use this model in your work.

@software{minishlab2024model2vec,
  author       = {Stephan Tulkens and {van Dongen}, Thomas},
  title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year         = {2024},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17270888},
  url          = {https://github.com/MinishLab/model2vec},
  license      = {MIT}
}
Downloads last month
17
Safetensors
Model size
7.68M params
Tensor type
F32
·
Inference Providers NEW
Feature Extraction
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train minishlab/blink-8M

Collection including minishlab/blink-8M