Hugging Face's logo

minishlab
/
blink-8M

Feature Extraction
Model2Vec
Safetensors
sentence-transformers
English
embeddings
static-embeddings
sentence-similarity
Model card Files
xet
 Community
2
Pringled commited on
Commit
61f9124
·
verified ·
1 Parent(s): eb594d7

MTEB table: bekko rerun in float32

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -76,7 +76,7 @@ MTEB(eng, v2), all 41 tasks (mteb 2.27.1). Every model is evaluated at 256 dimen
76
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
77
  | **blink-8M** | **55.35** | **38.86** | **43.12** | **78.42** | 67.88 | 41.49 | 73.56 | 22.48 |
78
  | lattice-retrieval | 55.24 | 38.01 | 42.96 | 77.51 | 68.25 | **42.11** | 73.41 | 22.56 |
79
- | bekko-static-embedding-v1 | 54.38 | 34.11 | 42.54 | 78.32 | 68.42 | 40.94 | **74.60** | 22.19 |
80
  | potion-base-32M | 54.02 | 31.94 | 42.52 | 77.33 | **70.48** | 41.49 | 73.17 | 24.14 |
81
  | potion-retrieval-32M | 53.70 | 34.64 | 41.95 | 75.94 | 66.67 | 41.76 | 72.89 | 19.95 |
82
  | potion-base-8M | 53.30 | 31.22 | 41.79 | 76.62 | 70.41 | 39.88 | 72.45 | **25.14** |
@@ -84,7 +84,7 @@ MTEB(eng, v2), all 41 tasks (mteb 2.27.1). Every model is evaluated at 256 dimen
84
 
85
  #### Summary
86
 
87
- - **Mean:** blink-8M has the highest mean (55.35), just ahead of lattice (55.24) and about 1 point ahead of bekko (54.38).
88
  - **Retrieval-type tasks:** blink-8M is best on retrieval, reranking and pair classification. On retrieval it scores 38.86, against 38.01 for lattice and 31.22 for potion-base-8M. Note that this task has significant overlap with BEIR, see below.
89
  - **Other tasks:** the potion-base models are best on classification (70.48 and 70.41) and potion-base-8M on summarization (25.14), with blink-8M at 67.88 and 22.48. Bekko is best on STS and lattice on clustering.
90
  - **Against potion-base-8M:** at roughly the same size, blink-8M is 2.05 points higher on the mean and 7.64 points higher on retrieval, but 2.53 points lower on classification.
 
76
  |---|---:|---:|---:|---:|---:|---:|---:|---:|
77
  | **blink-8M** | **55.35** | **38.86** | **43.12** | **78.42** | 67.88 | 41.49 | 73.56 | 22.48 |
78
  | lattice-retrieval | 55.24 | 38.01 | 42.96 | 77.51 | 68.25 | **42.11** | 73.41 | 22.56 |
79
+ | bekko-static-embedding-v1 | 55.21 | 37.62 | 42.54 | 78.32 | 68.42 | 40.77 | **74.60** | 22.24 |
80
  | potion-base-32M | 54.02 | 31.94 | 42.52 | 77.33 | **70.48** | 41.49 | 73.17 | 24.14 |
81
  | potion-retrieval-32M | 53.70 | 34.64 | 41.95 | 75.94 | 66.67 | 41.76 | 72.89 | 19.95 |
82
  | potion-base-8M | 53.30 | 31.22 | 41.79 | 76.62 | 70.41 | 39.88 | 72.45 | **25.14** |
 
84
 
85
  #### Summary
86
 
87
+ - **Mean:** blink-8M has the highest mean (55.35), just ahead of lattice (55.24) and bekko (55.21).
88
  - **Retrieval-type tasks:** blink-8M is best on retrieval, reranking and pair classification. On retrieval it scores 38.86, against 38.01 for lattice and 31.22 for potion-base-8M. Note that this task has significant overlap with BEIR, see below.
89
  - **Other tasks:** the potion-base models are best on classification (70.48 and 70.41) and potion-base-8M on summarization (25.14), with blink-8M at 67.88 and 22.48. Bekko is best on STS and lattice on clustering.
90
  - **Against potion-base-8M:** at roughly the same size, blink-8M is 2.05 points higher on the mean and 7.64 points higher on retrieval, but 2.53 points lower on classification.