Feature Extraction
Model2Vec
Safetensors
sentence-transformers
embeddings
static-embeddings
sentence-similarity
How to use from the
Use from the
Model2Vec library
from model2vec import StaticModel

model = StaticModel.from_pretrained("minishlab/blink-multilingual-64M")

embeddings = model.encode(["It's dangerous to go alone!", "It's a secret to everybody."])
print(embeddings.shape)
blink logo

blink-multilingual-64M is a fast multilingual static embedding model by Minish: a single 250,000 × 256 token embedding table (64.0M parameters, 256MB in float32) with mean pooling. It supports more than 100 languages. On MMTEB it has the highest mean of the static models we compare, at 256 dimensions, outperforming much larger models. Static models are models without any active parameters; all model parameters are just embedding parameters. Because of this, they are many orders of magnitude faster than transformers. Although this comes at a cost, this cost is much lower than you would expect.

Usage

With model2vec:

from model2vec import StaticModel

model = StaticModel.from_pretrained("minishlab/blink-multilingual-64M")
embeddings = model.encode(["a first sentence", "een tweede zin"])

With model2vec-rs

use anyhow::Result;
use model2vec_rs::model::StaticModel;

fn main() -> Result<()> {
    let model = StaticModel::from_pretrained(
        "minishlab/blink-multilingual-64M",
    )?;

    let sentences = vec![
        "Hello world".to_string(),
        "Hallo wereld".to_string(),
    ];

    let embeddings = model.encode(&sentences);
    println!("Generated {} embeddings.", embeddings.len());

    Ok(())
}

With Sentence Transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("minishlab/blink-multilingual-64M")
embeddings = model.encode(["a first sentence", "een tweede zin"])

Results

MMTEB

MTEB(Multilingual, v2), all 131 tasks (mteb 2.27.1). Every model is evaluated at 256 dimensions, see Compared models. The last row is bekko's 1024-dimensional leaderboard entry, for reference.

Model Mean (task) Mean (type) Bitext Cls Clust InstrRerank MultilabelCls PairCls Rerank Retrieval STS
blink-multilingual-64M 51.60 45.11 57.03 52.11 39.67 -1.33 17.47 73.47 52.61 47.52 67.47
bekko-static-embedding-v1 48.37 41.64 37.30 50.41 39.84 -1.79 14.32 72.60 50.78 47.62 63.72
potion-multilingual-128M 47.67 40.59 40.72 52.34 37.88 -1.65 15.15 71.38 47.08 41.22 61.15
static-similarity-mrl-multilingual-v1 46.88 41.06 48.50 47.98 34.46 -0.86 14.29 73.16 48.83 39.49 63.67
bekko-static-embedding-v1 (1024d) 50.18 43.15 43.68 52.40 40.03 -1.65 14.99 72.81 51.59 49.96 64.50

The number of tasks per type: Bitext 13, Cls 43, Clust 16, InstrRerank 3, MultilabelCls 5, PairCls 11, Rerank 6, Retrieval 18, STS 16.

Summary

  • Mean: blink-multilingual-64M has the highest mean (51.60), 3.2 points above bekko at 256 dimensions (48.37) and 1.4 points above bekko at its full 1024 dimensions (50.18).
  • By type: it is best on 5 of 9 task types. It leads clearly on bitext mining (57.03, against at most 48.50) and STS (67.47, against at most 63.72).
  • Other types: bekko is slightly ahead on clustering (by 0.17) and retrieval (by 0.10), potion-multilingual-128M on classification (by 0.23), and static-similarity-mrl on instruction reranking.

Compared models

We compare blink-multilingual-64M to three other multilingual static embedding models:

Every model is evaluated at 256 dimensions. bekko-static-embedding-v1 and static-similarity-mrl-multilingual-v1 are truncated to their first 256 dimensions. All models use L2 normalized embeddings and a 32,768-token limit.

We also evaluate on the non-English part of MMTEB and on HAKARI retrieval below.

MMTEB without English

35 of MMTEB's 131 tasks (27%) are English-only, so English carries a large weight in the overall mean, and a model that is strong in English can look stronger in other languages than it is. This table leaves those 35 tasks out and shows the remaining 96 (multilingual tasks keep their English subsets). Every model is evaluated at 256 dimensions. Instruction reranking only has English tasks, so it drops out.

Model Mean (task) Mean (type) Bitext Cls Clust MultilabelCls PairCls Rerank Retrieval STS
blink-multilingual-64M 51.38 49.62 57.03 50.86 31.72 17.47 70.23 61.76 44.94 63.00
bekko-static-embedding-v1 47.00 45.23 37.30 49.23 32.55 14.32 69.01 59.43 42.24 57.77
potion-multilingual-128M 47.39 44.44 40.72 51.17 32.74 15.15 68.74 54.75 36.44 55.81
static-similarity-mrl-multilingual-v1 46.27 44.21 48.50 46.81 27.87 14.29 70.06 57.36 31.67 57.10

The number of tasks per type: Bitext 13, Cls 38, Clust 9, MultilabelCls 5, PairCls 9, Rerank 5, Retrieval 7, STS 10.

Summary

  • Mean: blink-multilingual-64M leads by 4.0 points (51.38, against 47.39 for potion-multilingual-128M and 47.00 for bekko). On the 35 English-only tasks, blink and bekko are tied (52.19 and 52.12).
  • By type: it is best on 6 of 8 types. potion-multilingual-128M is best on classification and clustering, by 0.31 and 1.02.
HAKARI retrieval

HAKARI-Bench has small ("Nano") versions of retrieval benchmarks. We use three: MNanoBEIR (NanoBEIR in 14 languages: Arabic, German, English, Spanish, French, Italian, Japanese, Korean, Norwegian, Portuguese, Serbian, Swedish, Thai and Vietnamese), its English part NanoBEIR-en, and NanoMIRACL. Retrieval uses cosine similarity over the full corpus of each task, with texts truncated to 512 tokens. All scores are nDCG@10, and every model is evaluated at 256 dimensions.

Model MNanoBEIR NanoBEIR-en NanoMIRACL
blink-multilingual-64M 41.18 47.57 65.41
bekko-static-embedding-v1 39.62 47.58 66.52
potion-multilingual-128M 31.83 36.67 47.77
static-similarity-mrl-multilingual-v1 29.98 41.27 42.66

Summary

  • MNanoBEIR: blink-multilingual-64M has the highest mean (41.18, against 39.62 for bekko).
  • English: blink and bekko are tied on NanoBEIR-en (47.57 and 47.58).
  • MIRACL: bekko is ahead (66.52, against 65.41). Both models are trained on MIRACL training data.

Training

We followed a six-step procedure.

1. Tokenizer

A 250k uncased WordPiece tokenizer, trained on the contrastive pair data and multilingual C4, with small scripts topped up so they get enough tokens. Every letter of 57 scripts is in the vocabulary, and long runs of characters are split into chunks, so text in these scripts never becomes [UNK]. A standard WordPiece tokenizer lost 10% of Thai and 23% of Lao characters to [UNK] on Flores; this one loses none. Since WordPiece also splits Chinese and Japanese into single characters, we add 20k Chinese and 10k Japanese words as whole tokens.

2. Initialization

Every token is embedded with bge-m3 using mean pooling. The vectors are reduced to 256 dimensions with PCA and weighted by measured token frequencies (SIF).

3. Distillation (Tokenlearn)

The model learns to reproduce bge-m3's sentence embeddings for 4M multilingual C4 texts from minishlab/tokenlearn-c4-multilingual-bge-m3, with a cosine loss.

4. Contrastive training

Following Hotchpotch's bekko recipe, we train with MNRL at a low temperature (0.05) on pairs from hotchpotch/bekko-embedding-v1-unsupervised: mMARCO, MIRACL, Wikipedia IR, SWIM-IR, NLI, FineWeb2-IR, CC-News, OPUS-100, WikiMatrix, NLLB English bitext and English pairs. We add NLLB English bitext for nine languages bekko's data does not cover (Amharic, Assamese, Faroese, Sorani, Kurmanji, Tswana, Zulu, Swati and Minangkabau), with religious sources removed.

  • 64M pairs, sampled over files with temperature α = 0.5, for 2 epochs.
  • Learning rate 0.05 with 10% warmup and cosine decay, batch 4096.
  • One data source per batch, no duplicates within a batch, and hard negatives where the data has them.
  • Trained natively at 256 dimensions.

5. Weight averaging

The final weights are 0.8 × the contrastive model from step 4 + 0.2 × the Tokenlearn model from step 3. This gives a modest boost in performance.

6. Language codes

Contrastive training on bitext pulls translations of the same sentence together, which also erases which language a text is in. Language codes restore part of it. Each language gets a fixed random unit vector, and each token embedding is shifted towards the codes of the languages it occurs in:

v_t += λ · median‖v‖ · Σ_lang p(lang | t) · code_lang

p(lang | t) is counted on NLLB text for 104 languages, and λ = 0.05. The codes are baked into the vectors, so the model stays a plain static embedding table. By doing this, language identification tasks go up (most notably, IndicLangClassification goes from 26.3 to 64.9) at a small cost on other tasks (−0.07 on average across the tasks that are not language identification).

Library Authors

Model2Vec was developed by the Minish team consisting of Stephan Tulkens and Thomas van Dongen.

Citation

Please cite the Model2Vec repository if you use this model in your work.

@software{minishlab2024model2vec,
  author       = {Stephan Tulkens and {van Dongen}, Thomas},
  title        = {Model2Vec: Fast State-of-the-Art Static Embeddings},
  year         = {2024},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.17270888},
  url          = {https://github.com/MinishLab/model2vec},
  license      = {MIT}
}
Downloads last month
5
Safetensors
Model size
64M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train minishlab/blink-multilingual-64M

Collection including minishlab/blink-multilingual-64M