bekko-static-embedding-v1: State-of-the-Art Retrieval for Multilingual Static Embeddings

I've released bekko-static-embedding-v1, a multilingual static embedding model. It achieves state-of-the-art retrieval among static embeddings: about 11 points above the other multilingual static models on multilingual retrieval, and about 2 points above English-only static models on English retrieval.

Static embeddings skip the Transformer: each token maps to a fixed vector, and a sentence embedding is their average. They need no GPU and can run 100x to 400x faster on CPU than models like multilingual-e5-small.

bekko-static-embedding-v1 uses a custom 200k-token tokenizer and was trained contrastively on about 814 million text pairs covering 40 languages, with a batch size of 16,384.

It also performs close to Transformer models. On MMTEB's retrieval tasks, bekko-static-embedding-v1 comes within about 1 point of multilingual-e5-small, a 12-layer Transformer model.

Performance comparison

Against other static embedding models

bekko-static-embedding-v1 leads every public static model I compared, on both multilingual and English retrieval:

• Multilingual retrieval. On MNanoBEIR (14 languages × 13 tasks, nDCG@10 × 100) it scores 43.76, about 11 points above static-similarity-mrl-multilingual-v1 (32.29) and potion-multilingual-128M (32.23).
• English retrieval. On NanoBEIR-en (13 tasks) it scores 53.71, above the English-only stable-static-embedding-fast-retrieval-mrl-en-v2 (SST-v2 below, 51.57) and static-retrieval-mrl-en-v1 (50.35). Truncated to 512 dimensions (52.82), it still beats both.
• MMTEB. Across 131 tasks the average is 50.18, about 3 points above both multilingual static models. In the Retrieval category the margins are 8.75 and 12.10 points.
• Matryoshka dimensions. Truncating to 512 dimensions costs about 0.9 points of average retrieval score.

Table 1: Public static embedding models compared. NanoBEIR-en averages 13 English retrieval tasks, MNanoBEIR 182 retrieval tasks in 14 languages, and MMTEB 131 tasks including classification and clustering. Retrieval columns are nDCG@10 × 100; MMTEB is each task's main score × 100.

Against contextual models

How close can a model with no Transformer layers get to a Transformer encoder? A natural reference is multilingual-e5-small, a 12-layer encoder with 21.6M active parameters that is a common pick for compact multilingual retrieval.

Table 2: Comparison with contextual models. NanoBEIR-en and MNanoBEIR are HAKARI-Bench nDCG@10 × 100. MMTEB values for multilingual-e5-small and bekko-embedding-v1-a8m come from the bekko-embedding-v1-a8m model card, which uses the official MMTEB snapshot with the same 131 tasks and aggregation. Active parameters exclude the token embedding table.

On MMTEB Retrieval, bekko-static-embedding-v1 comes within about 1 point of multilingual-e5-small (49.96 vs 50.9) without running a single Transformer layer. multilingual-e5-small is still ahead on the Nano retrieval sets, and on the MMTEB mean, where most of its lead comes from BitextMining, Reranking and STS.

If you can afford a small Transformer, my bekko-embedding-v1-a8m scores above multilingual-e5-small in every column with only 7.7M active parameters, and it runs on CPU. BM25 also remains a strong retrieval baseline; I compare it in detail below.

I trained bekko-static-embedding-v1, and I also develop HAKARI-Bench, the retrieval benchmark used in this post.

How static embeddings work

A static embedding model splits text into tokens with a tokenizer and gives each token a fixed vector. Encoding a sentence is just a table lookup for each token followed by a mean. With no Transformer layers to run, vectors come out fast.

Usage with Sentence Transformers

The model loads directly with Sentence Transformers. This example encodes "sushi is delicious" in English, French, Spanish and Chinese.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("hotchpotch/bekko-static-embedding-v1", device="cpu")
vectors = model.encode(
    [
        "Sushi is delicious.",
        "Les sushis sont délicieux.",
        "El sushi es delicioso.",
        "寿司很好吃。",
    ],
    normalize_embeddings=True,
)
print(vectors.shape)  # (4, 1024)
print(model.similarity(vectors, vectors))

Queries and documents are encoded the same way; no query: or passage: prefixes are needed. For shorter vectors, pass truncate_dim (32, 64, 128, 256, 512 or 1024):

compact = model.encode(["A short query"], truncate_dim=512, normalize_embeddings=True)

Results on MMTEB

MMTEB (Massive Multilingual Text Embedding Benchmark) covers retrieval, classification, clustering, semantic textual similarity (STS) and more. I evaluated the 131 tasks of MTEB(Multilingual, v2) at the full 1,024 dimensions.

The comparison models are static-similarity-mrl-multilingual-v1 from Sentence Transformers and potion-multilingual-128M from Model2Vec. For them I used published evaluation results and averaged over the same 131 tasks.

Table 3: MMTEB main score × 100, averaged per category. Δ is bekko-static-embedding-v1 minus static-similarity-mrl-multilingual-v1.

The overall average is 50.18 for bekko-static-embedding-v1, about 3 points above static-similarity-mrl-multilingual-v1 (47.24) and potion-multilingual-128M (47.23).

Retrieval is where the gap is consistently large. bekko-static-embedding-v1 scores 49.96, 8.75 points above static-similarity-mrl-multilingual-v1 and 12.10 points above potion-multilingual-128M. Multilingual retrieval is this model's strength.

Against static-similarity-mrl-multilingual-v1 it wins 6 of 9 categories. The largest category gain is Clustering (+9.36), while STS is about even (+0.47).

static-similarity-mrl-multilingual-v1 is ahead on BitextMining, PairClassification and InstructionReranking. The BitextMining gap is the largest at 6.94 points (43.68 vs 50.62). If bitext mining or pair classification is your main use case, include static-similarity-mrl-multilingual-v1 in your evaluation.

English retrieval on HAKARI-Bench

I measured retrieval with the Nano datasets of HAKARI-Bench, which are smaller evaluation sets built from existing retrieval benchmarks. All HAKARI-Bench numbers below are nDCG@10 × 100 averaged with equal weight per task.

For English I used NanoBEIR-en: 13 tasks and 649 queries. The static comparison models are the English retrieval models static-retrieval-mrl-en-v1 and SST-v2. For reference I also include two contextual models, multilingual-e5-small and my own small bekko-embedding-v1-a8m, plus BM25, which ranks by word overlap and term frequency.

Table 4: NanoBEIR-en task average. bekko-static-embedding-v1 is shown at 1,024 and 512 dimensions; the other embedding models at their default dimensions.

At 1,024 dimensions bekko-static-embedding-v1 scores 53.71, above SST-v2 (51.57) and static-retrieval-mrl-en-v1 (50.35). At 512 dimensions it is still 1.25 points above SST-v2 at the same output width and wins 10 of 13 tasks.

The embedding tables differ in size, though: about 205M parameters for bekko-static-embedding-v1 against about 15.6M for SST-v2.

On English retrieval alone, bekko-embedding-v1-a8m (60.34), multilingual-e5-small (58.37) and BM25 (57.15) remain ahead of every static model.

Multilingual retrieval and BM25

For multilingual retrieval I compared static-similarity-mrl-multilingual-v1, potion-multilingual-128M, multilingual-e5-small, bekko-embedding-v1-a8m and BM25. MNanoBEIR has 14 languages × 13 tasks, 182 tasks in total. HAKARI-Bench covers 37 benchmarks and 550 tasks, including code, long-document and Japanese retrieval; the table shows 10 of them.

Table 5: Task average within each benchmark, nDCG@10 × 100. Columns: bekko-static-embedding-v1 at 1,024d and 512d, static-similarity-mrl-multilingual-v1 (1,024d), potion-multilingual-128M (256d), multilingual-e5-small (384d), bekko-embedding-v1-a8m (384d) and BM25. NanoMMTEB-v2 is HAKARI-Bench's retrieval subset, separate from the full MMTEB above.

bekko-static-embedding-v1 at 1,024 dimensions beats static-similarity-mrl-multilingual-v1 on the average of all 37 benchmarks, and on 523 of 550 individual tasks.

Against potion-multilingual-128M it wins 36 of 37 benchmark averages and 520 of 550 tasks. The exception is NanoIndicQA, where potion-multilingual-128M scores 53.69 against 35.35.

The contextual models are still ahead. multilingual-e5-small beats bekko-static-embedding-v1 on 28 of 37 benchmark averages and bekko-embedding-v1-a8m on 36; on MNanoBEIR they score 51.17 and 52.65.

On the long-document benchmarks NanoMLDR and NanoLongEmbed, bekko-static-embedding-v1 beats multilingual-e5-small, partly because multilingual-e5-small reads at most 512 tokens. Against bekko-embedding-v1-a8m, which reads up to 8,192 tokens, NanoLongEmbed goes to bekko-embedding-v1-a8m (68.19), while bekko-static-embedding-v1 is slightly ahead on NanoMLDR (55.72 vs 54.48).

Should you use a static embedding for retrieval?

If retrieval is all you need, BM25 is often enough. In this evaluation BM25 beats bekko-static-embedding-v1 on 21 of 37 benchmark averages and on 342 of 550 tasks. It wins by a wide margin on the long-document NanoMLDR and NanoLongEmbed and on Japanese NanoJMTEB-v2.

On NanoMIRACL, where natural-language questions are matched to documents, the static model wins: 68.03 for bekko-static-embedding-v1 against 57.15 for BM25, a 10.88-point gap. For some retrieval tasks, a static embedding is the better choice.

BM25 relies on matching words, so embedding search can help when queries and documents phrase the same thing differently. The reliable way to choose is to try both on your own corpus.

Matryoshka dimensions and INT8 weights

I trained 1,024, 512, 256, 128, 64 and 32 dimensions together with Matryoshka Representation Learning (MRL), which teaches the model to keep useful information in the leading dimensions. You can pick a vector length per use case and cut storage and search cost.

Besides FP16, I also provide INT8 weights. Here INT8 means the model weights are stored as 8-bit integers; the output vectors are not quantized.

Table 6: Retrieval scores by Matryoshka dimension and weight precision, nDCG@10 × 100. Overall is the 550-task average. INT8 results use ONNX.

Going from 1,024 to 512 dimensions halves the vector length and lowers the score by about 0.9 points on both NanoBEIR-en and MNanoBEIR. At 256 dimensions the drop widens to 2.88 and 2.72 points. Even so, at 128 dimensions the MNanoBEIR score of 38.03 is still above static-similarity-mrl-multilingual-v1 at 1,024 dimensions (32.29). INT8 and FP16 differ by less than 0.15 points at every dimension.

I would start with 512 dimensions, and pick INT8 at 512 or 256 dimensions when memory is tight.

Loading only the weights you need

Sentence Transformers' truncate_dim truncates the output vectors, but it still loads the full 1,024-dimensional table. If you only ever use 512 or 256 dimensions, that memory is wasted.

So the repository also ships ONNX and OpenVINO files that contain only the weights for one dimension. The model card provides a small get_model() helper to pick the backend, dimension and precision:

from transformers.dynamic_module_utils import get_class_from_dynamic_module


def get_model(backend="onnx", dim=512, precision="int8", revision="main"):
    repo = "hotchpotch/bekko-static-embedding-v1"
    ext = {"onnx": "onnx", "openvino": "xml"}[backend]
    return get_class_from_dynamic_module(
        "bekko_static_backend.BekkoStaticBackend", repo, revision=revision,
    ).load_sentence_transformer(
        repo, revision=revision, backend=backend,
        file_name=f"{backend}/model_{precision}_dim{dim}.{ext}",
    )


model = get_model(backend="onnx", dim=512, precision="int8")

Table 7: Model file size per dimension (decimal MB). ONNX and OpenVINO files are the same size; the tokenizer (about 5.1 MB) is not included.

INT8 at 512 dimensions is 103.2 MB, about a quarter of the 409.6 MB FP16 model at 1,024 dimensions. At 256 dimensions it shrinks to 52.0 MB.

Which model should you pick?

Based on these results, this is how I would choose:

• For cheap multilingual embeddings, start with bekko-static-embedding-v1 at 512 dimensions.
• If retrieval is the only goal, compare it against BM25 with a tokenizer suited to your language.
• For bitext mining or pair classification, also evaluate static-similarity-mrl-multilingual-v1.
• If retrieval quality matters most and you can afford Transformer inference, consider multilingual-e5-small or bekko-embedding.

bekko-embedding is a contextual embedding family that uses Transformer attention, unlike bekko-static-embedding-v1. Its Transformer part is kept small: bekko-embedding-v1-a8m has about 7.7M active parameters and runs on CPU. In this comparison it scores 60.34 on NanoBEIR-en and 52.65 on MNanoBEIR, both above multilingual-e5-small. If you want a balance of retrieval quality and inference speed, bekko-embedding is worth a look.

How I trained it

A custom tokenizer

In a static model, the tokenizer's units are exactly the units whose vectors get averaged. So besides the existing vocabularies of multilingual BERT and Gemma 2, I tried vocabularies fitted to the training data. bert-base-multilingual-uncased, used by static-similarity-mrl-multilingual-v1, is among the baselines.

The custom vocabularies use WordPiece. I first trained a 30k English vocabulary on equal numbers of Wikipedia and FineWeb documents. I then added vocabulary for 39 more languages while keeping the English token IDs, for 200k tokens in total.

To choose, I trained 8 models, one per tokenizer, on 10% of the training data. I evaluated them on 200 retrieval tasks: MNanoBEIR (14 languages × 13 tasks) and NanoMIRACL (18 languages), nDCG@10 × 100 averaged with equal weight.

"uncased" ignores letter case and "cased" keeps it. "script-balanced" is a candidate that rebalances vocabulary across writing systems. "200k, 30k English" means 200k tokens in total with a 30k English core reserved first.

Table 8: Retrieval score per tokenizer, nDCG@10 × 100. All 8 models share the same data and training settings. This 10% experiment used a different contrastive loss from the final model, so its scores are not directly comparable with the final model's.

The best was the 200k uncased vocabulary with a 30k English core. Its 41.89 at 1,024 dimensions beats multilingual BERT uncased (37.87) and Gemma 2 (38.73), so the final model uses the same tokenizer.

Training data and hyperparameters

Training uses the bekko-embedding unsupervised dataset, the multilingual corpus described in the Bekko Embedding paper.

From its roughly 1.15 billion rows, I selected 265 data configurations targeting 40 languages, about 814 million text pairs in total.

The 40 target languages are English, Arabic, Bulgarian, Bengali, Catalan, Czech, Danish, German, Greek, Spanish, Estonian, Persian, Finnish, French, Hebrew, Hindi, Croatian, Hungarian, Indonesian, Italian, Japanese, Korean, Dutch, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Serbian, Swedish, Swahili, Telugu, Thai, Turkish, Ukrainian, Vietnamese, Yoruba, Chinese.

The pairs include titles and bodies, questions and answers, generated search queries and translation pairs. For translation data I kept NLLB English HQ and dropped the other bitext sources. The selection also includes data derived from the MIRACL train split and from NLI.

The embedding table starts from random initialization and learns to pull matching pairs together. The loss is Multiple Negatives Ranking Loss (MNRL), which contrasts each pair against the other candidates in the batch, applied at the full 1,024 dimensions and at every Matryoshka dimension. There is no distillation from a teacher model.

Table 9: Training settings of the released model.

Reproduction scripts

The model repository includes scripts for tokenizer training, model training and FP16 export. They come with the training and data-selection configs, so you can follow the whole path from data selection to the FP16 release.

References

• Yuichi Tateno, Bekko Embedding: Parameter-Efficient Multilingual Retrieval with Ultra-Compact Encoders, 2026. The contextual Bekko Embedding paper and the source of the training data.
• Yuichi Tateno, HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions, 2026.
• Kenneth Enevoldsen et al., MMTEB: Massive Multilingual Text Embedding Benchmark, 2025.
• Aditya Kusupati et al., Matryoshka Representation Learning, 2022.
• Nandan Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021.
• Xinyu Zhang et al., Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages, 2023.
• Stephan Tulkens and Thomas van Dongen, Model2Vec: Fast State-of-the-Art Static Embeddings, 2024.
• Minish Lab, potion-multilingual-128M.
• Tom Aarsen, Train 400x faster Static Embedding Models with Sentence Transformers.
• RikkaBotan, Stable Static Embedding technical report and SSE Retrieval MRL v2 technical report.

Explore the results

All HAKARI-Bench results are on the leaderboard. The link filters to multilingual-e5-small, bekko-embedding-v1-a8m and every model with static in its name.