| --- |
| license: apache-2.0 |
| language: |
| - multilingual |
| - en |
| - de |
| - fr |
| - es |
| - pt |
| - it |
| - nl |
| - pl |
| - ru |
| - uk |
| - cs |
| - ro |
| - bg |
| - sv |
| - da |
| - 'no' |
| - fi |
| - hu |
| - el |
| - tr |
| - ar |
| - he |
| - fa |
| - ur |
| - hi |
| - bn |
| - ta |
| - te |
| - mr |
| - zh |
| - ja |
| - ko |
| - vi |
| - th |
| - id |
| - ms |
| - sw |
| - af |
| - am |
| - hy |
| - az |
| - eu |
| - ka |
| - kk |
| - km |
| library_name: sentence-transformers |
| pipeline_tag: sentence-similarity |
| base_model: jhu-clsp/mmBERT-base |
| datasets: |
| - HuggingFaceFW/fineweb-2 |
| - HuggingFaceFW/fineweb |
| tags: |
| - sentence-transformers |
| - feature-extraction |
| - sentence-similarity |
| - embeddings |
| - retrieval |
| - multilingual |
| - bge-m3 |
| - distillation |
| - modernbert |
| - mmbert |
| - onnx |
| - transformers.js |
| widget: |
| - source_sentence: "How tall is the Eiffel Tower?" |
| sentences: |
| - "La tour Eiffel mesure 330 mètres." |
| - "Der Eiffelturm ist das höchste Bauwerk von Paris." |
| - "The Statue of Liberty is 93 metres tall." |
| - "Ich esse gern Pizza." |
| example_title: "Cross-lingual" |
| --- |
| |
| # Multilingual Embedding (base, 308M) - bge-m3 compatible |
|
|
| A multilingual sentence and passage embedding model distilled from [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) into |
| [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base). It outputs 1024-dimensional vectors **in bge-m3's dense vector |
| space**, so you can: |
|
|
| - use it on its own for search, RAG, clustering and similarity in about 90 languages, 2x smaller than bge-m3; |
| - or **query an existing bge-m3 index** with it: embed queries with this model and keep the passages you already embedded |
| with bge-m3 (see the "→ bge-m3 index" row below). That gives cheap query encoding on CPU or in the browser. |
|
|
| A smaller, faster version is available as [Horizon-Labs/multilingual-embedding-small](https://huggingface.co/Horizon-Labs/multilingual-embedding-small). Apache-2.0. ONNX files for CPU and the browser (transformers.js) are included. |
|
|
| - Vectors are L2-normalised. Use cosine similarity (= dot product). No query/passage prefixes are needed. |
| - `onnx/model_quantized.onnx` (int8 embeddings, 642 MB; cosine >= 0.99997 to the fp32 model on test sentences) outputs the final normalised embedding as `sentence_embedding`. |
|
|
| ## Usage |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("Horizon-Labs/multilingual-embedding-base") |
| q = model.encode(["How tall is the Eiffel Tower?"]) |
| p = model.encode(["La tour Eiffel mesure 330 mètres.", "The Statue of Liberty is 93 metres tall."]) |
| print(q @ p.T) # higher = more similar |
| |
| # Querying an existing bge-m3 index: passages embedded earlier with BAAI/bge-m3 (dense, normalised) |
| # scores = model.encode(queries) @ bge_m3_passage_vectors.T |
| ``` |
|
|
| transformers.js: |
|
|
| ```js |
| import { AutoTokenizer, AutoModel } from "@huggingface/transformers"; |
| const tok = await AutoTokenizer.from_pretrained("Horizon-Labs/multilingual-embedding-base"); |
| const model = await AutoModel.from_pretrained("Horizon-Labs/multilingual-embedding-base", { dtype: "q8" }); |
| const { sentence_embedding } = await model(await tok(["Wie hoch ist der Eiffelturm?"], { padding: true, truncation: true })); |
| ``` |
|
|
| ## Evaluation |
|
|
| nDCG@10 for ranking each query's candidates by cosine similarity, on public reranking benchmarks (MTEB versions; used |
| only for evaluation). MIRACL: 60 queries per language with 100 candidates each. Wikipedia: 60 queries per language with 9 |
| candidates each. "Other" = ESCI (es, jp, us), RuBQ, T2Reranking, mMARCO-ja and AskUbuntu. Same script for every model |
| (`code/`); multilingual-e5 is run with its "query: "/"passage: " prefixes. |
|
|
| | model | MIRACL (18 languages) | Wikipedia (16 languages) | other (6 sets) | mean | note | |
| |---|---|---|---|---|---| |
| | **this model** (308M) | 0.753 | 0.934 | 0.777 | 0.821 | queries and passages embedded by this model | |
| | **this model** → bge-m3 index (308M) | 0.742 | 0.934 | 0.774 | 0.817 | queries by this model, passages by bge-m3 | |
| | [Horizon-Labs/multilingual-embedding-small](https://huggingface.co/Horizon-Labs/multilingual-embedding-small) (141M) | 0.750 | 0.935 | 0.768 | 0.818 | | |
| | [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) (568M) | 0.797 | 0.927 | 0.788 | 0.837 | the teacher | |
| | [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small) (118M) | 0.721 | 0.915 | 0.776 | 0.804 | | |
| | [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) (278M) | 0.736 | 0.918 | 0.775 | 0.810 | | |
| | [sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) (118M) | 0.387 | 0.837 | 0.687 | 0.637 | | |
| | [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) (23M) | 0.172 | 0.720 | 0.621 | 0.504 | English | |
|
|
| - The table shows the released checkpoint. Means over two training seeds: small own .817 / against a bge-m3 index .818; base own .822 / against a bge-m3 index .816. |
| - Models ahead of this one (own embeddings): MIRACL: bge-m3; Wikipedia: none; |
| other: bge-m3. |
| - This is a reranking-style test over given candidate lists, not a full-corpus retrieval benchmark. Rerankers on the same test: |
| [leaderboard](https://huggingface.co/spaces/Horizon-Labs/multilingual-retrieval-leaderboard). |
|
|
| ## Training |
|
|
| - **Distillation**: the student (mmBERT-base + mean pooling + a linear layer to 1024 dimensions) learns to reproduce bge-m3's |
| normalised dense embeddings: cosine + squared-error loss, plus a loss on the in-batch similarity matrix that keeps the |
| relative geometry. |
| - **Text**: about 4.65 million texts: snippets of 1-8 sentences and short spans from [FineWeb-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) |
| and [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) (ODC-BY) in 92 languages, plus about 850k search queries |
| written by Qwen3.8-27B (Apache-2.0) for web passages in 47 languages. |
| - Code: `code/` in this repository. |
|
|
| ## Limitations |
|
|
| - The student approximates bge-m3 and is weaker than it. The gap is larger for long passages, rare languages and |
| specialised domains. |
| - Only bge-m3's **dense** vectors are reproduced, not its sparse or multi-vector (ColBERT) outputs. |
| - Max length 512 tokens (trained with 256). Split long documents into passages. |
|
|
