---
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: feature-extraction
base_model: lightonai/mLateOn
base_model_relation: finetune
language:
- en
- fr
- de
- it
- es
- pt
- sv
- "no"
- ar
- code
datasets:
- lightonai/embeddings-fine-tuning-filtered-en
- lightonai/embeddings-fine-tuning-filtered-fr
- lightonai/embeddings-fine-tuning-filtered-de
- lightonai/embeddings-fine-tuning-filtered-it
- lightonai/embeddings-fine-tuning-filtered-es
- lightonai/embeddings-fine-tuning-filtered-pt
- lightonai/embeddings-fine-tuning-filtered-sv
- lightonai/embeddings-fine-tuning-filtered-no
- lightonai/embeddings-fine-tuning-filtered-ar
- lightonai/embeddings-fine-tuning-filtered-code
- lightonai/embeddings-fine-tuning-filtered-code-edit
tags:
- sparse
- sentence-transformers
- sentence-similarity
- custom-code
- multilingual
- code-search
- sparse-retrieval
- matryoshka
---
[](https://lighton.ai)
[](https://www.linkedin.com/company/lighton/)
[](https://x.com/LightOnIO)
π [Collection](https://huggingface.co/collections/lightonai/slideon-and-mslideon) | π [Blogpost](https://huggingface.co/blog/lightonai/slideon-mslideon) | π οΈ [Training and Evaluation](https://github.com/lightonai/slideon-mslideon/)
mSlideOn
Multilingual Sparse Retrieval Model by LightOn
SlideOn |
mSlideOn |
SlideOn-SAE |
mSlideOn-SAE
> π― **TL;DR**: A **339M-parameter multilingual sparse retrieval model** built from mLateOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. mSlideOn improves on mDenseOn by **4.76 points on MIRACL** and **10.19 points on MLDR**, and reaches **74.78 on MTEB Code**, surpassing both mDenseOn and mLateOn. It brings multilingual, long-context, and code retrieval to sparse representations compatible with inverted-index search, with **Matryoshka training** for adjustable query and document sparsity.
## About the SlideOn / mSlideOn Family
With [DenseOn and LateOn](https://huggingface.co/blog/lightonai/denseon-lateon), then [mDenseOn and mLateOn](https://huggingface.co/blog/lightonai/mdenseon-mlateon), we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.
**SlideOn** starts from [DenseOn](https://huggingface.co/lightonai/DenseOn) for English retrieval. **[mSlideOn](https://huggingface.co/lightonai/mSlideOn)** starts from [mLateOn](https://huggingface.co/lightonai/mLateOn) for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.
See our [blog post](https://huggingface.co/blog/lightonai/slideon-mslideon) for the full method, results, and ablations.
## mSlideOn
mSlideOn encodes queries and documents into **32,768-dimensional nonnegative sparse vectors**. Its sparse encoder lifts mLateOn's normalized 128-dimensional token representations to 768 dimensions and applies a residual MLP before selecting the top-32 latent activations and applying ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the **dot product** between these vectors.
The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.
mSlideOn notably:
- Reaches **56.00 on BEIR-15**, the highest score among the multilingual sparse models in our comparison.
- Reaches **63.21 on MIRACL target languages** and **62.78 across all 18 languages**, improving on mDenseOn by **4.76 points** on the full benchmark.
- Reaches **75.79 on MLDR target languages** and **61.68 across all 13 languages**, outperforming every dense baseline in our comparison.
- Reaches **74.78 on MTEB Code**, ahead of mDenseOn (**71.53**) and mLateOn (**73.48**).
- Supports inputs of up to **8,192 tokens** and independently adjustable query and document sparsity.
## Results
### Headline Results (nDCG@10)
Scores are average nDCG@10 on a 0β100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use **unpruned query and document vectors**. Dense and late-interaction comparisons follow our [mDenseOn and mLateOn blog post](https://huggingface.co/blog/lightonai/mdenseon-mlateon).
**MIRACL tgt** covers Arabic, English, French, German, and Spanish; **MLDR tgt** covers Arabic, English, French, German, Italian, Portuguese, and Spanish. These are benchmark languages included in retrieval training. Full **MIRACL** covers 18 languages, full **MLDR** covers 13, and **MTEB Code** averages 12 retrieval tasks.
We evaluate **MIRACL** using the dev split of [mteb/miracl-hard-negatives](https://huggingface.co/datasets/mteb/miracl-hard-negatives) and **MLDR** using the test split of [mteb/MultiLongDocRetrieval](https://huggingface.co/datasets/mteb/MultiLongDocRetrieval).
| Model |
Params. |
BEIR-15 β |
MIRACL tgt β |
MIRACL β |
MLDR tgt β |
MLDR β |
MTEB Code β |
FLOPS β |
| Sparse |
| mSlideOn |
339M |
56.00 |
63.21 |
62.78 |
75.79 |
61.68 |
74.78 |
44.88 |
| BGE-M3 | 568M | 40.20 | 46.55 | 55.48 | 71.82 | 62.13 | 42.03 | 1.69 |
| gte-multilingual-base | 305M | 40.27 | 48.74 | 57.49 | 80.19 | 70.75 | 51.09 | 1.70 |
| MILCO-300M | 330M | 53.33 | 64.11 | 70.05 | 62.74 | 51.78 | 41.25 | 17.43 |
| MILCO-650M | 593M | 52.12 | 64.14 | 70.02 | 68.63 | 58.30 | 33.06 | 13.96 |
| OpenSearch multilingual | 168M | 48.26 | 57.87 | 59.41 | 36.47 | 27.52 | 41.28 | 1.77 |
| Dense |
| mDenseOn |
307M |
56.70 |
59.61 |
58.02 |
65.62 |
51.50 |
71.53 |
β |
| pplx-embed-v1-0.6b | 596M | 56.70 | 63.18 | 68.20 | 57.08 | 43.98 | 75.18 | β |
| jina-v5-text-small | 677M | 56.70 | 61.03 | 66.56 | 53.00 | 43.83 | 73.01 | β |
| jina-v5-text-nano | 239M | 56.08 | 60.86 | 65.84 | 56.71 | 47.20 | 70.75 | β |
| arctic-embed-l-v2 | 568M | 55.55 | 60.78 | 66.53 | 57.01 | 48.03 | 53.17 | β |
| Qwen3-Embedding-0.6B | 596M | 55.33 | 57.64 | 60.62 | 59.11 | 50.06 | 73.75 | β |
| harrier-oss-v1-0.6b | 596M | 55.04 | 57.41 | 63.68 | 53.84 | 44.23 | 70.97 | β |
| harrier-oss-v1-270m | 268M | 50.82 | 48.95 | 57.11 | 52.31 | 42.63 | 63.79 | β |
| embeddinggemma-300m | 308M | 53.69 | 58.72 | 64.58 | 49.11 | 40.89 | 68.81 | β |
| gte-multilingual-base | 305M | 51.08 | 57.39 | 64.14 | 66.41 | 56.65 | 57.46 | β |
| voyage-4-nano | 346M | 49.96 | 49.29 | 58.65 | 62.10 | 51.97 | 76.43 | β |
| BGE-M3 | 568M | 48.78 | 62.22 | 69.62 | 61.16 | 52.47 | 51.49 | β |
| granite-311m-r2 | 312M | 49.24 | 53.38 | 59.74 | 50.53 | 41.86 | 63.50 | β |
| Late interaction |
| mLateOn |
312M |
57.56 |
65.61 |
67.04 |
87.69 |
77.92 |
73.48 |
β |
| pplx-embed-v1-late-0.6b | 596M | 56.11 | 63.67 | 69.55 | 68.69 | 55.51 | 63.29 | β |
| LFM2.5-ColBERT-350M | 353M | 54.50 | 57.40 | 42.45 | 78.39 | 61.05 | 51.62 | β |
| jina-colbert-v2 | 559M | 52.96 | 62.19 | 65.65 | 13.97 | 11.54 | 49.88 | β |
| GTE-ModernColBERT | 149M | 54.23 | 43.96 | 36.35 | 67.68 | 45.95 | 54.37 | β |
| ColBERT-Zero | 149M | 55.82 | 43.89 | 39.36 | 71.32 | 49.50 | 53.59 | β |
Retrieval results on BEIR-15, MIRACL, MLDR, and MTEB Code (β higher is better), with retrieval FLOPS (β lower is better). Blue denotes our models; bold values mark the best result within each family for each metric. OpenSearch multilingual corresponds to opensearch-neural-sparse-encoding-multilingual-v1.
**Retrieval FLOPS** estimates the expected number of shared active features per queryβdocument pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Measurements use the fixed sample of **6,000 documents and 1,271 queries** from SciFact, NFCorpus, and FiQA.
### Key Findings
**Multilingual and long-document retrieval.** mSlideOn improves over mDenseOn on full MIRACL and MLDR by **4.76** and **10.19 points**, respectively, while remaining behind mLateOn. The per-language results show gains beyond the languages included in retrieval training.
**Code retrieval.** mSlideOn outperforms every sparse and late-interaction baseline in this comparison on MTEB Code.
**English retrieval.** mSlideOn reaches 56.00 on BEIR-15, compared with 56.70 for mDenseOn and 57.56 for mLateOn. The gains on multilingual, long-document, and code tasks therefore come with a modest reduction in English benchmark performance.
Per-dataset and per-language differences in nDCG@10 points for mSlideOn versus mDenseOn. Teal bars indicate gains and red bars indicate declines relative to mDenseOn. Blue labels mark languages included in retrieval training.
## Related Checkpoints
| Model | Description |
| --- | --- |
| **[mSlideOn](https://huggingface.co/lightonai/mSlideOn)** *(this model)* | Multilingual sparse retrieval model for natural language, long documents, and code |
| [mSlideOn-SAE](https://huggingface.co/lightonai/mSlideOn-SAE) | mLateOn-based SAE initialization before retrieval fine-tuning |
| [SlideOn](https://huggingface.co/lightonai/SlideOn) | English sparse retrieval model, after SAE pretraining and retrieval fine-tuning |
| [SlideOn-SAE](https://huggingface.co/lightonai/SlideOn-SAE) | DenseOn-based SAE initialization before retrieval fine-tuning |
| [mLateOn](https://huggingface.co/lightonai/mLateOn) | Multilingual late-interaction retriever used to initialize mSlideOn's backbone |
The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.
## Training Data and Recipe
See our blog post for [training data](https://huggingface.co/blog/lightonai/slideon-mslideon#training-data), [SAE pretraining](https://huggingface.co/blog/lightonai/slideon-mslideon#sparse-autoencoder-pretraining), and [supervised retrieval fine-tuning](https://huggingface.co/blog/lightonai/slideon-mslideon#supervised-fine-tuning-for-sparse-retrieval).
## Model Details
### Model Description
| Property | Value |
| --- | --- |
| Model type | Learned sparse retriever; Sentence Transformers `SparseEncoder` |
| Backbone | mLateOn / mmBERT-base |
| Total parameters | 339M |
| Maximum sequence length | 8,192 tokens, including special tokens and markers |
| Output dimensionality | 32,768 learned latent features |
| Token representation | 128 dimensions, L2-normalized before sparse encoding |
| Sparse encoder | Lift to 768 dimensions, residual MLP, and latent projection |
| Token sparsity | Top-32 latent selection followed by ReLU |
| Sequence pooling | Maximum over token activations |
| Similarity function | Dot product |
| Prompts | `[Q]` marker for queries; `[D]` marker for documents, inserted automatically after the beginning-of-sequence token. No `query:` / `document:` text prefixes. |
| Retrieval training languages | English, French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, Arabic, plus code |
## Usage
### Direct Usage (Sentence Transformers)
Install the supported library versions:
```bash
pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"
```
Load the model and encode multilingual queries and documents:
```python
from sentence_transformers import SparseEncoder
model = SparseEncoder("lightonai/mSlideOn", trust_remote_code=True)
queries = [
"Qu'est-ce qui provoque une Γ©clipse solaire ?",
"Was verursacht eine Sonnenfinsternis?",
]
documents = [
"A solar eclipse occurs when the Moon passes between the Sun and Earth.",
"Um eclipse solar ocorre quando a Lua passa entre o Sol e a Terra.",
"Wind turbines convert the kinetic energy of wind into electricity.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([2, 32768]) torch.Size([3, 32768])
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
```
`encode_query()` inserts the `[Q]` query marker, and `encode_document()` inserts the `[D]` document marker, immediately after the beginning-of-sequence token. Pass raw text; the model adds these special tokens automatically and does not use `query:` or `document:` text prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.
### Choosing Query and Document Budgets
Set `max_active_dims=N` to keep the **top-N latents with the highest weights** in each pooled vector and set the rest to zero. You can choose a different N for queries and documents. This is separate from the per-token top-32 operation and does not change the vector dimensionality. For example, to use **q-128 / d-1024**:
```python
query_embeddings = model.encode_query(queries, max_active_dims=128)
document_embeddings = model.encode_document(documents, max_active_dims=1024)
```
Omitting `max_active_dims`, or setting it to `None`, uses unpruned vectors, matching the headline results.
Pruning budgets must be positive integers; `0` is invalid.
Dimensions represent **learned latents**, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; `SparseEncoder.decode()` cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.
## Citation
If you find this work useful for your research, please consider citing us:
```bibtex
@misc{moura2026slideon,
author = {Moura Junior, Paulo Roberto and Sourty, RaphaΓ«l},
title = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
year = {2026},
howpublished = {Hugging Face Blog},
url = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}
```