mSlideOn / README.md
Paulo de Moura
Declare Apache 2.0 model license
ef80160 about 23 hours ago
21.3 kB
---
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: feature-extraction
base_model: lightonai/mLateOn
base_model_relation: finetune
language:
- en
- fr
- de
- it
- es
- pt
- sv
- "no"
- ar
- code
datasets:
- lightonai/embeddings-fine-tuning-filtered-en
- lightonai/embeddings-fine-tuning-filtered-fr
- lightonai/embeddings-fine-tuning-filtered-de
- lightonai/embeddings-fine-tuning-filtered-it
- lightonai/embeddings-fine-tuning-filtered-es
- lightonai/embeddings-fine-tuning-filtered-pt
- lightonai/embeddings-fine-tuning-filtered-sv
- lightonai/embeddings-fine-tuning-filtered-no
- lightonai/embeddings-fine-tuning-filtered-ar
- lightonai/embeddings-fine-tuning-filtered-code
- lightonai/embeddings-fine-tuning-filtered-code-edit
tags:
- sparse
- sentence-transformers
- sentence-similarity
- custom-code
- multilingual
- code-search
- sparse-retrieval
- matryoshka
---
<div align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/5PySy2cTawrkFWwaqNtDA.png" alt="SlideOn and mSlideOn: a puppy sliding through a library with a bag of glowing stars" width="512" style="max-width: 100%; height: auto;">

[![Website](https://img.shields.io/badge/LightOn-Website-blue?logo=google-chrome)](https://lighton.ai)
[![LinkedIn](https://img.shields.io/badge/LightOn-LinkedIn-0A66C2?logo=linkedin)](https://www.linkedin.com/company/lighton/)
[![X](https://img.shields.io/badge/@LightOnIO-X-black?logo=x)](https://x.com/LightOnIO)

📚 [Collection](https://huggingface.co/collections/lightonai/slideon-and-mslideon) | 📝 [Blogpost](https://huggingface.co/blog/lightonai/slideon-mslideon) | 🛠️ [Training and Evaluation](https://github.com/lightonai/slideon-mslideon/)

</div>

<h1 align="center">mSlideOn</h1>
<h3 align="center">Multilingual Sparse Retrieval Model by LightOn</h3>

<p align="center">
<a href="https://huggingface.co/lightonai/SlideOn">SlideOn</a> |
<a href="https://huggingface.co/lightonai/mSlideOn">mSlideOn</a> |
<a href="https://huggingface.co/lightonai/SlideOn-SAE">SlideOn-SAE</a> |
<a href="https://huggingface.co/lightonai/mSlideOn-SAE">mSlideOn-SAE</a>
</p>

> 🎯 **TL;DR**: A **339M-parameter multilingual sparse retrieval model** built from mLateOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. mSlideOn improves on mDenseOn by **4.76 points on MIRACL** and **10.19 points on MLDR**, and reaches **74.78 on MTEB Code**, surpassing both mDenseOn and mLateOn. It brings multilingual, long-context, and code retrieval to sparse representations compatible with inverted-index search, with **Matryoshka training** for adjustable query and document sparsity.

## About the SlideOn / mSlideOn Family

With [DenseOn and LateOn](https://huggingface.co/blog/lightonai/denseon-lateon), then [mDenseOn and mLateOn](https://huggingface.co/blog/lightonai/mdenseon-mlateon), we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

**SlideOn** starts from [DenseOn](https://huggingface.co/lightonai/DenseOn) for English retrieval. **[mSlideOn](https://huggingface.co/lightonai/mSlideOn)** starts from [mLateOn](https://huggingface.co/lightonai/mLateOn) for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our [blog post](https://huggingface.co/blog/lightonai/slideon-mslideon) for the full method, results, and ablations.

## mSlideOn

mSlideOn encodes queries and documents into **32,768-dimensional nonnegative sparse vectors**. Its sparse encoder lifts mLateOn's normalized 128-dimensional token representations to 768 dimensions and applies a residual MLP before selecting the top-32 latent activations and applying ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the **dot product** between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

mSlideOn notably:

- Reaches **56.00 on BEIR-15**, the highest score among the multilingual sparse models in our comparison.
- Reaches **63.21 on MIRACL target languages** and **62.78 across all 18 languages**, improving on mDenseOn by **4.76 points** on the full benchmark.
- Reaches **75.79 on MLDR target languages** and **61.68 across all 13 languages**, outperforming every dense baseline in our comparison.
- Reaches **74.78 on MTEB Code**, ahead of mDenseOn (**71.53**) and mLateOn (**73.48**).
- Supports inputs of up to **8,192 tokens** and independently adjustable query and document sparsity.

## Results

### Headline Results (nDCG@10)

Scores are average nDCG@10 on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use **unpruned query and document vectors**. Dense and late-interaction comparisons follow our [mDenseOn and mLateOn blog post](https://huggingface.co/blog/lightonai/mdenseon-mlateon).

**MIRACL tgt** covers Arabic, English, French, German, and Spanish; **MLDR tgt** covers Arabic, English, French, German, Italian, Portuguese, and Spanish. These are benchmark languages included in retrieval training. Full **MIRACL** covers 18 languages, full **MLDR** covers 13, and **MTEB Code** averages 12 retrieval tasks.

We evaluate **MIRACL** using the dev split of [mteb/miracl-hard-negatives](https://huggingface.co/datasets/mteb/miracl-hard-negatives) and **MLDR** using the test split of [mteb/MultiLongDocRetrieval](https://huggingface.co/datasets/mteb/MultiLongDocRetrieval).

<div style="width: fit-content; max-width: 100%; margin: 0 auto;">
  <div style="overflow-x: auto;">
    <table cellpadding="6" style="margin: 0 auto; border-collapse: collapse; color: inherit; text-align: right; font-size: 0.85em; line-height: 1.6; font-variant-numeric: tabular-nums;">
      <thead>
        <tr style="border-bottom: 1px solid rgba(128, 128, 128, 0.35);">
          <th scope="col" style="text-align: left;">Model</th>
          <th scope="col">Params.</th>
          <th scope="col">BEIR-15 ↑</th>
          <th scope="col">MIRACL tgt ↑</th>
          <th scope="col">MIRACL ↑</th>
          <th scope="col">MLDR tgt ↑</th>
          <th scope="col">MLDR ↑</th>
          <th scope="col">MTEB Code ↑</th>
          <th scope="col">FLOPS ↓</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th colspan="9" scope="rowgroup" style="padding: 16px 6px 6px; text-align: left; font-size: 0.8em; letter-spacing: 0.08em; text-transform: uppercase;">Sparse</th>
        </tr>
        <tr>
          <td style="text-align: left; white-space: nowrap;"><strong style="color: #3b82f6;">mSlideOn</strong></td>
          <td style="color: #3b82f6;">339M</td>
          <td><strong style="color: #3b82f6;">56.00</strong></td>
          <td style="color: #3b82f6;">63.21</td>
          <td style="color: #3b82f6;">62.78</td>
          <td style="color: #3b82f6;">75.79</td>
          <td style="color: #3b82f6;">61.68</td>
          <td><strong style="color: #3b82f6;">74.78</strong></td>
          <td style="color: #3b82f6;">44.88</td>
        </tr>
        <tr><td style="text-align: left; white-space: nowrap;">BGE-M3</td><td>568M</td><td>40.20</td><td>46.55</td><td>55.48</td><td>71.82</td><td>62.13</td><td>42.03</td><td><strong>1.69</strong></td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">gte-multilingual-base</td><td>305M</td><td>40.27</td><td>48.74</td><td>57.49</td><td><strong>80.19</strong></td><td><strong>70.75</strong></td><td>51.09</td><td>1.70</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">MILCO-300M</td><td>330M</td><td>53.33</td><td>64.11</td><td><strong>70.05</strong></td><td>62.74</td><td>51.78</td><td>41.25</td><td>17.43</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">MILCO-650M</td><td>593M</td><td>52.12</td><td><strong>64.14</strong></td><td>70.02</td><td>68.63</td><td>58.30</td><td>33.06</td><td>13.96</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">OpenSearch multilingual</td><td>168M</td><td>48.26</td><td>57.87</td><td>59.41</td><td>36.47</td><td>27.52</td><td>41.28</td><td>1.77</td></tr>
      </tbody>
      <tbody style="border-top: 1px solid rgba(128, 128, 128, 0.25);">
        <tr>
          <th colspan="9" scope="rowgroup" style="padding: 16px 6px 6px; text-align: left; font-size: 0.8em; letter-spacing: 0.08em; text-transform: uppercase;">Dense</th>
        </tr>
        <tr>
          <td style="text-align: left; white-space: nowrap;"><strong style="color: #3b82f6;">mDenseOn</strong></td>
          <td style="color: #3b82f6;">307M</td>
          <td><strong style="color: #3b82f6;">56.70</strong></td>
          <td style="color: #3b82f6;">59.61</td>
          <td style="color: #3b82f6;">58.02</td>
          <td style="color: #3b82f6;">65.62</td>
          <td style="color: #3b82f6;">51.50</td>
          <td style="color: #3b82f6;">71.53</td>
          <td style="color: #3b82f6;">—</td>
        </tr>
        <tr><td style="text-align: left; white-space: nowrap;">pplx-embed-v1-0.6b</td><td>596M</td><td><strong>56.70</strong></td><td><strong>63.18</strong></td><td>68.20</td><td>57.08</td><td>43.98</td><td>75.18</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">jina-v5-text-small</td><td>677M</td><td><strong>56.70</strong></td><td>61.03</td><td>66.56</td><td>53.00</td><td>43.83</td><td>73.01</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">jina-v5-text-nano</td><td>239M</td><td>56.08</td><td>60.86</td><td>65.84</td><td>56.71</td><td>47.20</td><td>70.75</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">arctic-embed-l-v2</td><td>568M</td><td>55.55</td><td>60.78</td><td>66.53</td><td>57.01</td><td>48.03</td><td>53.17</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">Qwen3-Embedding-0.6B</td><td>596M</td><td>55.33</td><td>57.64</td><td>60.62</td><td>59.11</td><td>50.06</td><td>73.75</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">harrier-oss-v1-0.6b</td><td>596M</td><td>55.04</td><td>57.41</td><td>63.68</td><td>53.84</td><td>44.23</td><td>70.97</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">harrier-oss-v1-270m</td><td>268M</td><td>50.82</td><td>48.95</td><td>57.11</td><td>52.31</td><td>42.63</td><td>63.79</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">embeddinggemma-300m</td><td>308M</td><td>53.69</td><td>58.72</td><td>64.58</td><td>49.11</td><td>40.89</td><td>68.81</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">gte-multilingual-base</td><td>305M</td><td>51.08</td><td>57.39</td><td>64.14</td><td><strong>66.41</strong></td><td><strong>56.65</strong></td><td>57.46</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">voyage-4-nano</td><td>346M</td><td>49.96</td><td>49.29</td><td>58.65</td><td>62.10</td><td>51.97</td><td><strong>76.43</strong></td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">BGE-M3</td><td>568M</td><td>48.78</td><td>62.22</td><td><strong>69.62</strong></td><td>61.16</td><td>52.47</td><td>51.49</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">granite-311m-r2</td><td>312M</td><td>49.24</td><td>53.38</td><td>59.74</td><td>50.53</td><td>41.86</td><td>63.50</td><td>—</td></tr>
      </tbody>
      <tbody style="border-top: 1px solid rgba(128, 128, 128, 0.25); border-bottom: 1px solid rgba(128, 128, 128, 0.35);">
        <tr>
          <th colspan="9" scope="rowgroup" style="padding: 16px 6px 6px; text-align: left; font-size: 0.8em; letter-spacing: 0.08em; text-transform: uppercase;">Late interaction</th>
        </tr>
        <tr>
          <td style="text-align: left; white-space: nowrap;"><strong style="color: #3b82f6;">mLateOn</strong></td>
          <td style="color: #3b82f6;">312M</td>
          <td><strong style="color: #3b82f6;">57.56</strong></td>
          <td><strong style="color: #3b82f6;">65.61</strong></td>
          <td style="color: #3b82f6;">67.04</td>
          <td><strong style="color: #3b82f6;">87.69</strong></td>
          <td><strong style="color: #3b82f6;">77.92</strong></td>
          <td><strong style="color: #3b82f6;">73.48</strong></td>
          <td style="color: #3b82f6;">—</td>
        </tr>
        <tr><td style="text-align: left; white-space: nowrap;">pplx-embed-v1-late-0.6b</td><td>596M</td><td>56.11</td><td>63.67</td><td><strong>69.55</strong></td><td>68.69</td><td>55.51</td><td>63.29</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">LFM2.5-ColBERT-350M</td><td>353M</td><td>54.50</td><td>57.40</td><td>42.45</td><td>78.39</td><td>61.05</td><td>51.62</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">jina-colbert-v2</td><td>559M</td><td>52.96</td><td>62.19</td><td>65.65</td><td>13.97</td><td>11.54</td><td>49.88</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">GTE-ModernColBERT</td><td>149M</td><td>54.23</td><td>43.96</td><td>36.35</td><td>67.68</td><td>45.95</td><td>54.37</td><td>—</td></tr>
        <tr><td style="text-align: left; white-space: nowrap;">ColBERT-Zero</td><td>149M</td><td>55.82</td><td>43.89</td><td>39.36</td><td>71.32</td><td>49.50</td><td>53.59</td><td>—</td></tr>
      </tbody>
    </table>
  </div>
  <p style="max-width: 750px; margin: 14px auto 0; font-size: 0.85em; line-height: 1.5; text-align: center;">
    <em>Retrieval results on BEIR-15, MIRACL, MLDR, and MTEB Code (↑ higher is better), with retrieval FLOPS (↓ lower is better). <span style="color: #3b82f6;">Blue</span> denotes our models; <strong>bold</strong> values mark the best result within each family for each metric. OpenSearch multilingual corresponds to opensearch-neural-sparse-encoding-multilingual-v1.</em>
  </p>
</div>
**Retrieval FLOPS** estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Measurements use the fixed sample of **6,000 documents and 1,271 queries** from SciFact, NFCorpus, and FiQA.

### Key Findings

**Multilingual and long-document retrieval.** mSlideOn improves over mDenseOn on full MIRACL and MLDR by **4.76** and **10.19 points**, respectively, while remaining behind mLateOn. The per-language results show gains beyond the languages included in retrieval training.

**Code retrieval.** mSlideOn outperforms every sparse and late-interaction baseline in this comparison on MTEB Code.

**English retrieval.** mSlideOn reaches 56.00 on BEIR-15, compared with 56.70 for mDenseOn and 57.56 for mLateOn. The gains on multilingual, long-document, and code tasks therefore come with a modest reduction in English benchmark performance.

<div style="margin: 0 auto; text-align: center;">
  <img
    src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/fGrj4Ki_9AM0f7FjMJqyl.png"
    alt="Per-dataset and per-language performance differences between mSlideOn and mDenseOn on BEIR-15, MIRACL, MLDR, and MTEB Code."
    style="display: block; max-width: 100%; height: auto; margin: 0 auto;"
  />
  <p style="max-width: 750px; margin: 12px auto 0; font-size: 0.85em; line-height: 1.5;">
    <em>Per-dataset and per-language differences in nDCG@10 points for mSlideOn versus mDenseOn. <span style="color: #46a59b;">Teal</span> bars indicate gains and <span style="color: #ff4b57;">red</span> bars indicate declines relative to mDenseOn. <span style="color: #3b82f6;">Blue labels</span> mark languages included in retrieval training.</em>
  </p>
</div>
## Related Checkpoints

| Model | Description |
| --- | --- |
| **[mSlideOn](https://huggingface.co/lightonai/mSlideOn)** *(this model)* | Multilingual sparse retrieval model for natural language, long documents, and code |
| [mSlideOn-SAE](https://huggingface.co/lightonai/mSlideOn-SAE) | mLateOn-based SAE initialization before retrieval fine-tuning |
| [SlideOn](https://huggingface.co/lightonai/SlideOn) | English sparse retrieval model, after SAE pretraining and retrieval fine-tuning |
| [SlideOn-SAE](https://huggingface.co/lightonai/SlideOn-SAE) | DenseOn-based SAE initialization before retrieval fine-tuning |
| [mLateOn](https://huggingface.co/lightonai/mLateOn) | Multilingual late-interaction retriever used to initialize mSlideOn's backbone |

The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.

## Training Data and Recipe

See our blog post for [training data](https://huggingface.co/blog/lightonai/slideon-mslideon#training-data), [SAE pretraining](https://huggingface.co/blog/lightonai/slideon-mslideon#sparse-autoencoder-pretraining), and [supervised retrieval fine-tuning](https://huggingface.co/blog/lightonai/slideon-mslideon#supervised-fine-tuning-for-sparse-retrieval).

## Model Details

### Model Description

| Property | Value |
| --- | --- |
| Model type | Learned sparse retriever; Sentence Transformers `SparseEncoder` |
| Backbone | mLateOn / mmBERT-base |
| Total parameters | 339M |
| Maximum sequence length | 8,192 tokens, including special tokens and markers |
| Output dimensionality | 32,768 learned latent features |
| Token representation | 128 dimensions, L2-normalized before sparse encoding |
| Sparse encoder | Lift to 768 dimensions, residual MLP, and latent projection |
| Token sparsity | Top-32 latent selection followed by ReLU |
| Sequence pooling | Maximum over token activations |
| Similarity function | Dot product |
| Prompts | `[Q]` marker for queries; `[D]` marker for documents, inserted automatically after the beginning-of-sequence token. No `query:` / `document:` text prefixes. |
| Retrieval training languages | English, French, German, Italian, Spanish, Portuguese, Swedish, Norwegian, Arabic, plus code |

## Usage

### Direct Usage (Sentence Transformers)

Install the supported library versions:

```bash
pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"
```

Load the model and encode multilingual queries and documents:

```python
from sentence_transformers import SparseEncoder
model = SparseEncoder("lightonai/mSlideOn", trust_remote_code=True)
queries = [
    "Qu'est-ce qui provoque une éclipse solaire ?",
    "Was verursacht eine Sonnenfinsternis?",
]
documents = [
    "A solar eclipse occurs when the Moon passes between the Sun and Earth.",
    "Um eclipse solar ocorre quando a Lua passa entre o Sol e a Terra.",
    "Wind turbines convert the kinetic energy of wind into electricity.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([2, 32768]) torch.Size([3, 32768])
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
```

`encode_query()` inserts the `[Q]` query marker, and `encode_document()` inserts the `[D]` document marker, immediately after the beginning-of-sequence token. Pass raw text; the model adds these special tokens automatically and does not use `query:` or `document:` text prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.

### Choosing Query and Document Budgets

Set `max_active_dims=N` to keep the **top-N latents with the highest weights** in each pooled vector and set the rest to zero. You can choose a different N for queries and documents. This is separate from the per-token top-32 operation and does not change the vector dimensionality. For example, to use **q-128 / d-1024**:

```python
query_embeddings = model.encode_query(queries, max_active_dims=128)
document_embeddings = model.encode_document(documents, max_active_dims=1024)
```

Omitting `max_active_dims`, or setting it to `None`, uses unpruned vectors, matching the headline results.

Pruning budgets must be positive integers; `0` is invalid.

Dimensions represent **learned latents**, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; `SparseEncoder.decode()` cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.

## Citation

If you find this work useful for your research, please consider citing us:

```bibtex
@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}
```