README.md

---
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: feature-extraction
base_model: lightonai/DenseOn
base_model_relation: finetune
language:
- en
datasets:
- lightonai/embeddings-fine-tuning-filtered-en
tags:
- sparse
- sentence-transformers
- sentence-similarity
- custom-code
- sparse-retrieval
- matryoshka
---
<div align="center">
<img src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/5PySy2cTawrkFWwaqNtDA.png" alt="SlideOn: a puppy sliding through a library with a bag of glowing stars" width="512" style="max-width: 100%; height: auto;">

[![Website](https://img.shields.io/badge/LightOn-Website-blue?logo=google-chrome)](https://lighton.ai)
[![LinkedIn](https://img.shields.io/badge/LightOn-LinkedIn-0A66C2?logo=linkedin)](https://www.linkedin.com/company/lighton/)
[![X](https://img.shields.io/badge/@LightOnIO-X-black?logo=x)](https://x.com/LightOnIO)

📚 [Collection](https://huggingface.co/collections/lightonai/slideon-and-mslideon) | 📝 [Blogpost](https://huggingface.co/blog/lightonai/slideon-mslideon) | 🛠️ [Training and Evaluation](https://github.com/lightonai/slideon-mslideon/)

</div>

<h1 align="center">SlideOn</h1>
<h3 align="center">English Sparse Retrieval Model by LightOn</h3>

<p align="center">
<a href="https://huggingface.co/lightonai/SlideOn">SlideOn</a> |
<a href="https://huggingface.co/lightonai/mSlideOn">mSlideOn</a> |
<a href="https://huggingface.co/lightonai/SlideOn-SAE">SlideOn-SAE</a> |
<a href="https://huggingface.co/lightonai/mSlideOn-SAE">mSlideOn-SAE</a>
</p>

> 🎯 **TL;DR**: A **174M-parameter English sparse retrieval model** built from DenseOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. SlideOn reaches **56.68 nDCG@10 on BEIR-15**, surpassing DenseOn's 56.20, while producing sparse representations compatible with inverted-index search. Joint query/document **Matryoshka training** preserves retrieval quality under pruning, allowing search and storage costs to be adjusted at inference without retraining.

## About the SlideOn / mSlideOn Family

With [DenseOn and LateOn](https://huggingface.co/blog/lightonai/denseon-lateon), then [mDenseOn and mLateOn](https://huggingface.co/blog/lightonai/mdenseon-mlateon), we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

**SlideOn** starts from [DenseOn](https://huggingface.co/lightonai/DenseOn) for English retrieval. **[mSlideOn](https://huggingface.co/lightonai/mSlideOn)** starts from [mLateOn](https://huggingface.co/lightonai/mLateOn) for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our [blog post](https://huggingface.co/blog/lightonai/slideon-mslideon) for the full method, results, and ablations.

## SlideOn

SlideOn encodes queries and documents into **32,768-dimensional nonnegative sparse vectors**. A learned linear encoder maps each normalized DenseOn token representation to its top-32 latent activations, followed by ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the **dot product** between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

SlideOn notably:

- Reaches **56.68 on BEIR-15** with unpruned vectors, improving on DenseOn by **0.48 points**.
- Reaches **56.98** with unpruned queries and documents capped at **512 nonzero latents**.
- Retains **56.66** with queries capped at **64** and documents at **512**, reducing the measured retrieval FLOPS proxy to **8.72**.
- Supports inputs of up to **8,192 tokens** and independently adjustable query and document sparsity.

## Results

### Headline Results (BEIR-15, nDCG@10)

BEIR-15 scores are average nDCG@10 across 15 datasets, reported on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use **unpruned query and document vectors**. Dense and late-interaction results are taken from our [DenseOn and LateOn blog post](https://huggingface.co/blog/lightonai/denseon-lateon).

<div style="width: fit-content; max-width: 100%; margin: 0 auto;">
  <div style="overflow-x: auto;">
    <table style="margin: 0 auto; border-collapse: collapse; color: inherit; text-align: right; font-size: 0.85em; line-height: 1.6;">
      <thead>
        <tr style="border-bottom: 1px solid rgba(128, 128, 128, 0.35);">
          <th scope="col" style="padding: 8px 20px 8px 12px; text-align: left;">Model</th>
          <th scope="col" style="padding: 8px 16px;">Parameters</th>
          <th scope="col" style="padding: 8px 16px; white-space: nowrap;">BEIR-15 ↑</th>
          <th scope="col" style="padding: 8px 12px 8px 16px; white-space: nowrap;">Retrieval FLOPS ↓</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th colspan="4" scope="rowgroup" style="padding: 16px 12px 6px; text-align: left; font-size: 0.8em; letter-spacing: 0.08em; text-transform: uppercase;">Sparse</th>
        </tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left; color: #3b82f6;"><strong style="color: #3b82f6;">SlideOn</strong></td><td style="padding: 3px 16px; color: #3b82f6;">174M</td><td style="padding: 3px 16px; color: #3b82f6;">56.68</td><td style="padding: 3px 12px 3px 16px; color: #3b82f6;">30.87</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">LACONIC-8B</td><td style="padding: 3px 16px;">8.03B</td><td style="padding: 3px 16px;"><strong>60.06</strong></td><td style="padding: 3px 12px 3px 16px;">20.07</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">LACONIC-3B</td><td style="padding: 3px 16px;">3.21B</td><td style="padding: 3px 16px;">58.30</td><td style="padding: 3px 12px 3px 16px;">39.53</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">LACONIC-1B</td><td style="padding: 3px 16px;">1.24B</td><td style="padding: 3px 16px;">56.13</td><td style="padding: 3px 12px 3px 16px;">47.95</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">SPARSEUP</td><td style="padding: 3px 16px;">188M</td><td style="padding: 3px 16px;">54.46</td><td style="padding: 3px 12px 3px 16px;">6.20</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">Lion-SP-8B</td><td style="padding: 3px 16px;">8.0B</td><td style="padding: 3px 16px;">53.62</td><td style="padding: 3px 12px 3px 16px;">57.03</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">OpenSearch doc-v3-GTE</td><td style="padding: 3px 16px;">137M</td><td style="padding: 3px 16px;">52.68</td><td style="padding: 3px 12px 3px 16px;"><strong>0.72</strong></td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">OpenSearch v2-distill</td><td style="padding: 3px 16px;">67M</td><td style="padding: 3px 16px;">51.20</td><td style="padding: 3px 12px 3px 16px;">5.14</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">SPLADE-v3</td><td style="padding: 3px 16px;">133M</td><td style="padding: 3px 16px;">50.46</td><td style="padding: 3px 12px 3px 16px;">5.55</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">SPLADE-v3-DistilBERT</td><td style="padding: 3px 16px;">90M</td><td style="padding: 3px 16px;">49.04</td><td style="padding: 3px 12px 3px 16px;">5.79</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">Granite-30M</td><td style="padding: 3px 16px;">30M</td><td style="padding: 3px 16px;">48.86</td><td style="padding: 3px 12px 3px 16px;">6.42</td></tr>
        <tr><td style="padding: 3px 20px 12px 12px; text-align: left;">SAE-SPLADE</td><td style="padding: 3px 16px 12px;">117M</td><td style="padding: 3px 16px 12px;">48.21</td><td style="padding: 3px 12px 12px 16px;">2.67</td></tr>
      </tbody>
      <tbody style="border-top: 1px solid rgba(128, 128, 128, 0.25);">
        <tr>
          <th colspan="4" scope="rowgroup" style="padding: 16px 12px 6px; text-align: left; font-size: 0.8em; letter-spacing: 0.08em; text-transform: uppercase;">Dense</th>
        </tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left; color: #3b82f6;"><strong style="color: #3b82f6;">DenseOn</strong></td><td style="padding: 3px 16px; color: #3b82f6;">149M</td><td style="padding: 3px 16px; color: #3b82f6;"><strong style="color: #3b82f6;">56.20</strong></td><td style="padding: 3px 12px 3px 16px; color: #3b82f6;">—</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">gte-modernbert-base</td><td style="padding: 3px 16px;">149M</td><td style="padding: 3px 16px;">55.19</td><td style="padding: 3px 12px 3px 16px;">—</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">bge-large-en-v1.5</td><td style="padding: 3px 16px;">335M</td><td style="padding: 3px 16px;">54.34</td><td style="padding: 3px 12px 3px 16px;">—</td></tr>
        <tr><td style="padding: 3px 20px 12px 12px; text-align: left;">modernbert-embed-base</td><td style="padding: 3px 16px 12px;">149M</td><td style="padding: 3px 16px 12px;">52.89</td><td style="padding: 3px 12px 12px 16px;">—</td></tr>
      </tbody>
      <tbody style="border-top: 1px solid rgba(128, 128, 128, 0.25); border-bottom: 1px solid rgba(128, 128, 128, 0.35);">
        <tr>
          <th colspan="4" scope="rowgroup" style="padding: 16px 12px 6px; text-align: left; font-size: 0.8em; letter-spacing: 0.08em; text-transform: uppercase;">Late interaction</th>
        </tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left; color: #3b82f6;"><strong style="color: #3b82f6;">LateOn</strong></td><td style="padding: 3px 16px; color: #3b82f6;">149M</td><td style="padding: 3px 16px; color: #3b82f6;"><strong style="color: #3b82f6;">57.22</strong></td><td style="padding: 3px 12px 3px 16px; color: #3b82f6;">—</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">ColBERT-Zero</td><td style="padding: 3px 16px;">149M</td><td style="padding: 3px 16px;">55.39</td><td style="padding: 3px 12px 3px 16px;">—</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">GTE-ModernColBERT-v1</td><td style="padding: 3px 16px;">149M</td><td style="padding: 3px 16px;">54.75</td><td style="padding: 3px 12px 3px 16px;">—</td></tr>
        <tr><td style="padding: 3px 20px 3px 12px; text-align: left;">ColBERT-small</td><td style="padding: 3px 16px;">33M</td><td style="padding: 3px 16px;">53.79</td><td style="padding: 3px 12px 3px 16px;">—</td></tr>
        <tr><td style="padding: 3px 20px 12px 12px; text-align: left;">ColBERTv2</td><td style="padding: 3px 16px 12px;">110M</td><td style="padding: 3px 16px 12px;">48.63</td><td style="padding: 3px 12px 12px 16px;">—</td></tr>
      </tbody>
    </table>
  </div>

  <p style="max-width: 650px; margin: 14px auto 0; font-size: 0.85em; line-height: 1.5; text-align: center;">
  <em>Average nDCG@10 on BEIR-15 (↑ higher is better) and retrieval FLOPS (↓ lower is better). <span style="color: #3b82f6;">Blue</span> marks our models. <strong>Bold</strong> marks best per column within each family.</em>
</p>
</div>

SlideOn achieves the highest BEIR-15 score among the sparse models below 3B parameters in this comparison. It improves on SPARSEUP by **2.22 points** and on DenseOn by **0.48 points**, while remaining **0.54 points** behind LateOn. The figures below show how pruning changes the comparison when search and storage costs are taken into account.

### Retrieval Quality versus Search and Storage Cost

Query pruning reduces the number of posting lists consulted during search. Document pruning shortens those lists and reduces the number of stored document weights. Matryoshka training concentrates retrieval information in the strongest latents so that both budgets can be adjusted after training.

<div style="margin: 0 auto; text-align: center;">
  <img
    src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/biV_cKSk6mQfoo0x_GSSu.png"
    alt="BEIR-15 retrieval quality versus retrieval FLOPS for sparse models below 2B parameters, including SlideOn with different query pruning budgets."
    style="display: block; max-width: 100%; height: auto; margin: 0 auto;"
  />
  <p style="max-width: 750px; margin: 12px auto 0; font-size: 0.85em; line-height: 1.5;">
    <em>Retrieval quality versus search cost for SlideOn and sub-2B baselines. <span style="color: #3b82f6;">Blue</span> points show SlideOn with documents pruned to the top-512 latents and varying query budgets, where q-full denotes unpruned queries. Higher BEIR-15 ↑ scores and lower FLOPS ↓ are better. Retrieval FLOPS are measured on a subset of SciFact, NFCorpus, and FiQA.</em>
  </p>
</div>

<div style="margin: 0 auto; text-align: center;">
  <img
    src="https://cdn-uploads.huggingface.co/production/uploads/67e184d667350657a7d68aac/kzPT8yhDHf1lIUh2Z2_7V.png"
    alt="BEIR-15 retrieval quality versus average document NNZ for sparse models below 2B parameters, including SlideOn with different document pruning budgets."
    style="display: block; max-width: 100%; height: auto; margin: 0 auto;"
  />
  <p style="max-width: 750px; margin: 12px auto 0; font-size: 0.85em; line-height: 1.5;">
    <em>Retrieval quality versus index storage cost for SlideOn and sub-2B baselines. <span style="color: #3b82f6;">Blue</span> points show SlideOn with unpruned queries and varying document budgets, where d-full denotes unpruned documents. Higher BEIR-15 ↑ scores and lower NNZ ↓ are better. Document NNZ is measured on SciFact, NFCorpus, and FiQA.</em>
  </p>
</div>

**Retrieval FLOPS** estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. **Document NNZ** is the average number of nonzero document features and serves as a storage-cost proxy, rather than a measurement of index size in bytes. Efficiency measurements use the fixed sample of **6,000 documents and 1,271 queries** from SciFact, NFCorpus, and FiQA.

## Related Checkpoints

| Model | Description |
| --- | --- |
| **[SlideOn](https://huggingface.co/lightonai/SlideOn)** *(this model)* | English sparse retrieval model, after SAE pretraining and retrieval fine-tuning |
| [SlideOn-SAE](https://huggingface.co/lightonai/SlideOn-SAE) | DenseOn-based SAE initialization before retrieval fine-tuning |
| [mSlideOn](https://huggingface.co/lightonai/mSlideOn) | Multilingual sparse retrieval model for natural language, long documents, and code |
| [mSlideOn-SAE](https://huggingface.co/lightonai/mSlideOn-SAE) | mLateOn-based SAE initialization before retrieval fine-tuning |
| [DenseOn](https://huggingface.co/lightonai/DenseOn) | English dense retriever used to initialize SlideOn's backbone |

The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.

## Training Data and Recipe

See our blog post for [training data](https://huggingface.co/blog/lightonai/slideon-mslideon#training-data), [SAE pretraining](https://huggingface.co/blog/lightonai/slideon-mslideon#sparse-autoencoder-pretraining), and [supervised retrieval fine-tuning](https://huggingface.co/blog/lightonai/slideon-mslideon#supervised-fine-tuning-for-sparse-retrieval).

## Model Details

### Model Description

| Property | Value |
| --- | --- |
| Model type | Learned sparse retriever; Sentence Transformers `SparseEncoder` |
| Backbone | DenseOn / ModernBERT-base |
| Total parameters | 174M: 149M backbone + 25M sparse head |
| Maximum sequence length | 8,192 tokens, including prompts and special tokens |
| Output dimensionality | 32,768 learned latent features |
| Token representation | 768 dimensions, L2-normalized before sparse encoding |
| Token sparsity | Top-32 latent selection followed by ReLU |
| Sequence pooling | Maximum over token activations |
| Similarity function | Dot product |
| Prompts | `query: ` for queries; `document: ` for documents |
| Language | English |

## Usage

### Direct Usage (Sentence Transformers)

Install the supported library versions:

```bash
pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"
```

Load the model and encode queries and documents. This example uses the **q-64 / d-512** configuration:

```python
from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/SlideOn", trust_remote_code=True)

queries = ["What causes a solar eclipse?"]
documents = [
    "A solar eclipse occurs when the Moon passes between the Sun and Earth.",
    "Wind turbines convert the kinetic energy of wind into electricity.",
]

query_embeddings = model.encode_query(queries, max_active_dims=64)
document_embeddings = model.encode_document(documents, max_active_dims=512)

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([1, 32768]) torch.Size([2, 32768])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
```

`encode_query()` and `encode_document()` apply the appropriate prompts and preprocessing automatically. Pass the raw text without manually adding prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.

### Choosing Query and Document Budgets

Set `max_active_dims=N` to keep the **top-N latents with the highest weights** in each pooled vector and set the rest to zero. You can choose a different N for queries and documents. This is separate from the per-token top-32 operation and does not change the vector dimensionality.

Use a positive integer for a cap, or `None` for the saved unpruned defaults; `0` is invalid.

To reproduce the unpruned setting:

```python
query_embeddings = model.encode_query(queries, max_active_dims=None)
document_embeddings = model.encode_document(documents, max_active_dims=None)
```

Omitting `max_active_dims` also uses the saved unpruned defaults. To use the **q-full / d-512** setting, keep the query call at `max_active_dims=None` and set the document call to `max_active_dims=512`.

Dimensions represent **learned latents**, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; `SparseEncoder.decode()` cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.

## Citation

If you find this work useful for your research, please consider citing us:

```bibtex
@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}
```