SlideOn & mSlideOn: Sparsified Late-Interaction and Dense Retrievers

SlideOn | mSlideOn | Collection

We are releasing SlideOn and mSlideOn, two open state-of-the-art sparse retrieval models for English and multilingual retrieval, with 174M and 339M parameters, respectively. Through SAE pretraining and supervised fine-tuning, we build SlideOn from DenseOn and mSlideOn from mLateOn, reusing their existing retrieval knowledge to learn sparse representations compatible with efficient inverted-index search. SlideOn reaches 56.68 on BEIR-15, surpassing DenseOn’s 56.20, and uses sparse vector pruning to achieve leading performance on the search and storage efficiency Pareto frontiers among sub-2B models. mSlideOn supports multilingual, long-context, and code retrieval, outperforming mDenseOn by +4.76 points on multilingual retrieval (MIRACL) and +10.19 points on long-document retrieval (MLDR), while also surpassing mLateOn and mDenseOn on code retrieval (MTEB Code). mSlideOn brings the multilingual and long-context strengths of late interaction to efficient sparse retrieval, with strong generalization beyond its retrieval training languages. With Matryoshka-style training, both models maintain strong retrieval performance as pruning increases, allowing users to adjust them to fit their latency and storage budgets. As we always do, we release the SAE checkpoints, fine-tuned retrieval models, and training code under a permissive license to support further research.

Get started now:

• 🗂️ Collection: SlideOn & mSlideOn
• 🌐 Multilingual model: mSlideOn
• 🔤 English model: SlideOn
• 🧩 SAE pretrained checkpoints: SlideOn-SAE | mSlideOn-SAE
• 📚 Training data: EN | FR | ES | IT | DE | PT | AR | NO | SV | Code | Code Edit
• 🛠️ Training and evaluation scripts: slideon-mslideon

Introduction

Over the past few months, we released DenseOn and LateOn, our open dense and late-interaction models for English retrieval, then extended our recipe to multilingual, long-context and code retrieval to build mDenseOn and mLateOn. These releases validated our data recipe and training setup while enabling a controlled comparison of single-vector and multi-vector retrieval under a matched setup. Alongside this work, we explored how to regularize late-interaction models to work better with efficient ANN indexes such as SMVE/MUVERA (LateOn-regularized) and how to reduce its inference costs via hierarchical pooling training (LateOn-hpool-regularized).

The limitations of dense and late-interaction retrievers led us to take a closer look at sparse retrieval, a direction we had not yet explored in depth. Sparse retrieval represents each query and document as a vector in which most values are zero. The remaining nonzero are positive values and assign weights to a small set of features that we call latents, which may represent vocabulary terms or learned concepts.

An inverted index stores, for each latent, a list of documents where that latent was activated, called posting-lists. Search can then focus only on documents that share active latents with the query without scanning the entire corpus. This property can be particularly valuable for large collections, where reducing the amount of data accessed per query enables efficient search across billions of documents.

The SPLADE family (1,2,3,4,5,6) has long demonstrated that learned sparse retrieval (LSR) enables semantic matching with efficient inverted-index search. In SPLADE models, each vector component corresponds to a vocabulary term (“dog”, “cat”, “cow”), so the model learns vocabulary weights and which additional terms to activate beyond those present in the input tokens. This expansion helps address the vocabulary mismatch that affects lexical methods such as BM25, allowing related wording such as “car” and “automobile” to match. The resulting representations retain interpretable term weights while supporting compact indexes and low retrieval latency. LACONIC demonstrates that sparse retrievers running index search on CPU can achieve superior quality with similar latency compared to dense baselines that rely on GPUs for fast search, which are the industry standard now. They also show that depending on document sparsity, sparse retrieval can use substantially less index memory than dense vectors. In parallel, more recent work highlights the value of sparse representations for accelerating multi-vector retrieval. SMVE uses sparse representations to retrieve candidates before full late-interaction reranking, while SSR uses sparse coding and inverted indexes to replace expensive k-means clustering in multi-vector retrieval. Beyond these efficiency benefits, sparse representations may also offer advantages for long-context retrieval by easing the compression bottleneck of encoding an entire document into a single low-dimensional dense vector, as discussed here.

SPARSEUP is a notable recent contribution that pushes the frontier of retrieval quality and efficiency, achieving state-of-the-art BEIR-13 results under 150M parameters while supporting sub-millisecond index search on MS MARCO with the SEISMIC efficient sparse index. Built over LateOn-unsupervised checkpoint and trained on our open English fine-tuning mixture, it demonstrates how existing checkpoints and data can be reused to train competitive sparse retrievers. These results encouraged us to build on our previous work and investigate how to convert dense and late-interaction models into efficient state-of-the-art sparse retrievers.

Recent studies demonstrate that sparse autoencoders (SAEs) can be used as effective initialization for sparse retrievers. SAE-SPLADE demonstrates that SPLADE’s MLM-initialized sparse vocabulary can be replaced with a learned latent vocabulary while maintaining comparable retrieval performance. The latent vocabulary is learned through SAE pretraining, then fine-tuned with the backbone for retrieval. SPLARE explores the same principle with LLM backbones, reusing publicly available pretrained SAEs and fine-tuning only the backbone. It achieves results competitive with state-of-the-art dense retrievers, alongside strong cross-lingual generalization. Latent Terms shows that effective sparse retrieval features compatible with BM25 can be extracted from trained dense and late-interaction models with SAE training only. PUMA explores a supervised approach for multimodal embeddings, combining SAE reconstruction with retrieval fine-tuning of the sparse vocabulary while keeping the pretrained backbone frozen.

Compared with sparse representations learned from a tokenizer vocabulary, these SAE-learned latent vocabularies offer some advantages. Features can capture shared concepts across different expressions, distinguish different meanings of an ambiguous word, and align semantically related content across languages. Their dimensionality can also be chosen independently of the tokenizer vocabulary, giving more flexibility over representational capacity. These advantages motivated us to explore SAEs as a way to convert existing dense and late-interaction representations into sparse features while reusing their learned retrieval knowledge.

SAE pretraining and supervised fine-tuning are computationally much cheaper than large-scale retrieval pretraining, and the community now offers a large collection of openly released dense models alongside a growing set of strong late-interaction models. In this work, we therefore explore how to convert our matched late-interaction and dense retrievers into efficient learned sparse retrievers (LSRs) through SAE pretraining and supervised fine-tuning, building on their existing representations and our validated data recipe.

Training Data

We reuse our multilingual supervised fine-tuning data released with mDenseOn and mLateOn. We prioritize comparability with our dense and late-interaction models over broader language and script coverage, keeping the same training data and teacher distillation setup. The mixture contains about 16.3M query-positive pairs across nine natural languages and code. Most multilingual examples come from translations of our curated English datasets, supplemented with native multilingual supervision from only 4 of MIRACL’s 16 training language splits: Arabic, English, French, and Spanish. The mixture also includes long-document pairs from 7 of MLDR’s 13 training language splits and code retrieval data from LateOn-Code and CodeEditSearchTrain.

Per-dataset language mix and total number of query-positive pairs in the filtered contrastive fine-tuning data.

In this dataset, each query-positive pair contains 10 mined hard negatives. The positive and negative documents are annotated with relevance scores from the mxbai-rerank-large-v2 cross-encoder, providing teacher supervision for retrieval training. For English-only models, we select the English splits of this mixture.

Sparse Autoencoder Pretraining

Our initial goal was to convert our trained dense and late-interaction models into sparse retrievers by training only an SAE on their frozen representations, following Latent Terms. However, we found that explicitly optimizing these sparse representations for retrieval could bring substantial improvements, as we show in our SAE/SFT ablations. We therefore use SAE pretraining to learn an initial latent dictionary before adapting it through supervised sparse retrieval fine-tuning.

An SAE learns to reconstruct dense vectors using a small number of learned features. We train ours on individual L2-normalized token embeddings, using dense 768-dimensional or late-interaction 128-dimensional unpooled token representations. The encoder maps each embedding $x\in {\mathbb{R}}^{d}x\; \backslash in\; \backslash mathbb\{R\}^\{d\}$ to a sparse latent vector $h=\mathrm{e}\mathrm{n}\mathrm{c}(x)\in {\mathbb{R}}^{D}h=\backslash mathrm\{enc\}(x)\backslash in\backslash mathbb\{R\}^\{D\}$, where $D\gg dD\; \backslash gg\; d$ and most components are zero. The decoder combines the active latents to reconstruct the original embedding, producing $x^{\mathrm{\prime }}=\mathrm{d}\mathrm{e}\mathrm{c}(h)\in {\mathbb{R}}^{d}x\text{'}=\backslash mathrm\{dec\}(h)\backslash in\backslash mathbb\{R\}^\{d\}$. Training minimizes a reconstruction error, typically $\parallel x-x^{\mathrm{\prime }}{\parallel }_2^2\backslash lVert\; x-x\text{'}\backslash rVert\_2^2$, encouraging the sparse representation to preserve information from the pretrained model. To enforce sparsity of vectors $hh$, we use Top-K SAE setup, where only the top-k components of $hh$ are kept while the rest is set to 0. We additionally apply ReLU on $hh$, providing an extra sparsity mechanism to the SAE encoder. This choice is motivated by SAE-SPLADE and Latent Terms work, whose ablations support Top-K as a simple and effective choice for retrieval, with no consistent advantage from other variants. We use $k=32k=32$ and $D=\mathrm{32,768}D=32\{,\}768$ to balance retrieval quality, sparsity, and encoder size. We observed that increasing $kk$ from 16 to 32 improved retrieval despite higher mean NNZ (which we better control during SFT), while 64 brought no further gains. Similarly, $D=32,768D=32,768$ latents outperformed 16,384, while expanding to 65,536 increases encoder size without justifiable gains.

SAE Architecture for Dense Models (SlideOn)

For our SAE started from dense models, we use a single linear encoder projection $W_{\mathrm{e}\mathrm{n}\mathrm{c}}W\_\{\backslash mathrm\{enc\}\}$, together with a linear decoder $W_{\mathrm{d}\mathrm{e}\mathrm{c}}W\_\{\backslash mathrm\{dec\}\}$. The learned centering offset $b_{\mathrm{p}\mathrm{r}\mathrm{e}}b\_\{\backslash mathrm\{pre\}\}$ is subtracted before encoding and added back during decoding, allowing the SAE to represent variation around a shared baseline. The encoder bias $b_{\mathrm{e}\mathrm{n}\mathrm{c}}b\_\{\backslash mathrm\{enc\}\}$ adjusts the preactivation of each latent. $$h=\mathrm{e}\mathrm{n}\mathrm{c}(x)=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}({\mathrm{T}\mathrm{o}\mathrm{p}\mathrm{K}}_k(W_{\mathrm{e}\mathrm{n}\mathrm{c}}(x-b_{\mathrm{p}\mathrm{r}\mathrm{e}})+b_{\mathrm{e}\mathrm{n}\mathrm{c}}))h\; =\; \backslash mathrm\{enc\}(x)\; =\; \backslash mathrm\{ReLU\}(\backslash mathrm\{TopK\}\_k(W\_\{\backslash mathrm\{enc\}\}(x\; -\; b\_\{\backslash mathrm\{pre\}\})\; +\; b\_\{\backslash mathrm\{enc\}\}))$$ $$x^{\mathrm{\prime }}=\mathrm{d}\mathrm{e}\mathrm{c}(h)=W_{\mathrm{d}\mathrm{e}\mathrm{c}}h+b_{\mathrm{p}\mathrm{r}\mathrm{e}}x\text{'}\; =\; \backslash mathrm\{dec\}(h)\; =\; W\_\{\backslash mathrm\{dec\}\}h\; +\; b\_\{\backslash mathrm\{pre\}\}$$ We initialize the columns of $W_{\mathrm{d}\mathrm{e}\mathrm{c}}W\_\{\backslash mathrm\{dec\}\}$ uniformly sampling distinct token embeddings from our training data and set $W_{\mathrm{e}\mathrm{n}\mathrm{c}}=W_{\mathrm{d}\mathrm{e}\mathrm{c}}^{\mathrm{\top }}W\_\{\backslash mathrm\{enc\}\}=W\_\{\backslash mathrm\{dec\}\}^\backslash top$. The encoder bias $b_{\mathrm{e}\mathrm{n}\mathrm{c}}b\_\{\backslash mathrm\{enc\}\}$ starts at zero, while $b_{\mathrm{p}\mathrm{r}\mathrm{e}}b\_\{\backslash mathrm\{pre\}\}$ is initialized with the embeddings mean of the first training batch. Encoder and decoder weights are mirrored only at initialization and are optimized independently afterward, with L2-normalized $W_{\mathrm{d}\mathrm{e}\mathrm{c}}W\_\{\backslash mathrm\{dec\}\}$ rows so that the sparse activations carry the scaling by themselves.

SAE Architecture for Late-Interaction Models (mSlideOn)

For our SAE started from late-interaction models, the encoder architecture is a bit more complex. We first lift the smaller 128-dim token representations to 768-dim and apply a residual MLP $768\to 1536\to 768768\backslash rightarrow1536\backslash rightarrow768$ with ReLU activations. This gives the encoder more capacity to match dense SAE which naturally have more parameters. The decoder remains linear and reconstructs the original 128-dim embedding.

$$u=W_{\mathrm{l}\mathrm{i}\mathrm{f}\mathrm{t}}(x-b_{\mathrm{p}\mathrm{r}\mathrm{e}})u\; =\; W\_\{\backslash mathrm\{lift\}\}(x\; -\; b\_\{\backslash mathrm\{pre\}\})$$

$$v=u+MLP(u)v\; =\; u\; +\; MLP(u)$$

$$h=\mathrm{e}\mathrm{n}\mathrm{c}(x)=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}({\mathrm{T}\mathrm{o}\mathrm{p}\mathrm{K}}_k(W_{\mathrm{e}\mathrm{n}\mathrm{c}}v+b_{\mathrm{e}\mathrm{n}\mathrm{c}}))h\; =\; \backslash mathrm\{enc\}(x)\; =\; \backslash mathrm\{ReLU\}(\backslash mathrm\{TopK\}\_k(W\_\{\backslash mathrm\{enc\}\}v\; +\; b\_\{\backslash mathrm\{enc\}\}))$$

$$x^{\mathrm{\prime }}=\mathrm{d}\mathrm{e}\mathrm{c}(h)=W_{\mathrm{d}\mathrm{e}\mathrm{c}}h+b_{\mathrm{p}\mathrm{r}\mathrm{e}}x\text{'}\; =\; \backslash mathrm\{dec\}(h)\; =\; W\_\{\backslash mathrm\{dec\}\}h\; +\; b\_\{\backslash mathrm\{pre\}\}$$

We initialize $W_{\mathrm{l}\mathrm{i}\mathrm{f}\mathrm{t}}W\_\{\backslash mathrm\{lift\}\}$ with random orthonormal columns. We initialize the columns of $W_{\mathrm{d}\mathrm{e}\mathrm{c}}W\_\{\backslash mathrm\{dec\}\}$ from sampled token embeddings after subtracting their sample mean and L2-normalizing them. This centering step is specific to the late-interaction dictionary initialization, motivated by the higher cosine similarity between randomly sampled token embeddings (0.175 for DenseOn vs. 0.959 for LateOn on FiQA). Subtracting the mean helps the dictionary capture variation around this shared component. We then set $W_{\mathrm{e}\mathrm{n}\mathrm{c}}=(W_{\mathrm{l}\mathrm{i}\mathrm{f}\mathrm{t}}{W}_{\mathrm{d}\mathrm{e}\mathrm{c}}{)}^{\mathrm{\top }}W\_\{\backslash mathrm\{enc\}\}=(W\_\{\backslash mathrm\{lift\}\}W\_\{\backslash mathrm\{dec\}\})^\backslash top$ and initialize the weights and biases of the MLP’s final layer to zero to ensure $v=uv=u$ initially. Together, these choices make the encoder initially behave like a centered linear dictionary encoder. As in the dense SAE setup, $b_{\mathrm{e}\mathrm{n}\mathrm{c}}b\_\{\backslash mathrm\{enc\}\}$ starts at zero and $b_{\mathrm{p}\mathrm{r}\mathrm{e}}b\_\{\backslash mathrm\{pre\}\}$ is initialized to the mean of the first training token batch. Encoder and decoder weights are optimized independently afterward, with the columns of $W_{\mathrm{d}\mathrm{e}\mathrm{c}}W\_\{\backslash mathrm\{dec\}\}$ kept at unit L2 norm.

SAE Training Setup

We train the SAE on batches of $B=4096B=4096$ contextualized token embeddings extracted from the same texts used for subsequent retrieval fine-tuning. We encode complete sequences with the frozen backbone, then shuffle their contextualized token embeddings before batching, including special and prompt tokens but excluding padding. We combine reconstruction MSE with an auxiliary loss to help prevent dead latents, features that stop activating and no longer receive gradients from the main reconstruction objective. We flag a latent as dead after more than 2,500 consecutive training steps without an activation.

For SlideOn, which we train from DenseOn, we use the original AuxK loss from Top-k SAE, which trains a subset of dead latents to reconstruct the remaining residual error, providing gradient updates to them without directly encouraging diverse latent usage across the dictionary. The AuxK loss is given by $${\mathcal{L}}_{\mathrm{A}\mathrm{u}\mathrm{x}\mathrm{K}}=\frac{1}{d}{\parallel (x-x^{\mathrm{\prime }})-W_{\mathrm{d}\mathrm{e}\mathrm{c}}{h}_{\mathrm{a}\mathrm{u}\mathrm{x}}\parallel }_2^2,\backslash mathcal\{L\}\_\{\backslash mathrm\{AuxK\}\}\; =\; \backslash frac\{1\}\{d\}\backslash left\backslash lVert\; (x-x\text{'})\; -\; W\_\{\backslash mathrm\{dec\}\}h\_\{\backslash mathrm\{aux\}\}\backslash right\backslash rVert\_2^2,$$ where $h_{\mathrm{a}\mathrm{u}\mathrm{x}}h\_\{\backslash mathrm\{aux\}\}$ retains only the $\mathrm{t}\mathrm{o}\mathrm{p}-k_{\mathrm{a}\mathrm{u}\mathrm{x}}\backslash mathrm\{top-\}k\_\{\backslash mathrm\{aux\}\}$ dead latents (we use $k_{\mathrm{a}\mathrm{u}\mathrm{x}}=1024k\_\{\backslash mathrm\{aux\}\}=1024$), with all other entries set to zero. We average this loss across the batch. The complete SAE objective for SlideOn is then

$${\mathcal{L}}_{\mathrm{S}\mathrm{A}\mathrm{E}}^{\mathrm{S}\mathrm{l}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{O}\mathrm{n}}=\frac{1}{Bd}\sum _{i=1}^B\parallel x_i-x_i^{\mathrm{\prime }}{\parallel }_2^2+{\lambda }_{\mathrm{A}\mathrm{u}\mathrm{x}\mathrm{K}}{\mathcal{L}}_{\mathrm{A}\mathrm{u}\mathrm{x}\mathrm{K}},\backslash mathcal\{L\}\_\{\backslash mathrm\{SAE\}\}^\{\backslash mathrm\{SlideOn\}\}\; =\; \backslash frac\{1\}\{Bd\}\backslash sum\_\{i=1\}^\{B\}\backslash lVert\; x\_i-x\text{'}\_i\backslash rVert\_2^2\; +\; \backslash lambda\_\{\backslash mathrm\{AuxK\}\}\backslash mathcal\{L\}\_\{\backslash mathrm\{AuxK\}\},$$

where ${\lambda }_{\mathrm{A}\mathrm{u}\mathrm{x}\mathrm{K}}\backslash lambda\_\{\backslash mathrm\{AuxK\}\}$ controls the strength of the AuxK loss, which we set to 1/16.

For mSlideOn, which we train from mLateOn, we use a load-balancing loss inspired by Switch Transformers to encourage broader latent usage. For each training batch, we count how often each latent appears among the Top-K selections, before ReLU. Dividing this count by the total number of selections gives its selection frequency $f_{j}f\_j$. Separately, we apply a softmax to all latent scores for each token and average the resulting probabilities across tokens to obtain ${\bar{p}}_j\backslash bar\{p\}\_j$. The balancing loss combines these statistics to discourage concentrating probability on already frequently selected latents. The counts are treated as fixed, while gradients flow through the softmax probabilities, allowing even unselected latents to receive a training signal, given by

$${\mathcal{L}}_{\mathrm{B}\mathrm{a}\mathrm{l}}=D\sum _{j=1}^D\mathrm{s}\mathrm{t}\mathrm{o}\mathrm{p}\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{d}(f_j){\bar{p}}_j,\backslash mathcal\{L\}\_\{\backslash mathrm\{Bal\}\}\; =\; D\backslash sum\_\{j=1\}^\{D\}\backslash mathrm\{stopgrad\}(f\_j)\backslash bar\{p\}\_j,$$

where $DD$ is the number of latents. The complete SAE objective for mSlideOn is then

$${\mathcal{L}}_{\mathrm{S}\mathrm{A}\mathrm{E}}^{\mathrm{m}\mathrm{S}\mathrm{l}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{O}\mathrm{n}}=\frac{1}{Bd}\sum _{i=1}^B\parallel x_i-x_i^{\mathrm{\prime }}{\parallel }_2^2+{\lambda }_{\mathrm{B}\mathrm{a}\mathrm{l}}{\mathcal{L}}_{\mathrm{B}\mathrm{a}\mathrm{l}},\backslash mathcal\{L\}\_\{\backslash mathrm\{SAE\}\}^\{\backslash mathrm\{mSlideOn\}\}\; =\; \backslash frac\{1\}\{Bd\}\backslash sum\_\{i=1\}^\{B\}\backslash lVert\; x\_i-x\text{'}\_i\backslash rVert\_2^2\; +\; \backslash lambda\_\{\backslash mathrm\{Bal\}\}\backslash mathcal\{L\}\_\{\backslash mathrm\{Bal\}\},$$ where ${\lambda }_{\mathrm{B}\mathrm{a}\mathrm{l}}\backslash lambda\_\{\backslash mathrm\{Bal\}\}$ controls the strength of the load balance loss, which we set to 1e-4.

We justify our choice of backbone and auxiliary loss in our dense versus LI ablations, which compare their effects on reconstruction, latent usage, downstream retrieval performance, and search efficiency.

Supervised Fine-Tuning for Sparse Retrieval

During SFT, we discard the SAE decoder and train only the backbone and the sparse encoder. Following SAE setup, each normalized token embedding is mapped to a sparse vector with 32,768 latents using top-32 latent pooling followed by ReLU. We then apply latent-wise max pooling across tokens to obtain a single sparse representation for each query or document. Unlike prior approaches that apply $\log (1+z)\backslash log(1+z)$ to latent activations, we retain their original magnitudes and compute retrieval scores with a raw dot product. We find that omitting this transformation improves retrieval performance when applying pruning to the final sparse vectors.

We combine the InfoNCE contrastive loss with KL-divergence distillation from cross-encoder scores, matching the mDenseOn/mLateOn setup. We apply Matryoshka training to both losses over pruned query–document pairs to support flexible sparsity at inference, as demonstrated in our SlideOn results section. Pruning retains only the top-n latents with the largest activation values in each final pooled vector. For SlideOn, we retain 32, 64, or 128 query latents and 512 or 1024 document latents. For mSlideOn, we retain 128 or 256 query latents and 512, 1024, or 2048 document latents. We train on all pair-wise combinations for each model. We train SlideOn with a batch size of 128 and mSlideOn with a batch size of 512, both using 7 mined hard negatives and in-batch negatives. We use a contrastive temperature of 0.02 and for KL distillation, we use a student temperature of 0.02 and a teacher temperature of 1.

We retain the load-balancing objective, averaging its query and document terms equally, to keep the number of dead latents low during retrieval fine-tuning. We further add FLOPS regularization terms for the pooled query and document vectors. We find that FLOPS regularization improves training stability with dot-product scoring, despite producing only small changes in overall sparsity, which is mainly controlled by top-k mechanism. We hypothesize that penalizing large average activations encourages the model to allocate weight more selectively to latents that contribute to retrieval. The full fine-tuning objective is thus $${\mathcal{L}}_{\mathrm{S}\mathrm{F}\mathrm{T}}={\mathcal{L}}_{\mathrm{I}\mathrm{n}\mathrm{f}\mathrm{o}\mathrm{N}\mathrm{C}\mathrm{E}}+{\mathcal{L}}_{\mathrm{K}\mathrm{L}}+{\lambda }_{\mathrm{B}\mathrm{a}\mathrm{l}}{\mathcal{L}}_{\mathrm{B}\mathrm{a}\mathrm{l}}+{\lambda }_q{\mathcal{L}}_{\mathrm{F}\mathrm{L}\mathrm{O}\mathrm{P}\mathrm{S},q}+{\lambda }_d{\mathcal{L}}_{\mathrm{F}\mathrm{L}\mathrm{O}\mathrm{P}\mathrm{S},d},\backslash mathcal\{L\}\_\{\backslash mathrm\{SFT\}\}\; =\; \backslash mathcal\{L\}\_\{\backslash mathrm\{InfoNCE\}\}\; +\; \backslash mathcal\{L\}\_\{\backslash mathrm\{KL\}\}\; +\; \backslash lambda\_\{\backslash mathrm\{Bal\}\}\; \backslash mathcal\{L\}\_\{\backslash mathrm\{Bal\}\}\; +\; \backslash lambda\_q\; \backslash mathcal\{L\}\_\{\backslash mathrm\{FLOPS\},q\}\; +\; \backslash lambda\_d\; \backslash mathcal\{L\}\_\{\backslash mathrm\{FLOPS\},d\},$$ where ${\mathcal{L}}_{\mathrm{I}\mathrm{n}\mathrm{f}\mathrm{o}\mathrm{N}\mathrm{C}\mathrm{E}}\backslash mathcal\{L\}\_\{\backslash mathrm\{InfoNCE\}\}$ and ${\mathcal{L}}_{\mathrm{K}\mathrm{L}}\backslash mathcal\{L\}\_\{\backslash mathrm\{KL\}\}$ aggregate the corresponding losses across the six pruning configurations used by each model.

We train on sequences of up to 8,192 tokens for both queries and documents to support long-context retrieval, using GradCache to keep GPU memory controlled when needed. We apply multinomial sampling with $\alpha =0.5\backslash alpha=0.5$ to increase the contribution of smaller datasets, following mDenseOn & mLateOn setup. The load-balancing weight ${\lambda }_{\mathrm{b}\mathrm{a}\mathrm{l}}\backslash lambda\_\{\backslash mathrm\{bal\}\}$ is set to 0.01, while both FLOPS weights are kept at 0.01 with a warmup ramp.

Results

English Retrieval: SlideOn

We evaluate all sparse retrievers using dot-product scores between their query and document representations. For each query, we run exact search over the full evaluation corpus. This setting avoids differences caused by approximate search, candidate selection, or retrieval-engine settings. To allow a systematic comparison of sparse, dense, and late-interaction retrieval, we also include published dense and late-interaction results from the DenseOn & LateOn release for English-only models. We report BEIR-15 performance as the average nDCG@10 across 15 BEIR datasets.

We report retrieval FLOPS for sparse models as a measure of sparse retrieval efficiency. The FLOPS metric estimates how many active latents a query and a document share on average and is given by $$\mathrm{R}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{e}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{F}\mathrm{L}\mathrm{O}\mathrm{P}\mathrm{S}=\sum _{j=1}^D{p}_j^q{p}_j^d,\backslash mathrm\{Retrieval\backslash \; FLOPS\}=\backslash sum\_\{j=1\}^\{D\}p\_j^q\; p\_j^d,$$ $$p_j^q=\frac{1}{N_q}\sum _{i=1}^{N_q}\mathbf{1}[q_{ij}>0],p\_j^q=\backslash frac\{1\}\{N\_q\}\backslash sum\_\{i=1\}^\{N\_q\}\backslash mathbf\{1\}[q\_\{ij\}>0],$$ $$p_j^d=\frac{1}{N_d}\sum _{i=1}^{N_d}\mathbf{1}[d_{ij}>0]\mathrm{.}p\_j^d=\backslash frac\{1\}\{N\_d\}\backslash sum\_\{i=1\}^\{N\_d\}\backslash mathbf\{1\}[d\_\{ij\}>0].$$ where $p_j^{q}p\_j^q$ and $p_j^{d}p\_j^d$ tell us how often latent $jj$ is active in queries and corpus documents, $N_{q}N\_q$ and $N_{d}N\_d$ are the numbers of evaluation queries and corpus documents and $DD$ is the total number of latents in the vector. This metric estimates the number of scoring operations per query, normalized by corpus size, providing a proxy for search cost. FLOPS also addresses a limitation of mean nonzero latent counts (nnz), since models can have similar nnz values for queries and documents, but the overlap between their active latents is more important.

SlideOn reaches 56.68 on BEIR-15 at 174M parameters, achieving the highest score among the sparse models below 3B parameters in our comparison. It also outperforms DenseOn and scores higher than all the dense baselines included here. SlideOn remains slightly behind LateOn, with a gap of just 0.5 average nDCG@10. This is encouraging given the cost of late-interaction retrieval. Compared with SPARSEUP which is trained over LateOn-unsupervised, SlideOn scores +2.2 points higher, however, SPARSEUP produces lower Retrieval FLOPS by default.

To improve both retrieval efficiency and index storage, we prune the final query and document sparse vectors by keeping only their $n$ largest activations. Query pruning directly increases search efficiency because fewer posting lists need to be consulted, while document pruning reduces the length of posting lists, improving search efficiency and reducing the memory required to store documents in the index. With documents pruned to the top-512 latents (d-512) and query budgets ranging from full vectors to the top-32 latents (q-32), SlideOn lies on the quality-efficiency frontier among the compared sub-2B models. Similarly, with unpruned queries and document budgets ranging from full vectors to the top-128 latents (d-128), SlideOn lies on the quality-storage cost frontier among compared models.

We trained SlideOn with a Matryoshka-style setup during SFT to concentrate retrieval information in the strongest query and document latents, limiting the performance drop under aggressive pruning. As shown below, this setup better preserves retrieval performance as query or document pruning increases. Although it comes with a small drop in unpruned performance, we chose to retain the Matryoshka setup for its better quality-efficiency tradeoff under pruning. Prioritizing the strongest latents also helps limit weak or unhelpful latents, reducing retrieval FLOPS even with unpruned query vectors (q-full).

Multilingual Retrieval: mSlideOn

We follow the same evaluation protocol as for the English models, running exact search over each full evaluation corpus and scoring pairs with dot product and unpruned query/document vectors. Alongside BEIR-15 for English retrieval, we evaluate multilingual retrieval on MIRACL, long-context retrieval on MLDR, and code retrieval on MTEB Code. Since our supervised retrieval training mixture covers a limited set of languages, we report both target-language (tgt) and full-benchmark scores for MIRACL and MLDR, following the mDenseOn and mLateOn. Target scores average performance over benchmark languages included in our training mixture, while full scores cover all benchmark languages, including those absent from retrieval fine-tuning. This lets us assess performance on training languages alongside broader multilingual generalization.

The most controlled comparison is with mDenseOn and mLateOn, since we use the same backbone, supervised training data, and cross-encoder distillation setup. We also include other sparse, dense, and late-interaction baselines to position mSlideOn against the broader state of the art.

On English retrieval measured on BEIR-15, mSlideOn reaches 56.00, the strongest result among the multilingual sparse models in our comparison. Its performance remains relatively close to mDenseOn at 56.70, while trailing mLateOn at 57.56.

On multilingual retrieval measured on MIRACL target languages (en, fr, es, de, ar), mSlideOn achieves 63.21, improving on mDenseOn by +3.60 points and remaining competitive with strong baselines such as pplx-embed-v1-0.6b at 63.18, while still trailing mLateOn at 65.61. On full MIRACL (18 languages), mSlideOn reaches 62.78, improving on mDenseOn by +4.76 points, while still trailing mLateOn at 67.04 and other baselines that were explicitly trained on many more languages. The improvements over mDenseOn show that sparse retrieval can outperform dense models across diverse multilingual retrieval tasks. Our training data covers only 4 training splits (Arabic, English, French, and Spanish) of the 18 MIRACL's languages. By comparison, BGE-M3 and MILCO use MIRACL training data spanning 16 languages. This broader supervision provides context for their stronger performance.

On long-document retrieval evaluated on MLDR, mSlideOn reaches 75.79 on target languages (en, fr, es, de, ar, it, pt) and 61.68 on the full benchmark (13 languages), strongly improving on mDenseOn by +10.17 and +10.18 points, respectively, while still trailing mLateOn at 87.68/77.92. On MLDR, mSlideOn outperforms every dense baseline in our comparison, including models explicitly trained with MLDR data such as Qwen3-Embedding, mGTE and BGE-M3, which is consistent with prior findings that sparse representations can be more effective than pooled dense embeddings for long-context retrieval. Among other baselines, it trails mGTE on both language sets and BGE-M3 sparse on the full languages, which is expected since these models are broadly trained on MDLR data, whereas our mixture restricts MLDR supervision to only 7 target languages.

Perhaps the most surprising result is on code retrieval evaluated on MTEB Code. mSlideOn reaches 74.78, outperforming every compared sparse and late-interaction baseline, including mLateOn at 73.48. It also remains competitive with the strongest dense baselines that do not disclose their training data. This demontrates the strong potential of sparse models for code retrieval tasks.

The detailed per-task and per-language performance gaps can be seen below, showing that mSlideOn outperforms mDenseOn on most evaluated tasks and languages, while still trailing mDenseOn on particular BEIR-15 tasks.

Contributions of SAE Pretraining and Supervised Retrieval Fine-Tuning

We found that supervised retrieval fine-tuning can substantially improve performance over SAE pretraining alone. To assess the contribution of each stage, we compare four configurations. SAE only trains the SAE on frozen backbone representations, while SFT only trains the sparse retriever without SAE pretraining using our token-sampling initialization. SAE → SFT uses SAE pretraining to initialize the sparse encoder before retrieval fine-tuning. Finally, SAE + SFT jointly optimizes the SAE reconstruction and supervised retrieval objectives within a single training stage. We additionally evaluate SPARSEUP SFT under the same training data, distillation, backbone model, and top-32 pooling to compare its MLM-based setup with our methods. As shown below, SAE → SFT achieves the highest BEIR-15 score, motivating our choice of sequential SAE pretraining followed by SFT. SPARSEUP SFT yields the lowest retrieval FLOPS among the evaluated configurations, with slightly reduced retrieval performance. This trade-off suggests that its MLM-based initialization is promising for ultra-sparse retrieval setups.

Dense and Late-Interaction Initialization for Sparse Retrieval

We compare sparse retrievers initialized from dense and late-interaction models to understand how the starting representations affect sparse retrieval and to choose the best setups for SlideOn and mSlideOn. We study both DenseOn and LateOn, alongside their multilingual counterparts, mDenseOn and mLateOn. A strictly controlled comparison is difficult because their token embedding spaces differ substantially in dimensionality and geometry. With a sparse latent space of $D=32,768D\; =\; 32,768$, a direct linear encoder projection requires 768 x 32,768 weights (25.2M) for our dense models, compared with 128 x 32,768 (4.2M) for our late-interaction models. Randomly sampled dense token embeddings also have substantially lower cosine similarity than late-interaction embeddings (0.175 for DenseOn vs. 0.959 for LateOn on FiQA). In our experiments, the spatial anisotropy makes it harder for the SAE to learn accurate reconstructions. As explained in the SAE section, we address these differences through centered dictionary initialization and a more complex encoder to increase capacity for late-interaction models.

A recurring observation is that our sparse models initialized from late interaction produce higher dead-latent rates during both SAE pretraining and SFT. We observe that without sufficient control, latent usage can collapse onto a small subset of the encoder dictionary, reducing effective representational capacity and increasing retrieval FLOPS. In our experiments, starting from LateOn SAE and adding the load-balancing loss during SFT reduces the proportion of dead latents from 54.85% to 0.128%. Starting from DenseOn SAE, it reduces retrieval FLOPS from 57.02 to 52.89 while slightly improving BEIR-15 nDCG@10 from 56.73 to 57.03, highlighting the benefits of directly incentivizing broader latent usage during SFT. These findings led us to explore load balancing during SAE pretraining as a potential replacement for the AuxK loss.

As shown above, load balancing nearly eliminates dead latents during SAE pretraining, while maintaining comparable reconstruction quality (low FVU). Nevertheless, AuxK pretraining yields slightly better English retrieval performance and lower FLOPS for both initializations. These results suggest that encouraging broader latent usage during SAE pretraining without jointly optimizing a retrieval objective may, in some cases, slightly reduce downstream retrieval quality and increase retrieval FLOPS. For English retrieval, DenseOn initilization outperforms LateOn on BEIR-15 under both auxiliary losses, with a small quality advantage and lower retrieval FLOPS. These results motivate our choice of DenseOn AuxK SAE as the initialization for SlideOn.

The multilingual models show a different pattern from the English-only experiments, with load balancing offering a more favorable trade-off between retrieval quality and FLOPS. As shown below, load-balancing eliminates dead latents during SAE pretraining for both backbones. For mDenseOn, it preserves reconstruction quality and slightly improves retrieval across all four benchmarks, with a modest increase in FLOPS from 59.01 to 63.75. For mLateOn, it strongly reduces FLOPS from 76.47 to 53.05 while maintaining broadly comparable retrieval performance. These findings support an advantage of load balancing during SAE pretraining for multilingual retrieval.

Comparing the two initializations, mDenseOn achieves sligthly better English (BEIR-15) and code retrieval (MTEB Code) scores, while mLateOn performs substantially better on multilingual (MIRACL) and long-document retrieval (MLDR). These advantages mirror mLateOn’s strengths over mDenseOn, including stronger multilingual generalization, especially to languages absent from retrieval fine-tuning, and better long-document retrieval. This suggests that sparse retrievers initialized from late-interaction checkpoints inherit these properties, even if they do not fully match the retrieval performance of their late-interaction initialization in our setup. These results motivate our choice of mLateOn load-balance SAE as the initialization for mSlideOn.

Conclusion

In this work, we show that existing dense and late-interaction checkpoints can be converted into effective sparse retrievers through SAE pretraining and supervised retrieval fine-tuning, both substantially cheaper than large-scale retrieval pretraining.

SlideOn improves over DenseOn on BEIR-15 and uses sparse vector pruning to achieve leading performance on both the quality–search cost and quality–storage cost Pareto frontiers among the compared sub-2B models. mSlideOn improves over mDenseOn in multilingual, long-document, and code retrieval, including languages absent from retrieval fine-tuning. These results highlight the potential of sparse retrievers to replace dense models while maintaining strong retrieval quality and enabling inverted-index search.

Our ablations show that the choice of backbone and SAE setup matters. DenseOn with AuxK provides the best English initialization in our experiments, while mLateOn with load balancing offers a stronger foundation for posterior multilingual fine-tuning. Matryoshka training complements these choices by preserving retrieval quality under pruning, allowing query and document sparsity to be adjusted at inference without retraining.

Future work can expand supervision to more languages and scripts, further investigate how to better preserve late-interaction matching capabilities during sparsification, and improve the SAE setup. We release our SAE checkpoints, fine-tuned retrieval models, and training scripts to enable further research.

Acknowledgements

We thank Amélie Chatelain for her thoughtful feedback and valuable contributions to this work. We appreciate the time and care she dedicated to helping us refine our explanations and improve the clarity and presentation of our results.

Citation

If you find this work useful for your research, please consider citing us:

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}