README.md

metadata

license: apache-2.0
library_name: sentence-transformers
pipeline_tag: feature-extraction
base_model: lightonai/DenseOn
base_model_relation: finetune
language:
  - en
datasets:
  - lightonai/embeddings-fine-tuning-filtered-en
tags:
  - sparse
  - sentence-transformers
  - sentence-similarity
  - custom-code
  - sparse-retrieval
  - matryoshka

📚 Collection | 📝 Blogpost | 🛠️ Training and Evaluation

SlideOn

English Sparse Retrieval Model by LightOn

SlideOn | mSlideOn | SlideOn-SAE | mSlideOn-SAE

🎯 TL;DR: A 174M-parameter English sparse retrieval model built from DenseOn through sparse autoencoder (SAE) pretraining and supervised retrieval fine-tuning. SlideOn reaches 56.68 nDCG@10 on BEIR-15, surpassing DenseOn's 56.20, while producing sparse representations compatible with inverted-index search. Joint query/document Matryoshka training preserves retrieval quality under pruning, allowing search and storage costs to be adjusted at inference without retraining.

About the SlideOn / mSlideOn Family

With DenseOn and LateOn, then mDenseOn and mLateOn, we explored dense and late-interaction retrieval using our open data recipe. SlideOn and mSlideOn reuse these models' retrieval knowledge to learn sparse representations through SAE pretraining followed by supervised fine-tuning.

SlideOn starts from DenseOn for English retrieval. mSlideOn starts from mLateOn for multilingual, long-context, and code retrieval. We release both the pretrained SAE initializations and the final retrieval models.

See our blog post for the full method, results, and ablations.

SlideOn

SlideOn encodes queries and documents into 32,768-dimensional nonnegative sparse vectors. A learned linear encoder maps each normalized DenseOn token representation to its top-32 latent activations, followed by ReLU. Max pooling across tokens produces one vector per query or document, and retrieval uses the dot product between these vectors.

The dimensions are learned features, independent of the tokenizer vocabulary. Active latents can be stored in an inverted index, where query latents select posting lists and their weights contribute to document scores.

SlideOn notably:

• Reaches 56.68 on BEIR-15 with unpruned vectors, improving on DenseOn by 0.48 points.
• Reaches 56.98 with unpruned queries and documents capped at 512 nonzero latents.
• Retains 56.66 with queries capped at 64 and documents at 512, reducing the measured retrieval FLOPS proxy to 8.72.
• Supports inputs of up to 8,192 tokens and independently adjustable query and document sparsity.

Results

Headline Results (BEIR-15, nDCG@10)

BEIR-15 scores are average nDCG@10 across 15 datasets, reported on a 0–100 scale. Sparse models use dot-product scoring and exact search over each full evaluation corpus. All rows use unpruned query and document vectors. Dense and late-interaction results are taken from our DenseOn and LateOn blog post.

Model	Parameters	BEIR-15 ↑	Retrieval FLOPS ↓
Sparse
SlideOn	174M	56.68	30.87
LACONIC-8B	8.03B	60.06	20.07
LACONIC-3B	3.21B	58.30	39.53
LACONIC-1B	1.24B	56.13	47.95
SPARSEUP	188M	54.46	6.20
Lion-SP-8B	8.0B	53.62	57.03
OpenSearch doc-v3-GTE	137M	52.68	0.72
OpenSearch v2-distill	67M	51.20	5.14
SPLADE-v3	133M	50.46	5.55
SPLADE-v3-DistilBERT	90M	49.04	5.79
Granite-30M	30M	48.86	6.42
SAE-SPLADE	117M	48.21	2.67
Dense
DenseOn	149M	56.20	—
gte-modernbert-base	149M	55.19	—
bge-large-en-v1.5	335M	54.34	—
modernbert-embed-base	149M	52.89	—
Late interaction
LateOn	149M	57.22	—
ColBERT-Zero	149M	55.39	—
GTE-ModernColBERT-v1	149M	54.75	—
ColBERT-small	33M	53.79	—
ColBERTv2	110M	48.63	—

Average nDCG@10 on BEIR-15 (↑ higher is better) and retrieval FLOPS (↓ lower is better). Blue marks our models. Bold marks best per column within each family.

SlideOn achieves the highest BEIR-15 score among the sparse models below 3B parameters in this comparison. It improves on SPARSEUP by 2.22 points and on DenseOn by 0.48 points, while remaining 0.54 points behind LateOn. The figures below show how pruning changes the comparison when search and storage costs are taken into account.

Retrieval Quality versus Search and Storage Cost

Query pruning reduces the number of posting lists consulted during search. Document pruning shortens those lists and reduces the number of stored document weights. Matryoshka training concentrates retrieval information in the strongest latents so that both budgets can be adjusted after training.

Retrieval quality versus search cost for SlideOn and sub-2B baselines. Blue points show SlideOn with documents pruned to the top-512 latents and varying query budgets, where q-full denotes unpruned queries. Higher BEIR-15 ↑ scores and lower FLOPS ↓ are better. Retrieval FLOPS are measured on a subset of SciFact, NFCorpus, and FiQA.

Retrieval quality versus index storage cost for SlideOn and sub-2B baselines. Blue points show SlideOn with unpruned queries and varying document budgets, where d-full denotes unpruned documents. Higher BEIR-15 ↑ scores and lower NNZ ↓ are better. Document NNZ is measured on SciFact, NFCorpus, and FiQA.

Retrieval FLOPS estimates the expected number of shared active features per query–document pair. It is a search-cost proxy, not measured latency or encoder FLOPs. Document NNZ is the average number of nonzero document features and serves as a storage-cost proxy, rather than a measurement of index size in bytes. Efficiency measurements use the fixed sample of 6,000 documents and 1,271 queries from SciFact, NFCorpus, and FiQA.

Related Checkpoints

Model	Description
SlideOn (this model)	English sparse retrieval model, after SAE pretraining and retrieval fine-tuning
SlideOn-SAE	DenseOn-based SAE initialization before retrieval fine-tuning
mSlideOn	Multilingual sparse retrieval model for natural language, long documents, and code
mSlideOn-SAE	mLateOn-based SAE initialization before retrieval fine-tuning
DenseOn	English dense retriever used to initialize SlideOn's backbone

The SAE checkpoints are starting points for training. Use SlideOn or mSlideOn for the retrieval results reported in the blog.

Training Data and Recipe

See our blog post for training data, SAE pretraining, and supervised retrieval fine-tuning.

Model Details

Model Description

Property	Value
Model type	Learned sparse retriever; Sentence Transformers SparseEncoder
Backbone	DenseOn / ModernBERT-base
Total parameters	174M: 149M backbone + 25M sparse head
Maximum sequence length	8,192 tokens, including prompts and special tokens
Output dimensionality	32,768 learned latent features
Token representation	768 dimensions, L2-normalized before sparse encoding
Token sparsity	Top-32 latent selection followed by ReLU
Sequence pooling	Maximum over token activations
Similarity function	Dot product
Prompts	query: for queries; document: for documents
Language	English

Usage

Direct Usage (Sentence Transformers)

Install the supported library versions:

pip install -U "sentence-transformers>=6.1,<6.2" "transformers>=5.3,<6"

Load the model and encode queries and documents. This example uses the q-64 / d-512 configuration:

from sentence_transformers import SparseEncoder

model = SparseEncoder("lightonai/SlideOn", trust_remote_code=True)

queries = ["What causes a solar eclipse?"]
documents = [
    "A solar eclipse occurs when the Moon passes between the Sun and Earth.",
    "Wind turbines convert the kinetic energy of wind into electricity.",
]

query_embeddings = model.encode_query(queries, max_active_dims=64)
document_embeddings = model.encode_document(documents, max_active_dims=512)

print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([1, 32768]) torch.Size([2, 32768])

scores = model.similarity(query_embeddings, document_embeddings)
print(scores)

encode_query() and encode_document() apply the appropriate prompts and preprocessing automatically. Pass the raw text without manually adding prefixes. The returned embeddings are PyTorch sparse COO tensors. The model and custom inference code are included in this repository; loading does not require the training package or PyLate.

Choosing Query and Document Budgets

Set max_active_dims=N to keep the top-N latents with the highest weights in each pooled vector and set the rest to zero. You can choose a different N for queries and documents. This is separate from the per-token top-32 operation and does not change the vector dimensionality.

Use a positive integer for a cap, or None for the saved unpruned defaults; 0 is invalid.

To reproduce the unpruned setting:

query_embeddings = model.encode_query(queries, max_active_dims=None)
document_embeddings = model.encode_document(documents, max_active_dims=None)

Omitting max_active_dims also uses the saved unpruned defaults. To use the q-full / d-512 setting, keep the query call at max_active_dims=None and set the document call to max_active_dims=512.

Dimensions represent learned latents, so they do not map directly to tokenizer vocabulary entries. Inspect the sparse indices and weights; SparseEncoder.decode() cannot label them as vocabulary tokens. Retrieval uses raw dot products, so leave final vector normalization disabled.

Citation

If you find this work useful for your research, please consider citing us:

@misc{moura2026slideon,
  author       = {Moura Junior, Paulo Roberto and Sourty, Raphaël},
  title        = {{SlideOn} \& {mSlideOn}: Sparsified late-interaction and dense retrievers},
  year         = {2026},
  howpublished = {Hugging Face Blog},
  url          = {https://huggingface.co/blog/lightonai/slideon-mslideon}
}