algoprog/mDenseOn-ONNX

mDenseOn in ONNX

lightonai/mDenseOn, LightOn's multilingual dense retriever (ModernBERT, 307M parameters, 768 dimensions), exported to ONNX: exactly in float32, and in an 8-bit build a quarter of the size that ranks the same. It runs on a CPU with ONNX Runtime and Hugging Face's tokenizer alone, without PyTorch or Transformers.

Made for Hydor, where the 8-bit build runs memory's search by meaning on the user's own computer.

File	Size	What it is	Cosine to PyTorch float32
onnx/model.onnx	1.2 GB	float32, the model as it is	1.000000
onnx/model_q8.onnx	308 MB	weights in 8 bits, arithmetic in float32	≥ 0.9995 (mean 0.9996)

Both take input_ids and attention_mask (int64, any batch and length) and give last_hidden_state; the embedding is its first token's vector (CLS pooling), normalized. Texts start with mDenseOn's prompts, query: or document: .

Retrieval

nDCG@10 on two BEIR sets (the whole corpus, test queries), CLS pooling, cosine:

	SciFact	NFCorpus
float32 (the same as PyTorch)	74.35	37.40
8-bit	74.25 (−0.10, 95% interval −0.40 to +0.09)	37.59 (+0.19, +0.01 to +0.40)
Top-10 overlap with float32	0.982	0.976

Documents embedded by the original model in PyTorch (its word table in fp16) and searched with this build's queries score 74.55 and 37.61, so vectors made before can stay. The intervals are bootstrapped over queries.

Speed and memory

From a fresh process on a CPU, 4 threads (ONNX Runtime 1.31, the model files in the page cache):

	float32	8-bit
Imports, tokenizer and model loaded, first vector	1.6–2.0 s	0.95–1.06 s
Memory once loaded	2.0 GB	1.04 GB
One query	13 ms	31–38 ms
One 512-token text	180 ms	200–230 ms

Use

# pip install onnxruntime tokenizers numpy huggingface_hub
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

repo = "algoprog/mDenseOn-ONNX"
tok = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))
tok.enable_truncation(512)
tok.enable_padding(pad_id=0, pad_token="<pad>")
model = ort.InferenceSession(hf_hub_download(repo, "onnx/model_q8.onnx"), providers=["CPUExecutionProvider"])


def embed(texts, kind):  # kind: "query" or "document"
    enc = tok.encode_batch([f"{kind}: {t}" for t in texts])
    ids = np.array([e.ids for e in enc], dtype=np.int64)
    mask = np.array([e.attention_mask for e in enc], dtype=np.int64)
    h = model.run(None, {"input_ids": ids, "attention_mask": mask})[0][:, 0]
    return h / np.linalg.norm(h, axis=1, keepdims=True)


q = embed(["Which planet is red?"], "query")
d = embed(["Mars is often called the Red Planet.", "Jupiter is the largest planet."], "document")
print(q @ d.T)

The model reads up to 8,192 tokens; truncate where your texts need. model_q8.onnx needs ONNX Runtime 1.23 or newer (8-bit MatMulNBits on the CPU).

How it was made

• Export (scripts/export.py): PyTorch 2.15 and Transformers 5.22, torch.onnx.export(dynamo=True), opset 20, dynamic batch and length, eager attention (the sliding-window masks are in the graph). Checked against PyTorch at 5 to 512 tokens, alone and in batches of 1 to 8: cosine 1.000000 throughout; the tokenizer here gives the same ids as Transformers'.
• 8 bits (scripts/quantize.py): the 88 weight matrices through ONNX Runtime's MatMulNBitsQuantizer (8 bits, blocks of 32, symmetric); the 256k × 768 word table, which is 64% of the model, as int8 rows with one float32 scale each (largest magnitude / 127; each row's cosine to the original ≥ 0.9993, mean 0.99997), looked up and scaled back with Gather → Cast → Mul, standard operators any runtime has. Activations stay float32.

License and credit

Apache 2.0, as the original. The model is LightOn's; converted by Chris Samarinas.

@misc{sourty2026denseonlateonfullyopen,
  title         = {DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search},
  author        = {Raphaël Sourty and Antoine Chaffin and Paulo Roberto Moura Junior and Amélie Chatelain},
  year          = {2026},
  eprint        = {2607.27178},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2607.27178},
}