---
license: cc-by-4.0
tags:
- onnx
- quantized
- retrieval
- reranker
- embeddings
datasets:
- nynefold/FreshRet-Small
- mteb/MIRACLReranking
---
# retrieval-quant: quantized ONNX retrievers

Four open retrieval models, two embedders and two rerankers, quantized to 4 and 8 bit for ONNX
Runtime and measured against their own FP32 versions. Every variant we built is published, failed
ones included, so the size/quality trade-off can be read from the tables below. Tested with ONNX
Runtime 1.31.

Repos: [`nynefold/pplx-embed-v1-0.6b-ONNX`](https://huggingface.co/nynefold/pplx-embed-v1-0.6b-ONNX),
[`nynefold/F2LLM-v2-1.7B-ONNX`](https://huggingface.co/nynefold/F2LLM-v2-1.7B-ONNX),
[`nynefold/Qwen3-Reranker-0.6B-ONNX`](https://huggingface.co/nynefold/Qwen3-Reranker-0.6B-ONNX),
[`nynefold/zerank-1-small-ONNX`](https://huggingface.co/nynefold/zerank-1-small-ONNX), and the evaluation set
[`nynefold/FreshRet-Small`](https://huggingface.co/datasets/nynefold/FreshRet-Small). `pipeline_example.py` in this repo
retrieves with pplx-embed and reranks with zerank, using `inference.py` (ONNX Runtime, `tokenizers`
and NumPy only). All six repos are in the collection
[retrieval-quant: quantized ONNX retrievers](https://huggingface.co/collections/nynefold/retrieval-quant-quantized-onnx-retrievers-6adb9c3a7f6012982d819511).

**Also here:** [pplx-embed-v2-late 0.6B and 9B](pplx-embed-v2-late.md), late-interaction models quantized the same way,
and how to index with the 9B and query with the 0.6B.

## Results in short

- Each model has a pick that stays within 1 nDCG@10 point of its FP32 version at **3.4 to 6x smaller**
  files and **1.8 to 3.3x less RAM**.
- **pplx-embed + zerank-1-small** is the recommended pair: 0.870 nDCG@10 on FreshRet Small in 1.9 GB.
  The reranker adds about 12 points over the embedder alone.
- The cheapest useful setup is **pplx-embed alone**: 0.746 in 401 MB.
- Qwen3-Reranker-0.6B adds nothing on FreshRet Small over the embedder alone, at FP32 as well as quantized.
- The two Qwen3-1.7B-based models (F2LLM, zerank) have MLP activation outliers. F2LLM needs
  SmoothQuant for INT8 compute; zerank only gains a little from it at 4 bit. The 0.6B models don't
  need it.

## Picks

One recommended variant per model. Size is the ONNX file; RAM is peak process memory while embedding
one 256-token passage (embedders) or scoring 20 candidates of 170 tokens (rerankers), in a process
with only ONNX Runtime, `tokenizers` and NumPy loaded. Quality deltas are paired against FP32 in nDCG@10 points, with 95%
bootstrap intervals (10,000 resamples). Picks were chosen on FreshRet Small; MIRACL ([mteb/MIRACLReranking](https://huggingface.co/datasets/mteb/MIRACLReranking)) is
the held-out check.

| model | repo | pick | size MB (FP32) | RAM MB (FP32) | FreshRet Small nDCG@10 vs FP32 | MIRACL nDCG@10 vs FP32 |
|---|---|---|---:|---:|---|---|
| pplx-embed-v1-0.6b | [`nynefold/pplx-embed-v1-0.6b-ONNX`](https://huggingface.co/nynefold/pplx-embed-v1-0.6b-ONNX) | `mix02-gptq-a4-pc` | 401 (2387) | 571 (1892) | -0.08 [-0.91, +0.74] | -0.38 [-1.38, +0.55] |
| F2LLM-v2-1.7B | [`nynefold/F2LLM-v2-1.7B-ONNX`](https://huggingface.co/nynefold/F2LLM-v2-1.7B-ONNX) | `mix50-gptq-sq50-a4` | 1629 (6885) | 1788 (5603) | +0.19 [-0.44, +0.85] | -0.38 [-1.22, +0.28] |
| Qwen3-Reranker-0.6B | [`nynefold/Qwen3-Reranker-0.6B-ONNX`](https://huggingface.co/nynefold/Qwen3-Reranker-0.6B-ONNX) | `e8-w8-rtn-a4` | 692 (2386) | 1473 (2589) | -0.02 [-0.48, +0.43] | -0.07 [-1.08, +0.89] |
| zerank-1-small | [`nynefold/zerank-1-small-ONNX`](https://huggingface.co/nynefold/zerank-1-small-ONNX) | `mix30-gptq-sq50-a4` | 1483 (6885) | 2505 (6445) | -0.30 [-0.93, +0.33] | -0.36 [-1.42, +0.54] |

![size and RAM, FP32 against the recommended variant](fig/savings.png)

![FreshRet Small nDCG@10 vs FP32 for every measured variant](fig/quality-vs-fp32.png)

## Pipelines

The embedder retrieves the top 20 from FreshRet Small, the reranker reorders them, and the top 10
are scored. All models are the quantized picks.

![nDCG@10 against total size](fig/pipeline-frontier.png)

| embedder | reranker | total MB | nDCG@10 | R@10 | MRR@10 | vs best (pt, 95% CI) |
|---|---|---:|---:|---:|---:|---|
| pplx-embed-v1-0.6b `mix02-gptq-a4-pc` | none | 401 | 0.746 | 0.898 | 0.775 | -12.57 [-14.96, -10.17] |
| pplx-embed-v1-0.6b `mix02-gptq-a4-pc` | Qwen3-Reranker-0.6B `e8-w8-rtn-a4` | 1093 | 0.735 | 0.908 | 0.747 | -13.68 [-15.95, -11.39] |
| F2LLM-v2-1.7B `mix50-gptq-sq50-a4` | none | 1629 | 0.742 | 0.891 | 0.762 | -12.98 [-15.47, -10.50] |
| pplx-embed-v1-0.6b `mix02-gptq-a4-pc` | zerank-1-small `mix30-gptq-sq50-a4` | 1885 | 0.870 | 0.935 | 0.921 | -0.16 [-1.02, +0.65] |
| F2LLM-v2-1.7B `mix50-gptq-sq50-a4` | Qwen3-Reranker-0.6B `e8-w8-rtn-a4` | 2321 | 0.737 | 0.907 | 0.750 | -13.45 [-15.58, -11.31] |
| F2LLM-v2-1.7B `mix50-gptq-sq50-a4` | zerank-1-small `mix30-gptq-sq50-a4` | 3113 | 0.872 | 0.937 | 0.922 | best |

pplx-embed + zerank: 0.870 in 1885 MB, +12.40 [+10.13, +14.69] points over pplx-embed alone (0.746,
401 MB). F2LLM + zerank scores the same within noise at 1.2 GB more. Nothing sits usefully
between the two sizes. The embedders already put about 95% of the relevant documents in their top
20, so the reranker's ordering is what moves nDCG.

**Rerank depth** (zerank pick, pplx-embed pool, vs reranking 20): 15 candidates cost 0.38 points
[-0.80, +0.01], 10 cost 1.45 [-2.30, -0.67], 5 cost 3.33. Reranking cost grows with the number of
candidates, so 15 is a reasonable saving and 10 still keeps about 11 points over the embedder alone.

On the MIRACL slice (120 queries, BM25 top-20 pools, sparse labels) neither FP32 reranker beats
pplx-embed alone; MIRACL has few judged passages per query, so FreshRet Small, fully judged, is the
set that separates the rerankers.


## Method

**Quantizer.** Our own writer for ONNX Runtime's `MatMulNBits` (linear layers) and
`GatherBlockQuantized` (token table) contrib ops, so bits, block size, algorithm and compute mode can
be set per tensor. Blocks of 32 input channels, asymmetric unless noted. Algorithms: round-to-nearest
(RTN), clip search (MSE, optionally Hessian-weighted), HQQ, and GPTQ with act-order (Hessians from a
calibration split of MIRACL dev queries).

**Mixed precision.** A sensitivity scan in PyTorch (fake-quant on cached layer inputs) estimates the
damage of putting each tensor at 4 bit. `mixNN` starts from all-4-bit and promotes the most damaging
tensors to 8 bit until NN% of the size gap to all-8-bit is spent.

**Gates.** Stage A, fidelity to FP32 on public MIRACL sets: embedders need cosine mean >= 0.99 and
p1 >= 0.97; rerankers need per-query Spearman mean >= 0.98. Stage B (embedders) and C (rerankers),
retrieval quality on FreshRet Small (300 queries, 6 languages, a third cross-lingual): nDCG@10 may
drop at most 1.0 point vs FP32. "Strict" means the whole 95% interval clears -1.0. Embedder picks
are the smallest variant passing both gates strictly. For rerankers, stage C was run on candidates
near Spearman 0.99, since 0.987 already lost 1.2 points; their picks are the smallest variant that
passed stage C strictly, and smaller variants that only just passed stage A were not measured.
FreshRet Small is never used for calibration or recipe choice.

**Sets.** Fidelity: 60 MIRACL queries + 360 passages (embedders), 360 query-passage pairs
(rerankers). Quality: FreshRet Small in full; reranker pools are its top 20 by reciprocal-rank fusion
of BM25 and both FP32 embedders. Held-out check: 120 MIRACL dev queries (20 per language: en, es,
fr, de, zh, ar) with BM25 top-20 pools. All MIRACL data is the dev split of [mteb/MIRACLReranking](https://huggingface.co/datasets/mteb/MIRACLReranking),
MTEB's reranking version of [MIRACL](https://huggingface.co/datasets/miracl/miracl); calibration,
fidelity and held-out queries are disjoint.


![fidelity against size](fig/fidelity-vs-size.png)

## What we learned

- **GPTQ is what makes 4 bit work for the 0.6B embedder.** pplx-embed at uniform 4 bit: RTN, MSE and
  HQQ all land near cosine 0.90; GPTQ reaches 0.992 at the same size. The base model's own 4-bit ONNX
  sits at 0.90 too.
- **Fidelity damage is uneven.** F2LLM's damage piles up in its last layers (layer 27's up-projection
  alone drops cosine to 0.74); a few percent of the tensors at 8 bit fixes most of it. Qwen3-Reranker's
  is spread thin, so mixing barely helps and it stays at 8 bit.
- **Per-block INT8 compute breaks F2LLM without SmoothQuant.** F2LLM at 8-bit
  weights falls from cosine 0.995 to 0.977 once the activations are INT8 too; the MLP inputs have 2-3x
  the outlier ratio of pplx-embed's. SmoothQuant (alpha 0.5, folded exactly into the weights) brings
  it back to 0.992. zerank, the other Qwen3-1.7B model, has the same outliers but doesn't need it:
  at 8-bit weights INT8 compute keeps Spearman 0.9963 with or without SmoothQuant, and at 4 bit it
  raises 0.9808 to 0.9865. On pplx-embed and Qwen3-Reranker it changes nothing.
- **Stage A has to be tight for rerankers.** Spearman ~0.987 still lost 1.2 nDCG points on FreshRet
  Small (zerank uniform 4 bit); ~0.990 was safe.
- **Pruning pplx-embed's vocabulary** to the tokens seen in the MIRACL corpus (six languages) saves
  69 MB. It keeps every MIRACL token, so MIRACL results are unchanged by construction; on FreshRet
  Small, which was not used to build it, it costs -0.29 points [-0.78, +0.22] against the unpruned
  variant, -1.00 [-1.95, -0.27] on Arabic documents.
- The `a4` variants (INT8 compute) are meant for CPUs with fast INT8 instructions; `a0` variants
  compute in FP32 from the quantized weights.

## All variants

Stage A fidelity for every variant; FreshRet Small quality where it was measured (finalists only).
"Below quality gate" means it failed stage A, or its nDCG@10 drop is more than 1.0 point.
"pass (estimate only)" means the estimate is within 1.0 point but the interval is not; "pass (stage A
only)" means it was not measured on FreshRet Small.


### pplx-embed-v1-0.6b

Base model [`perplexity-ai/pplx-embed-v1-0.6b`](https://huggingface.co/perplexity-ai/pplx-embed-v1-0.6b), FP32 2387 MB. Files: [`nynefold/pplx-embed-v1-0.6b-ONNX`](https://huggingface.co/nynefold/pplx-embed-v1-0.6b-ONNX).

| variant | MB | cos mean (gate 0.99) | FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt) | verdict |
|---|---:|---:|---|---|
| `e4-w4-rtn-a4` | 385 | 0.9014 |  | below quality gate |
| **`mix02-gptq-a4-pc`** | 401 | 0.9932 | -0.08 [-0.91, +0.74] | **pick**, pass |
| `e8-w4-rtn-b128-a4` | 419 | 0.8669 |  | below quality gate |
| `e8-w4-rtn-sym-a4` | 458 | 0.8714 |  | below quality gate |
| `e8-w4-gptq-a0` | 465 | 0.9922 |  | pass (stage A only) |
| `e8-w4-gptq-a4` | 465 | 0.9921 | +0.19 [-0.53, +0.93] | pass |
| `e8-w4-gptq-sq50-a4` | 465 | 0.9921 |  | pass (stage A only) |
| `e8-w4-mse-a4` | 465 | 0.9020 |  | below quality gate |
| `e8-w4-rtn-a4` | 465 | 0.9019 |  | below quality gate |
| `mix02-gptq-a4` | 470 | 0.9932 | +0.20 [-0.49, +0.90] | pass |
| `mix02-gptq-sq50-a4` | 470 | 0.9931 |  | pass (stage A only) |
| `mix02-mse-a4` | 470 | 0.9837 |  | below quality gate |
| `mix02-msew-a4` | 470 | 0.9860 |  | below quality gate |
| `mix02-rtn-a4` | 470 | 0.9828 |  | below quality gate |
| `mix10-gptq-a4` | 488 | 0.9958 | +0.06 [-0.54, +0.63] | pass |
| `mix10-mse-a4` | 488 | 0.9904 |  | pass (stage A only) |
| `mix10-msew-a4` | 488 | 0.9917 |  | pass (stage A only) |
| `mix10-rtn-a4` | 488 | 0.9898 |  | below quality gate |
| `e8-w4-hqq-a0` | 513 | 0.9055 |  | below quality gate |
| `mix30-gptq-a4` | 533 | 0.9977 |  | pass (stage A only) |
| `mix30-mse-a4` | 533 | 0.9952 |  | pass (stage A only) |
| `mix30-msew-a4` | 533 | 0.9958 |  | pass (stage A only) |
| `mix30-rtn-a4` | 533 | 0.9949 |  | pass (stage A only) |
| `e8-w8-rtn-a0` | 692 | 0.9994 |  | pass (stage A only) |
| `e8-w8-rtn-a4` | 692 | 0.9993 | +0.15 [-0.05, +0.40] | pass |
| `e32-w8-rtn-a4` | 1135 | 0.9993 |  | pass (stage A only) |

### F2LLM-v2-1.7B

Base model [`codefuse-ai/F2LLM-v2-1.7B`](https://huggingface.co/codefuse-ai/F2LLM-v2-1.7B), FP32 6885 MB. Files: [`nynefold/F2LLM-v2-1.7B-ONNX`](https://huggingface.co/nynefold/F2LLM-v2-1.7B-ONNX).

| variant | MB | cos mean (gate 0.99) | FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt) | verdict |
|---|---:|---:|---|---|
| `e8-w4-gptq-a0` | 1266 | 0.9540 |  | below quality gate |
| `e8-w4-gptq-a4` | 1266 | 0.9370 |  | below quality gate |
| `e8-w4-gptq-gu0` | 1266 | 0.9508 |  | below quality gate |
| `e8-w4-gptq-sq50-a4` | 1266 | 0.9490 |  | below quality gate |
| `mix02-gptq-a4` | 1280 | 0.9471 |  | below quality gate |
| `mix10-gptq-gu0` | 1338 | 0.9808 |  | below quality gate |
| `mix30-gptq-gu0` | 1483 | 0.9872 |  | below quality gate |
| `mix30-gptq-sq50-a4` | 1483 | 0.9895 |  | below quality gate |
| `mix50-gptq-gu0` | 1629 | 0.9943 |  | pass (stage A only) |
| **`mix50-gptq-sq50-a4`** | 1629 | 0.9922 | +0.19 [-0.44, +0.85] | **pick**, pass |
| `e8-w8-rtn-a0` | 1993 | 0.9953 | +0.33 [-0.01, +0.74] | pass |
| `e8-w8-rtn-a4` | 1993 | 0.9772 |  | below quality gate |
| `w8-qkv0` | 1993 | 0.9780 |  | below quality gate |
| `w8-qkvgu0` | 1993 | 0.9929 |  | pass (stage A only) |
| `w8-sq50-a4` | 1993 | 0.9919 |  | pass (stage A only) |
| `w8-sq75-a4` | 1993 | 0.9924 |  | pass (stage A only) |
| `e32-w8-rtn-a4` | 2877 | 0.9772 |  | below quality gate |

### Qwen3-Reranker-0.6B

Base model [`Qwen/Qwen3-Reranker-0.6B`](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B), FP32 2386 MB. Files: [`nynefold/Qwen3-Reranker-0.6B-ONNX`](https://huggingface.co/nynefold/Qwen3-Reranker-0.6B-ONNX).

| variant | MB | Spearman mean (gate 0.98) | FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt) | verdict |
|---|---:|---:|---|---|
| `e8-w4-gptq-a0` | 465 | 0.9694 |  | below quality gate |
| `e8-w4-gptq-a4` | 465 | 0.9687 |  | below quality gate |
| `e8-w4-gptq-sq50-a4` | 465 | 0.9657 |  | below quality gate |
| `e8-w4-rtn-a4` | 465 | 0.9209 |  | below quality gate |
| `mix02-gptq-a4` | 469 | 0.9700 |  | below quality gate |
| `mix10-gptq-a4` | 488 | 0.9700 |  | below quality gate |
| `mix30-gptq-a4` | 533 | 0.9704 |  | below quality gate |
| `mix30-gptq-sq50-a4` | 533 | 0.9801 |  | pass (stage A only) |
| `mix50-gptq-a4` | 578 | 0.9781 | -0.92 [-1.77, -0.11] | below quality gate |
| `mix50-gptq-sq50-a4` | 578 | 0.9785 |  | below quality gate |
| `e8-w8-rtn-a0` | 692 | 0.9976 |  | pass (stage A only) |
| **`e8-w8-rtn-a4`** | 692 | 0.9899 | -0.02 [-0.48, +0.43] | **pick**, pass |

### zerank-1-small

Base model [`zeroentropy/zerank-1-small-reranker`](https://huggingface.co/zeroentropy/zerank-1-small-reranker), FP32 6885 MB. Files: [`nynefold/zerank-1-small-ONNX`](https://huggingface.co/nynefold/zerank-1-small-ONNX).

| variant | MB | Spearman mean (gate 0.98) | FreshRet Small nDCG@10 vs FP32 (gate -1.0 pt) | verdict |
|---|---:|---:|---|---|
| `e8-w4-gptq-a0` | 1266 | 0.9825 |  | pass (stage A only) |
| `e8-w4-gptq-a4` | 1266 | 0.9808 |  | pass (stage A only) |
| `e8-w4-gptq-sq50-a4` | 1266 | 0.9865 | -1.22 [-1.91, -0.58] | below quality gate |
| `mix10-gptq-a4` | 1338 | 0.9825 |  | pass (stage A only) |
| `mix10-gptq-sq50-a4` | 1338 | 0.9872 |  | pass (stage A only) |
| `mix30-gptq-a4` | 1483 | 0.9875 |  | pass (stage A only) |
| **`mix30-gptq-sq50-a4`** | 1483 | 0.9899 | -0.30 [-0.93, +0.33] | **pick**, pass |
| `mix50-gptq-a4` | 1629 | 0.9859 |  | pass (stage A only) |
| `mix50-gptq-sq50-a4` | 1629 | 0.9899 |  | pass (stage A only) |
| `e8-w8-rtn-a0` | 1993 | 0.9973 |  | pass (stage A only) |
| `e8-w8-rtn-a4` | 1993 | 0.9963 |  | pass (stage A only) |
| `w8-sq50-a4` | 1993 | 0.9963 | +0.14 [-0.08, +0.45] | pass |

### Variant names

| part | meaning |
|---|---|
| `e8`, `e4`, `e32` | token table at 8 bit, 4 bit, or kept FP32 |
| `w4`, `w8` | linear layers at 4 or 8 bit |
| `mixNN` | 4 bit, with the most sensitive tensors at 8 bit (NN% of the 4-to-8-bit size gap) |
| `rtn`, `mse`, `msew`, `hqq`, `gptq` | quantization algorithm (`msew` = Hessian-weighted MSE) |
| `b128`, `sym` | block size 128 (default 32); symmetric (default asymmetric) |
| `a0`, `a4` | ONNX Runtime `accuracy_level`: FP32 compute, or INT8 compute |
| `sq50`, `sq75` | SmoothQuant on the MLP inputs, alpha 0.5 / 0.75 |
| `gu0`, `qkv0`, `qkvgu0` | INT8 compute except FP32 for gate/up, q/k/v, or both |
| `pc` | vocabulary pruned to the tokens seen in the MIRACL corpus (pplx-embed) |


## Licences

The quantized models are derivatives of their base models and keep their licences: pplx-embed-v1-0.6b
MIT; F2LLM-v2-1.7B, Qwen3-Reranker-0.6B and zerank-1-small Apache-2.0. FreshRet Small is CC-BY-4.0.

## Licence of this repo

Text and figures: CC-BY-4.0. The example scripts `inference.py` and `pipeline_example.py`: MIT. See `LICENSE`.