Nemotron-3.5-Rerank-8B-MLX-8bit

An 8-bit MLX build of nvidia/Nemotron-3.5-Rerank-8B-BF16 (snapshot f88f8d91), produced by OpenGryd. The text path (bidirectional Ministral3-8B tower + the checkpoint's 4096→1 score head) is quantized uniformly to 8-bit affine weights (group 64); the vision encoder and projector are neither shipped nor served — this is a text-only artifact. 7.87 GiB of quantized weights.

Serving

mlx_lm/mlx_vlm cannot load this checkpoint (bidirectional Mistral3ForSequenceClassification) — use this repository's own module:

import nemotron35_rerank_mlx as nem

model, tok = nem.load("OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit")

scores = nem.rerank(model, tok, "What is NVIDIA NeMo Framework?", [
    "NeMo Framework offers comprehensive functionalities for developing both LLMs and Multimodal Models.",
    "A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences.",
])
# sigmoid(relevance logit) per document — higher = more relevant;
# monotone in the raw logit, so ranking matches the vendor's raw-logit
# order.

The pair tokenizer renders the checkpoint's trained format — query: {q}\n\npassage: {d} with the template's literal-[/< escaping and add_special_tokens=False (BOS/EOS deliberately omitted, matching training token-for-token).

Fidelity

Verified against the PyTorch reference (transformers 5.20, CPU fp32, AutoModelForSequenceClassification with the vendor card's query/document template recipe) on a multilingual text battery: identical ranking, worst-case |Δlogit| documented in the conversion notes (2026-11-12). The upstream BF16 MLX forward was compared in the same run.

License

OpenMDW-1.1 (see LICENSE, NOTICE, THIRD_PARTY_NOTICES.md, inherited verbatim from the upstream release). The Ministral base models are Apache-2.0.

Downloads last month
79
Safetensors
Model size
8B params
Tensor type
U32
·
BF16
·
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit