Instructions to use OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit --local-dir Nemotron-3.5-Rerank-8B-MLX-8bit
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Nemotron-3.5-Rerank-8B-MLX-8bit
An 8-bit MLX build of nvidia/Nemotron-3.5-Rerank-8B-BF16
(snapshot f88f8d91), produced by OpenGryd. The text path (bidirectional
Ministral3-8B tower + the checkpoint's 4096→1 score head) is quantized
uniformly to 8-bit affine weights (group 64); the vision encoder and
projector are neither shipped nor served — this is a text-only
artifact. 7.87 GiB of quantized weights.
Serving
mlx_lm/mlx_vlm cannot load this checkpoint (bidirectional
Mistral3ForSequenceClassification) — use this repository's own module:
import nemotron35_rerank_mlx as nem
model, tok = nem.load("OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit")
scores = nem.rerank(model, tok, "What is NVIDIA NeMo Framework?", [
"NeMo Framework offers comprehensive functionalities for developing both LLMs and Multimodal Models.",
"A biological foundation model designed to analyze and generate DNA, RNA, and protein sequences.",
])
# sigmoid(relevance logit) per document — higher = more relevant;
# monotone in the raw logit, so ranking matches the vendor's raw-logit
# order.
The pair tokenizer renders the checkpoint's trained format —
query: {q}\n\npassage: {d} with the template's literal-[/<
escaping and add_special_tokens=False (BOS/EOS deliberately omitted,
matching training token-for-token).
Fidelity
Verified against the PyTorch reference (transformers 5.20, CPU fp32,
AutoModelForSequenceClassification with the vendor card's
query/document template recipe) on a multilingual text battery:
identical ranking, worst-case |Δlogit| documented in the conversion
notes (2026-11-12). The upstream BF16 MLX forward was compared in the
same run.
License
OpenMDW-1.1 (see LICENSE, NOTICE, THIRD_PARTY_NOTICES.md,
inherited verbatim from the upstream release). The Ministral base
models are Apache-2.0.
- Downloads last month
- 79
Quantized
Model tree for OpenGryd/Nemotron-3.5-Rerank-8B-MLX-8bit
Base model
mistralai/Ministral-3-8B-Base-2512