Qwen3-Reranker-4B-oQ8e-bf16

An MLX oQ8e quantization of Qwen/Qwen3-Reranker-4B, created with oMLX 0.7.3rc1 on Apple Silicon.

This model scores the relevance of a document to a query. It is a reranker, not an embedding model or a general-purpose chat model.

Quantization

Property	Value
Source model	Qwen/Qwen3-Reranker-4B, original checkpoint
Architecture	Qwen3ForCausalLM
Quantization	oQ8e: 8-bit affine with enhanced, importance-matrix-weighted quantization
Group size	64
Non-quantized floating-point tensors	BF16 (bfloat16)
Quantization scales and biases	BF16
Packed quantized weights	U32 storage
MTP	Not included; not needed for yes/no logit scoring

The saved Safetensors header was checked directly: 651 BF16 tensors and 253 U32 tensors, with no FP16 tensors. Of the BF16 tensors, 506 are quantization scales/biases and 145 are other tensors. config.json declares torch_dtype: "bfloat16", matching the stored floating-point tensors.

The oMLX calibration report records 128 samples of 512 tokens from the bundled oqe_code_multilingual dataset. Enhanced quantization was applied to 252 modules, with no missing or mismatched imatrix entries. This is calibration information, not a retrieval benchmark.

Download

hf download trigger2k25/Qwen3-Reranker-4B-oQ8e-bf16 \
  --local-dir ./Qwen3-Reranker-4B-oQ8e-bf16

Score query/document pairs with MLX

On a compatible Apple Silicon Mac, install MLX LM in your Python environment:

python3 -m pip install mlx-lm

The following example follows the original model's yes/no logit scoring recipe. Higher scores indicate greater relevance. It processes one document at a time and does not generate text.

import mlx.core as mx
from mlx_lm import load

model, tokenizer = load("trigger2k25/Qwen3-Reranker-4B-oQ8e-bf16")

prefix = (
    '<|im_start|>system\nJudge whether the Document meets the requirements '
    'based on the Query and the Instruct provided. Note that the answer '
    'can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n'
)
suffix = '<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
prefix_ids = tokenizer.encode(prefix, add_special_tokens=False)
suffix_ids = tokenizer.encode(suffix, add_special_tokens=False)
no_id = tokenizer.convert_tokens_to_ids("no")
yes_id = tokenizer.convert_tokens_to_ids("yes")

def score(query, document, instruction=None, max_length=8192):
    if instruction is None:
        instruction = "Given a web search query, retrieve relevant passages that answer the query"
    text = f"<Instruct>: {instruction}\n<Query>: {query}\n<Document>: {document}"
    budget = max_length - len(prefix_ids) - len(suffix_ids)
    if budget <= 0:
        raise ValueError("max_length is too small for the prompt template")
    body_ids = tokenizer.encode(text, add_special_tokens=False)[:budget]
    inputs = mx.array([prefix_ids + body_ids + suffix_ids])
    logits = model(inputs)[:, -1, :].astype(mx.float32)
    pair_logits = mx.stack([logits[0, no_id], logits[0, yes_id]])
    return mx.softmax(pair_logits)[1].item()

query = "What is the capital of France?"
documents = ["Paris is the capital of France.", "Jupiter is a gas giant."]
ranked = sorted(
    [(score(query, document), document) for document in documents],
    reverse=True,
)
for relevance, document in ranked:
    print(f"{relevance:.6f}\t{document}")

The example truncates query/document input to fit 8,192 tokens including the template. Scores are model relevance scores, not calibrated probabilities of factual correctness. For production, evaluate truncation, batching, ranking quality and latency on your own documents.

MLX-format weights can also be used in servers that support this quantization and Qwen3 reranking. Compatibility with every server/version is not implied.

Validation and limitations

• Configuration, tensor dtypes and Safetensors byte ranges were inspected locally.
• The Python scoring example above was tested with MLX LM 0.32.3.dev8 using this local checkpoint: the relevant Paris passage scored approximately 0.928409 and the unrelated Jupiter passage 0.000045. This is a smoke test, not a benchmark.
• No claim is made that BF16 is faster or more accurate than an FP16 oQ8 build.
• No full retrieval benchmark or equivalence evaluation against the original model has been performed.
• Original model capabilities, limitations and recommended prompting are described in the upstream model card.

Attribution and license

Original model: Qwen team, Qwen/Qwen3-Reranker-4B, Apache License 2.0.

Quantization: trigger2k25, using oMLX oQ. This repository distributes quantized derivative weights and the accompanying tokenizer/configuration files. It is not an official Qwen release.