Pollux-4B-Judge-mlx-q8

This is an 8-bit quantized MLX conversion of Pollux-4B-Judge (a Qwen3-4B based LLM-as-a-judge model).

It was converted from the original Hugging Face checkpoint using mlx-lm with 8-bit quantization (group size 64), and runs natively on Apple Silicon. On the sanity-check prompt it produced output identical to the bf16 model, at roughly half the memory and ~2x the generation throughput.

Architecture Qwen3ForCausalLM (4B)
Quantization 8-bit, group size 64, affine (~8.5 bits/weight)
Residual dtype bfloat16 (non-quantized tensors: norms, scales)
On-disk size ~4.3 GB
Peak inference memory ~4.4 GB

Note: config.json lists "dtype": "bfloat16" โ€” that is the type of the non-quantized tensors. The actual 8-bit quantization lives in the "quantization" block ({"bits": 8, "group_size": 64, "mode": "affine"}).

Installation

Requires Apple Silicon (M-series). Install mlx-lm:

pip install mlx-lm

You can run the model straight from the Hub (replace <your-hf-username> with the account you upload it to) or from a local path.

Prompt format

The model is an LLM-as-a-judge: it scores one response against one criterion and returns a numeric score (plus a <think> rationale). Format each evaluation as a single user message using this template:

### ะ—ะฐะดะฐะฝะธะต ะดะปั ะพั†ะตะฝะบะธ:
{instruction}

### ะญั‚ะฐะปะพะฝะฝั‹ะน ะพั‚ะฒะตั‚:
{reference_answer}

### ะžั‚ะฒะตั‚ ะดะปั ะพั†ะตะฝะบะธ:
{answer}

### ะšั€ะธั‚ะตั€ะธะน ะพั†ะตะฝะบะธ:
{criteria_name}

### ะจะบะฐะปะฐ ะพั†ะตะฝะธะฒะฐะฝะธั ะฟะพ ะบั€ะธั‚ะตั€ะธัŽ:
{criteria_rubrics}

The ### ะญั‚ะฐะปะพะฝะฝั‹ะน ะพั‚ะฒะตั‚: (reference answer) block is optional โ€” drop it when you have no gold answer.

Quick start (CLI)

mlx_lm.generate --model <your-hf-username>/Pollux-4B-Judge-mlx-q8 --temp 0.0 --max-tokens 512 \
  --prompt $'### ะ—ะฐะดะฐะฝะธะต ะดะปั ะพั†ะตะฝะบะธ:\nะกะบะพะปัŒะบะพ ะฑัƒะดะตั‚ 2+2?\n\n### ะญั‚ะฐะปะพะฝะฝั‹ะน ะพั‚ะฒะตั‚:\n4\n\n### ะžั‚ะฒะตั‚ ะดะปั ะพั†ะตะฝะบะธ:\nะ‘ัƒะดะตั‚ 4\n\n### ะšั€ะธั‚ะตั€ะธะน ะพั†ะตะฝะบะธ:\nะŸั€ะฐะฒะธะปัŒะฝะพัั‚ัŒ ะพั‚ะฒะตั‚ะฐ\n\n### ะจะบะฐะปะฐ ะพั†ะตะฝะธะฒะฐะฝะธั ะฟะพ ะบั€ะธั‚ะตั€ะธัŽ:\n0: ะ”ะฐะฝ ะฝะตะฟั€ะฐะฒะธะปัŒะฝั‹ะน ะพั‚ะฒะตั‚ ะธะปะธ ะพั‚ะฒะตั‚ ะพั‚ััƒั‚ัั‚ะฒัƒะตั‚.\n1: ะžั‚ะฒะตั‚ ะผะพะดะตะปะธ ะฝะตะฟะพะปะฝั‹ะน.\n2: ะžั‚ะฒะตั‚ ะผะพะดะตะปะธ ัะพะฒะฟะฐะดะฐะตั‚ ั ัั‚ะฐะปะพะฝะฝั‹ะผ ะธะปะธ ัะบะฒะธะฒะฐะปะตะฝั‚ะตะฝ ะตะผัƒ.'

Python

from mlx_lm import load, generate

model, tokenizer = load("<your-hf-username>/Pollux-4B-Judge-mlx-q8")

PROMPT_TEMPLATE = """### ะ—ะฐะดะฐะฝะธะต ะดะปั ะพั†ะตะฝะบะธ:
{instruction}

### ะญั‚ะฐะปะพะฝะฝั‹ะน ะพั‚ะฒะตั‚:
{reference_answer}

### ะžั‚ะฒะตั‚ ะดะปั ะพั†ะตะฝะบะธ:
{answer}

### ะšั€ะธั‚ะตั€ะธะน ะพั†ะตะฝะบะธ:
{criteria_name}

### ะจะบะฐะปะฐ ะพั†ะตะฝะธะฒะฐะฝะธั ะฟะพ ะบั€ะธั‚ะตั€ะธัŽ:
{criteria_rubrics}
"""

prompt = PROMPT_TEMPLATE.format(
    instruction="ะกะบะพะปัŒะบะพ ะฑัƒะดะตั‚ 2+2?",
    reference_answer="4",
    answer="ะ‘ัƒะดะตั‚ 4",
    criteria_name="ะŸั€ะฐะฒะธะปัŒะฝะพัั‚ัŒ ะพั‚ะฒะตั‚ะฐ",
    criteria_rubrics=(
        "0: ะ”ะฐะฝ ะฝะตะฟั€ะฐะฒะธะปัŒะฝั‹ะน ะพั‚ะฒะตั‚ ะธะปะธ ะพั‚ะฒะตั‚ ะพั‚ััƒั‚ัั‚ะฒัƒะตั‚.\n"
        "1: ะžั‚ะฒะตั‚ ะผะพะดะตะปะธ ะฝะตะฟะพะปะฝั‹ะน.\n"
        "2: ะžั‚ะฒะตั‚ ะผะพะดะตะปะธ ัะพะฒะฟะฐะดะฐะตั‚ ั ัั‚ะฐะปะพะฝะฝั‹ะผ ะธะปะธ ัะบะฒะธะฒะฐะปะตะฝั‚ะตะฝ ะตะผัƒ."
    ),
)

messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt=text, max_tokens=512, verbose=True))

OpenAI-compatible server + curl

Start the server (it exposes /v1/chat/completions on port 8080):

mlx_lm.server --model <your-hf-username>/Pollux-4B-Judge-mlx-q8 --port 8080

Then send a judge request:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Pollux-4B-Judge-mlx-q8",
    "temperature": 0.0,
    "max_tokens": 512,
    "messages": [
      {"role": "user", "content": "### ะ—ะฐะดะฐะฝะธะต ะดะปั ะพั†ะตะฝะบะธ:\nะกะบะพะปัŒะบะพ ะฑัƒะดะตั‚ 2+2?\n\n### ะญั‚ะฐะปะพะฝะฝั‹ะน ะพั‚ะฒะตั‚:\n4\n\n### ะžั‚ะฒะตั‚ ะดะปั ะพั†ะตะฝะบะธ:\nะ‘ัƒะดะตั‚ 4\n\n### ะšั€ะธั‚ะตั€ะธะน ะพั†ะตะฝะบะธ:\nะŸั€ะฐะฒะธะปัŒะฝะพัั‚ัŒ ะพั‚ะฒะตั‚ะฐ\n\n### ะจะบะฐะปะฐ ะพั†ะตะฝะธะฒะฐะฝะธั ะฟะพ ะบั€ะธั‚ะตั€ะธัŽ:\n0: ะ”ะฐะฝ ะฝะตะฟั€ะฐะฒะธะปัŒะฝั‹ะน ะพั‚ะฒะตั‚ ะธะปะธ ะพั‚ะฒะตั‚ ะพั‚ััƒั‚ัั‚ะฒัƒะตั‚.\n1: ะžั‚ะฒะตั‚ ะผะพะดะตะปะธ ะฝะตะฟะพะปะฝั‹ะน.\n2: ะžั‚ะฒะตั‚ ะผะพะดะตะปะธ ัะพะฒะฟะฐะดะฐะตั‚ ั ัั‚ะฐะปะพะฝะฝั‹ะผ ะธะปะธ ัะบะฒะธะฒะฐะปะตะฝั‚ะตะฝ ะตะผัƒ."}
    ]
  }'

The model returns a short <think> rationale followed by the numeric score (2 for this example; 0 if you swap in a wrong answer).

See the original model card for evaluation details, full rubrics, and intended use.

Downloads last month
28
Safetensors
Model size
4B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ledgergap/Pollux-4B-Judge-mlx-q8

Finetuned
Qwen/Qwen3-4B
Quantized
(2)
this model