leonsarmiento/Arete-27B-4bit-mlx

Uniform 4-bit MLX quantization of NousResearch/Arete-27B — plain vanilla, no bundled MTP. Vision tower preserved (multimodal via mlx-vlm). Works in LM Studio and oMLX.

About Arete-27B

Nous Research post-trained Qwen/Qwen3.8-27B to reinforce western values on a Chinese-trained model. Pure supervised fine-tuning — no RL, no preference optimization, no architectural modification: ~20K prompts / 52.8M tokens across seven value categories (freedom of expression, state ideology, state vs. free market, education, data privacy & surveillance, democracy, environmental policy), distilled from the open teacher Inkling, trained at 64k context over 2 epochs.

Measured on the Propaganda Eval (803 prompts, 7 categories): fact mean 2.32 → 2.98 (out of 5) and answers containing propaganda 22% → 0.9%, with the largest change in freedom of expression (fact score 0.89 → 2.44, propaganda 60% → 1%). General capability held: Hermes Bench 65.77% → 65.60% (within noise), Terminal-Bench-2.1 67% → 70%.

Training details worth knowing for quantization: all language-model weights were trained, the vision encoder was frozen (text-only data), and the MTP head was disabled during training — and the source repo ships no mtp.* tensors at all, so this quant carries no bundled MTP heads (nothing to graft). text_config.mtp_num_hidden_layers stays 0. For MTP-accelerated Qwen3.8-27B siblings, see leonsarmiento/Qwen3.8-27B-heretic-ara-4bit-mtp-mlx.

For the data-aware alternative of the same model, see leonsarmiento/Arete-27B-oQ4e (imatrix-guided 4/5/6/8-bit mix, +1.6 GB).

Quantization Details

• All quantized layers (attention, linear attention, MLP, embeddings, lm_head, vision tower): uniform 4-bit affine, group_size=64
• Norms: bf16
• Vision tower: preserved at 4-bit (multimodal via mlx-vlm)
• Chat template: byte-identical to source, reasoning_effort defaulting to xhigh
• MTP: none (source ships none — see About)

Stats: 15.62 GB · 3 shards · ~4.56 bpw · mlx-vlm · vision preserved · all shards carry format: mlx metadata.

Verification

• Offline (23/23 PASS): shard metadata + index consistency, zero mtp.* keys, norms bf16, global 4-bit g64 affine config, chat template present in both chat_template.jinja and tokenizer_config.json. Dequant spot-checks against the source BF16 across embed / linear-attn / full-attn / early+late MLP / vision: rel-L1 0.097–0.104 (text) and 0.166 (vision), correlation ≥ 0.992 — the expected envelope for 4-bit g64 affine on this architecture.
• Runtime (oMLX): loads and generates coherently — math (15×37=555), fact recall (Canberra), clean <think> reasoning with proper turn closure.

Usage

Reasoning parsing / tool calling: reasoning_parser: qwen3, tool_call_parser: qwen3_xml.

Sampling — thinking mode (Qwen3.8 family defaults): temperature 1.0, top_p 0.95, top_k 20.

Sampling — non-thinking mode (Qwen recommended): temperature 0.7, top_p 0.8, presence_penalty 1.5, with enable_thinking: false.

Reasoning effort: the chat template accepts reasoning_effort ∈ {low, medium, high, xhigh} (default xhigh).

(The source repo documents no sampling recommendations; the above are the Qwen3.8-27B family defaults this fine-tune inherits.)

Source

NousResearch/Arete-27B — built on Qwen/Qwen3.8-27B: dense hybrid-attention VLM, 64 layers (48 linear attention + 16 full attention), 262K context, vision tower. Blog post · Evaluation dataset.

Acknowledgements

Thanks to Nous Research for Arete-27B and the open Propaganda Eval.