leonsarmiento/Arete-27B-oQ4e

Data-aware oQ4e MLX quantization of NousResearch/Arete-27B — global 4-bit with imatrix-guided precision elevation to 5/6/8-bit on the tensors that need it. Vision tower preserved (multimodal via mlx-vlm).

About Arete-27B

Nous Research post-trained Qwen/Qwen3.8-27B to reinforce western values on a Chinese-trained model. Pure supervised fine-tuning — no RL, no preference optimization, no architectural modification: ~20K prompts / 52.8M tokens across seven value categories (freedom of expression, state ideology, state vs. free market, education, data privacy & surveillance, democracy, environmental policy), distilled from the open teacher Inkling, trained at 64k context over 2 epochs.

Measured on the Propaganda Eval (803 prompts, 7 categories): fact mean 2.32 → 2.98 (out of 5) and answers containing propaganda 22% → 0.9%, with the largest change in freedom of expression (fact score 0.89 → 2.44, propaganda 60% → 1%). General capability held: Hermes Bench 65.77% → 65.60% (within noise), Terminal-Bench-2.1 67% → 70%.

Training details worth knowing for quantization: all language-model weights were trained, the vision encoder was frozen (text-only data), and the MTP head was disabled during training — and the source repo ships no mtp.* tensors at all, so this quant carries no bundled MTP heads (nothing to graft). text_config.mtp_num_hidden_layers stays 0.

About oQ4e (data-aware quantization)

Unlike data-agnostic recipes (uniform bit-widths, or static importance heuristics), oQ4e runs a calibration pass over real activations and elevates the tensors whose quantization error the imatrix flags as costly. The trade-off: data-aware methods can skew representation toward the calibration domain — here oqe_code_multilingual (coding + multilingual text).

Calibration: oqe_code_multilingual · 128 samples × 512 ctx (adaptive budget up to 1024) · 496 activation entries · mlx-vlm qwen3_5 model class.

Quantization Details

• Global default: 4-bit affine, group_size=64 (everything not listed below, including the vision tower)
• Elevated to 5-bit (235 tensors, imatrix-driven):
  • embed_tokens — 5-bit
  • All 16 full-attention layers (every 4th layer, full_attention_interval: 4): 62 self_attn q/k/v/o tensors
  • mlp.down_proj of layers 0–17 (18 tensors) — early-layer FFN sensitivity
  • linear_attn.in_proj across most layers (154 tensors)
• Elevated to 6-bit (4 tensors): lm_head, layers.63.self_attn.{k,v}_proj, layers.52.linear_attn.in_proj_a
• Elevated to 8-bit (4 tensors): layers.50/62.linear_attn.in_proj_{a,b} — the hottest tensors in the calibration pass
• Norms: bf16
• Vision tower: preserved at 4-bit (frozen during the source fine-tune)
• Chat template: byte-identical to source, reasoning_effort defaulting to xhigh
• MTP: none (source ships none — see About)

Stats: 17.19 GB · 4 shards · ~5.03 bpw · mlx-vlm · vision preserved · all shards carry format: mlx metadata.

Usage

Works in LM Studio and oMLX. No Lightning MTP on this build (no MTP heads in the source) — for MTP-accelerated Qwen3.8-27B siblings, see leonsarmiento/Qwen3.8-27B-heretic-ara-4bit-mtp-mlx.

Reasoning parsing / tool calling: reasoning_parser: qwen3, tool_call_parser: qwen3_xml.

Sampling — thinking mode (Qwen3.8 family defaults): temperature 1.0, top_p 0.95, top_k 20.

Sampling — non-thinking mode (Qwen recommended): temperature 0.7, top_p 0.8, presence_penalty 1.5, with enable_thinking: false.

Reasoning effort: the chat template accepts reasoning_effort ∈ {low, medium, high, xhigh} (default xhigh).

(The source repo documents no sampling recommendations; the above are the Qwen3.8-27B family defaults this fine-tune inherits.)

Source

NousResearch/Arete-27B — built on Qwen/Qwen3.8-27B: dense hybrid-attention VLM, 64 layers (48 linear attention + 16 full attention), 262K context, vision tower. Blog post · Evaluation dataset.

Acknowledgements

Thanks to Nous Research for Arete-27B and the open Propaganda Eval, and to Jundot for the oQ4e data-aware recipe this quant follows.