Arete-27B-NVFP4

Mixed-precision NVFP4 + FP8 post-training quantization of NousResearch/Arete-27B, produced with NVIDIA Model Optimizer (ModelOpt 0.48) following the recipe NVIDIA published for nvidia/Qwen3.8-27B-NVFP4 — the same architecture family (Qwen3_5ForConditionalGeneration).

• Base model: NousResearch/Arete-27B (64 hybrid text layers + 27-block vision tower, vocab 248320)
• Quantization: MIXED_PRECISION — MLP and lm_head in NVFP4 (Local-Hessian calibration), attention projections in FP8 per-tensor, everything else BF16
• Calibration: 2,048 sequences × 2,048 tokens
• Checkpoint size: ~20 GiB in 3 shards
• Produced by: quantizing the upstream release; no additional post-training

What is quantized

Module	Format	Notes
mlp.gate_proj, mlp.up_proj, mlp.down_proj (64 layers)	NVFP4	E2M1, block 16, static block scales, FP8 E4M3 block scales
lm_head	NVFP4	same block format
self_attn.q/k/v/o_proj (16 full-attention layers)	FP8	E4M3 per-tensor, W8A8
linear_attn.in_proj_qkv, in_proj_z, out_proj (48 GDN layers)	FP8	E4M3 per-tensor, W8A8
embed_tokens, vision tower (model.visual.*), conv1d, in_proj_a/b, A_log, dt_bias, norms	BF16	401 quantized modules total (193 NVFP4 + 208 FP8)

Quantizer-level parity with the NVIDIA reference checkpoint was verified exhaustively: 1,699 quantizers compared, 0 differences (scripts/compare_recipe.py).

Requirements

Blackwell-class hardware (SM ≥ 100) for the NVFP4 half of the model. Supported runtimes:

• vLLM ≥ 0.29 — detects the checkpoint as modelopt_mixed automatically
• TensorRT-LLM
• Transformers is not supported for pre-quantized NVFP4/FP8 ModelOpt checkpoints (5.x only implements on-the-fly quantization)

Usage

vLLM

vllm serve veladan/Arete-27B-NVFP4 \
    --tensor-parallel-size 1 \
    --max-model-len 32768

vLLM infers quantization=modelopt_mixed from hf_quant_config.json; passing --quantization modelopt_fp4 also resolves to the mixed loader.

TensorRT-LLM

trtllm-serve veladan/Arete-27B-NVFP4 --max_batch_size 8

Quantization details

Property	Value
Method	Mixed precision: NVFP4 (W4A4, block 16, static block scales) + FP8 (W8A8, per-tensor)
Calibration algorithm	Local-Hessian with FP8 scale sweep for the NVFP4 weights; max for FP8
Calibration data	2,048 sequences × 2,048 tokens
Output format	Hugging Face safetensors + hf_quant_config.json + .quant_summary.txt

Calibration corpus — deviation from the NVIDIA recipe

NVIDIA calibrated this recipe on the Nemotron post-training dataset (nvidia/Nemotron-Post-Training-Dataset-v2, gated). That corpus was not accessible during this build, so calibration used an open mixture instead:

Source	Share	Domain
HuggingFaceTB/smoltalk	50 %	general chat / instructions
nvidia/OpenCodeInstruct	20 %	code
AI-MO/NuminaMath-CoT	30 %	math / reasoning

The quantization recipe (formats, block sizes, algorithms, excluded modules) is identical to NVIDIA's; only the calibration corpus differs. This is recorded in quantization_metadata.json.

How this checkpoint was produced

Everything below is reproducible with the scripts in the accompanying repository (scripts/recipe.py, scripts/quantize.py, scripts/compare_recipe.py, scripts/validate_quant.py).

Hardware: 1× NVIDIA RTX PRO 6000 Blackwell Max-Q (96 GB), 2× in total on the build host. CUDA 13.2, driver 615.71.

Software: nvidia-modelopt==0.48.0, torch==2.15.0+cu132, transformers==5.22.2, triton==3.8.0 (pinned — see note below), flash-linear-attention==0.6.0.

Recipe (--preset nvidia-mixed): ModelOpt's NVFP4_W4A4_WEIGHT_LOCAL_HESSIAN_CFG as the base — NVFP4 E2M1 weights with block size 16 and static block scales, activations in FP4 with dynamic block scaling — plus two overrides on top:

1. self-attention (q/k/v/o_proj) and linear-attention (GDN) projections (in_proj_qkv, in_proj_z, out_proj) switched to FP8 E4M3 per-tensor (num_bits=(4,3), axis=None), weight and input quantizers only.
2. lm_head enabled for NVFP4 (ModelOpt excludes it by default; NVIDIA quantizes it).

Embeddings, the vision tower, conv1d, in_proj_a/b, A_log and dt_bias stay in BF16, as in the reference checkpoint.

Calibration: the Local-Hessian algorithm (method=local_hessian, fp8_scale_sweep=True) with layer-by-layer calibration enabled; it runs a max pass over all quantizers first, then one Hessian-accumulation pass, then an MSE weight search per block. 2,048 sequences of 2,048 tokens.

Command:

python scripts/quantize.py --preset nvidia-mixed \
    --calib-kind nemotron --calib-samples 2048 --calib-seqlen 2048
python scripts/compare_recipe.py --ref nvidia/Qwen3.8-27B-NVFP4 \
    --summaries output/Arete-27B-NVFP4/.quant_summary.txt <ref>/.quant_summary.txt
python scripts/validate_quant.py --layers 5

Measured cost: model load 15 s, calibration tokenization 19 s, quantization 55 min, export 2 min.

Verification performed: quantizer-by-quantizer comparison against the NVIDIA reference checkpoint (1,699 quantizers, 0 differences), tensor layout comparison for 14 module categories (identical dtype/shape for every weight and scale), and weight reconstruction error against the BF16 source (table below).

Known deviations from the reference:

Item	NVIDIA reference	This checkpoint
Calibration corpus	Nemotron post-training dataset (gated)	Open mix: smoltalk 50 %, OpenCodeInstruct 20 %, NuminaMath-CoT 30 %
hf_quant_config.json exclude_modules	lists only mtp*	lists the 146 explicitly excluded modules (equivalent information; both encode the same quantization)
MTP layer	1 layer (excluded from quantization)	none (Arete-27B has mtp_num_hidden_layers=0)

Toolchain note: ModelOpt 0.48's NVFP4 dynamic-block fake-quantization kernel uses Triton block pointers, which the Triton 3.9 compiler no longer accepts (Block pointers have been removed in favor of the tensor descriptor API). The build pins triton==3.8.0; with 3.9 the calibration aborts on the first forward pass.

Verification

Weight-level reconstruction error versus the BF16 source (scripts/validate_quant.py, dequantizing sample tensors):

Format	Mean relative L2 error	Tensors sampled
FP8 (attention projections)	0.0266	5
NVFP4 (MLP, lm_head)	0.0874	3

For reference, a plain NVFP4-with-max-calibration attempt of the same model gave 0.0950 on the MLP tensors: the Local-Hessian calibration used here is measurably better, which is why the NVIDIA recipe was adopted.

Smoke evaluation (arc_easy, 200 questions, 0-shot, served by vLLM): 0.825. Full benchmark numbers are pending — see Evaluation.

Evaluation

Both checkpoints were served side by side with vLLM 0.29 on identical hardware and evaluated with lm-evaluation-harness using the same settings (MMLU: 5-shot via /v1/completions; GSM8K 5-shot and IFEval 0-shot via /v1/chat/completions with the model's chat template, greedy decoding).

Benchmark	Arete-27B (BF16)	Arete-27B-NVFP4	Δ
MMLU (14,042 q, 5-shot, acc)	0.8336	0.8220	−0.0115
GSM8K (1,319 q, 5-shot, strict)	0.4041	0.4185	+0.0144
GSM8K (flexible extract)	0.6073	0.6262	+0.0190
IFEval (541 prompts, prompt-level strict)	0.3161	0.3142	−0.0018
IFEval (instruction-level loose)	0.5132	0.5024	−0.0108

Averaged over the 64 MMLU subjects the quantized model is −0.0097 below BF16, with per-subject standard deviation comparable to the difference. GSM8K moves in the opposite direction, i.e. within run-to-run noise for a benchmark of this size (σ ≈ 0.014).

Reading of these numbers: the only consistent, measurable degradation is a ~1 point MMLU drop; conversation-style and math accuracy are unchanged within noise. This is the expected profile of the NVIDIA recipe, which keeps attention in FP8 and only pushes the MLP weights to 4 bits.

Not yet measured: GPQA Diamond, Terminal-Bench, AA-LCR, MMMU-Pro, SciCode and IFBench — the benchmarks NVIDIA lists for the reference checkpoint. They require additional harnesses (agentic execution, multimodal inputs), so the comparison above is text-only.

Limitations

• Quantization is lossy. Weight error is ~9 % (NVFP4) and ~3 % (FP8) relative L2; end-to-end effect on reasoning-heavy tasks is not yet measured.
• Calibrated on English chat/code/math data. Other languages or specialized domains may degrade relative to the BF16 model.
• The vision tower remains in BF16 and was not calibrated with image inputs; image understanding inherits the base model without quantization-specific tuning.
• Outputs may contain biases, inaccuracies or unsafe content inherited from the base model. Not intended for high-stakes use without human oversight.

Ethical considerations & safety

This is a compressed derivative of NousResearch/Arete-27B and inherits its behaviour, including biases and failure modes. No safety tuning was performed or removed during quantization. Users are responsible for evaluating fitness for purpose, complying with applicable law, and disclosing that outputs are machine-generated.

Citation

@misc{arete27b_nvfp4,
  title  = {Arete-27B-NVFP4},
  author = {veladan},
  year   = {2025},
  note   = {Mixed NVFP4/FP8 quantization of NousResearch/Arete-27B via NVIDIA Model Optimizer},
  url    = {https://huggingface.co/veladan/Arete-27B-NVFP4}
}

License

Released under the Apache License 2.0 (see LICENSE), consistent with the upstream model. The base model's terms (https://huggingface.co/NousResearch/Arete-27B) continue to apply to any derivative work. Calibration datasets are used only for calibration and are not redistributed.