Arete-27B NVFP4
NousResearch/Arete-27B quantized with ModelOpt 0.48.0 and NVIDIA's recipe for Qwen3.8-27B (the layout of nvidia/Qwen3.8-27B-NVFP4). Made and measured on a DGX Spark (GB10).
Layers
| Part | Format |
|---|---|
| MLP, lm_head | NVFP4 (Local-Hessian) |
| Full and linear attention | FP8 |
| Embeddings, vision | BF16 |
Calibration: 512 samples from 7 Nemotron post-training v3 sets. Base revision 108f0afb.
Quality against BF16
Thinking off, temperature 0. Both measured by me with the same harness and vLLM build.
| GPQA Diamond (198) | IFEval (200) | |
|---|---|---|
| BF16 | 81.8% | 85.5% |
| This model | 80.3% | 83.0% |
Speed on one DGX Spark
One user, vLLM 0.33.0 nightly. Size 20 GB (BF16 is 55 GB).
| Context | Decode |
|---|---|
| short | 13.2 tok/s |
| 32K | 11.8 tok/s |
| 64K | 10.7 tok/s |
| 128K | 9.1 tok/s |
128K prompt read: 82 s. Second turn on the same 128K: 1.1 s with prefix caching. Lowest free memory during the 128K run: 65 GiB.
Run
vllm serve atakans/Arete-27B-NVFP4 --max-model-len 262144 --reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prefix-caching
Needs Blackwell (NVFP4). No MTP weights in Arete, so no draft model.
- Downloads last month
- 133