Instructions to use DAXZEIT/Arete-27B-NInfer-v3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NInfer
How to use DAXZEIT/Arete-27B-NInfer-v3 with NInfer:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Arete-27B β NInfer v3
NousResearch/Arete-27B in NInfer v3 format, with Q4_K_M and Q6_K text weights, a native Q8 vision projector, and embedded MTP weights. Separate variants also include a DFlash2 draft model.
The Q4 version targets 24 GB GPUs. A tested RTX 3090 serving configuration uses a 160,000-token context capacity, RK8V4 KV cache, resident GPU vision, and MTP3.
Available files
| Variant | File size | Text | Vision | MTP | DFlash2 |
|---|---|---|---|---|---|
| Q4_K_M | 16.76 GiB | Q4_K_M | Native Q8 | Q6_K / Q8_0 | β |
| Q4_K_M + DFlash2 | 18.83 GiB | Q4_K_M | Native Q8 | Q6_K / Q8_0 | Q8 / BF16 |
| Q6_K | 22.74 GiB | Q6_K | Native Q8 | Q6_K / Q8_0 | β |
| Q6_K + DFlash2 | 24.81 GiB | Q6_K | Native Q8 | Q6_K / Q8_0 | Q8 / BF16 |
MTP and DFlash2 are alternative speculative decoding paths. File sizes describe the containers; runtime memory also depends on context, KV cache, component residency and the selected decoding path.
Q4 vs Q6: numerical comparison
The two text quants were evaluated on 6,144 next-token positions across six domains, using the same token sequences and runtime settings.
Global results:
- Perplexity: Q4 4.17055, Q6 4.16112 β a 0.23% reduction with Q6.
- Top-1 token agreement: 95.82% between Q4 and Q6.
- Mean KL divergence: 0.01033 nats for Q4 β Q6 and 0.01024 nats for Q6 β Q4.
Q4 remains close to Q6 on these segments. Q6 has slightly lower pooled perplexity and lower perplexity in four of the six domains.
Per-domain results
| Domain | Evaluated tokens | Q4 PPL β | Q6 PPL β | KL Q4 β Q6 β | KL Q6 β Q4 β | Top-1 agreement |
|---|---|---|---|---|---|---|
| arXiv | 1,024 | 6.58308 | 6.49938 | 0.01110 | 0.01109 | 94.43% |
| GitHub code | 1,024 | 1.52209 | 1.52766 | 0.00458 | 0.00453 | 98.24% |
| Own-code segment | 1,024 | 2.74500 | 2.73666 | 0.00985 | 0.00975 | 96.68% |
| Synthetic chat | 1,024 | 4.02362 | 4.01489 | 0.01176 | 0.01142 | 95.90% |
| Wikipedia EN | 1,024 | 6.70140 | 6.74311 | 0.01041 | 0.01050 | 94.92% |
| Wikipedia ZH | 1,024 | 7.09524 | 7.05675 | 0.01429 | 0.01413 | 94.73% |
| Pooled | 6,144 | 4.17055 | 4.16112 | 0.01033 | 0.01024 | 95.82% |
Measurement setup
- GPU: NVIDIA RTX 3090, 24 GB.
- Runtime: NInfer revision
39a10890, with a logits-capture hook. - Inputs: one fixed 4,096-token sequence per domain; the final 1,024 positions scored using their preceding context, without an added BOS, EOS or chat template.
- Execution: BF16 KV cache, embeddings in host RAM for both quants, concurrency 1; speculative decoding, vision, graphs, cache reuse and A8 prefill disabled.
- Distributions: BF16 logits captured from the physical output vocabulary; metrics computed over the complete 248,077-token public vocabulary.
- Reference: Q6 is the comparison reference, not the original BF16 model. Top-1 agreement measures agreement between quants, not task accuracy.
- Aggregation: global NLL and perplexity are computed from all 6,144 scored positions, not by averaging the per-domain perplexities. Global NLL is 1.42804835 for Q4 and 1.42578343 for Q6.
Each domain was captured in Q4 β Q6 β Q4 order. All six Q4 repeats were byte-identical. Every capture passed the internal bitwise-F32 target-logprob oracle, all 18,432 independent CPU row checks passed the 5e-5 absolute-error threshold, and all GPU traces completed with zero dropped events.
Measured Q4 serving performance
Long-generation benchmark on an RTX 3090, comparing no speculation with MTP3 / full draft head. Each measured request generated 4,096 tokens.
Prose versus code
| Workload | No speculation | MTP3 | Speedup | MTP acceptance |
|---|---|---|---|---|
| English technical prose | 49.00 tok/s | 88.28 tok/s | 1.80Γ | 49.00% |
| French prose | 48.89 tok/s | 83.92 tok/s | 1.72Γ | 45.00% |
| Python code + explanation | 48.86 tok/s | 94.97 tok/s | 1.94Γ | 55.30% |
| Pooled | 48.92 tok/s | 88.83 tok/s | 1.82Γ | 49.55% |
Across the two prose workloads, pooled throughput is 48.95 tok/s without speculation and 86.04 tok/s with MTP3, a 1.76Γ speedup. The code-oriented workload reaches 94.97 tok/s, with higher MTP acceptance than either prose workload.
Latency and end-to-end throughput
| Metric | No speculation | MTP3 |
|---|---|---|
| Median client time to first token | 241.50 ms | 244.03 ms |
| Client TTFT range | 239.98β338.79 ms | 242.82β350.49 ms |
| Pooled end-to-end throughput | 48.77 tok/s | 88.31 tok/s |
Benchmark setup
- Runtime / model: NInfer
39a10890, Q4_K_M; no runtime rebuild or profiler attached. - Serving profile: RK8V4 KV, 160,000-token configured context and shared KV capacity, resident GPU vision, prefill chunk 1,024; two configured lanes, one active request.
- Requests: identical prompts and sampling settings between arms; one 256-token warm-up followed by the three measured requests per arm. Actual input lengths: 275 / 328 / 308 tokens. All six measured outputs reached 4,096 tokens; reasoning disabled.
- Sampling: temperature 1.0, top-p 0.95, top-k 20, min-p 0, no presence/frequency penalty, seed 42. N-gram drafting disabled.
- Cache: production prefix cache enabled, 8,192 MiB host budget; zero cached input tokens observed on all measured requests. TTFT is measured with a warm model.
- Rates: native decode uses
(generated tokens β 1) / native decode time. Pooled rates use summed token intervals divided by summed decode time; MTP acceptance uses total accepted drafts divided by total proposed drafts. Client end-to-end throughput includes prefill and request overhead.
Each workload was measured once per arm, with the non-speculative server first and the MTP3 server second. The Python workload includes explanation as well as code. The 160K setting is allocated context capacity, not the input length of this benchmark. These serving measurements are separate from the BF16-KV numerical comparison above.
DFlash2 performance β common 64K profile
DFlash2 was measured against fresh non-speculative and MTP3 runs at a common 64,000-token context / shared KV capacity, retaining resident GPU vision and device embeddings. Both speculative modes use three drafts per round.
| Workload | No speculation | MTP3 | DFlash2 / 3 | DFlash2 vs baseline | DFlash2 vs MTP3 | DFlash2 acceptance |
|---|---|---|---|---|---|---|
| English technical prose | 48.41 tok/s | 87.79 tok/s | 102.38 tok/s | 2.11Γ | 1.17Γ | 58.30% |
| French prose | 48.56 tok/s | 82.97 tok/s | 93.15 tok/s | 1.92Γ | 1.12Γ | 50.40% |
| Python code + explanation | 48.56 tok/s | 94.44 tok/s | 112.51 tok/s | 2.32Γ | 1.19Γ | 68.00% |
| Pooled | 48.51 tok/s | 88.15 tok/s | 102.07 tok/s | 2.10Γ | 1.16Γ | 58.34% |
Across the two prose workloads, DFlash2 reaches 97.55 tok/s, compared with 85.31 tok/s for MTP3 and 48.49 tok/s without speculation. On the code-oriented workload, it reaches 112.51 tok/s with 68.00% draft acceptance.
Comparison setup
Same RTX 3090, production NInfer 39a10890, sampling, prompts and rate definitions as above. All three modes use the Q4_K_M + DFlash2 container; only the components required by each backend are loaded. MTP and DFlash2 are alternatives, not chained. DFlash2 uses its native selector policy, with automatic head selection resolving to the full head.
One 256-token warm-up and three measured requests per mode; all nine measured outputs reached 4,096 tokens, with no reasoning tokens or measured prefix-cache hits. Actual prompts contain 275 / 328 / 308 tokens. RK8V4 KV, C2 configured with one active request, prefill chunk 1,024, resident GPU vision and device embeddings remain fixed. Fixed execution order: DFlash2, no speculation, MTP3; one measured pass per workload and mode.
64K was the first admitted capacity on the tested 160K / 128K / 96K / 64K ladder for DFlash2 in this configuration. All three modes were therefore freshly measured at 64K. The earlier 160K no-speculation/MTP3 results remain a separate benchmark, not the baseline for this table.
Provenance
- Base model: NousResearch/Arete-27B.
- Text quantization sources and BF16 vision projector: bartowski/NousResearch_Arete-27B-GGUF.
- Vision conversion: the source BF16 projector is encoded as native NInfer Q8.
- MTP donor: Qwen3.8-27B UD-Q4_K_XL GGUF, with Q6_K / Q8_0 matrices.
- Optional DFlash2 donor: z-lab/Qwen3.8-27B-DFlash2, revision
50307d4c4cde6860d4eee73e2547cd786fe8e8a4.
Artifact conversion was checked against the source payloads with independent audits. File checksums are provided in SHA256SUMS.
License
The base Arete model is released under Apache-2.0. See the included license and the linked upstream repositories for source attribution and auxiliary components.
- Downloads last month
- 483