8-bit weights and activations: 97.9% top-1 agreement with BF16 at 54% of its size, on one card.
FP8 build of Nanbeige/Nanbeige4.3-33B at 35.53 GB, against 66.24 GB for the BF16 release.
Every linear layer in the 52 decoder blocks stores FP8 E4M3 weights with one scale per output channel; activations are quantized to FP8 per token at runtime. Embeddings, the LM head and the norms stay BF16.
Drift is the extra negative log-likelihood per token a build assigns to 400 reasoning traces that BF16 wrote on the same card; top-1 agreement is how often its most likely next token is BF16's. The pass is deterministic.
Why this build
- 🎯 Drift +5.95×10⁻³ nats per token from BF16, top-1 agreement 97.9%. The NVFP4 build sits at +64.3 and 93.7%.
- 📏 Knowledge 89.3, tool calls 80.4, abstain 63.3; BF16 scores 89.7, 80.8 and 63.3. 99.1% of answers finish inside the budget.
- ⚡ 35.4 tok/s single-stream and 225 at concurrency 32 on one card, against 20.1 and 129 for BF16 on the same card.
- 🗜️ 35.53 GB, against 66.24 GB in BF16. At a 32,768-token window the KV pool is 262,448 tokens, against 113,648 for BF16.
Serve it
hf download primitive-ai/Nanbeige4.3-33B-FP8 --local-dir ./Nanbeige4.3-33B-FP8
docker run --gpus '"device=0"' --ipc=host -p 8000:8000 -v $PWD:/models \
vllm/vllm-openai:v0.32.0 \
--model /models/Nanbeige4.3-33B-FP8 \
--max-model-len 131072 --gpu-memory-utilization 0.92 \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
Checked with exactly this command on an idle card: boot 171 s, 33.44 GiB of weights, a 251,504-token KV pool at the 131,072-token window. What is 17*23? comes back as 391 with the reasoning split off by the parser, and as 391 with chat_template_kwargs: {"enable_thinking": false}; a weather question comes back as the parsed call get_weather {"city": "Lisbon"}. The model's window is 524,288 tokens; raise --max-model-len as far as the KV pool allows.
Measured
One RTX PRO 6000 Blackwell, 96 GB, no tensor parallelism, vLLM 0.32.0, BF16 KV cache. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell. KV pool is at a 32,768-token window and --gpu-memory-utilization 0.92.
| build | format | size | drift (×10⁻³ nats/token) | top-1 agreement | knowledge | call | abstain | finished | tok/s @1 | tok/s @32 | KV pool (tokens) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Nanbeige4.3-33B (BF16 release) | BF16 | 66.24 GB | 0 (reference) | 100% | 89.7 | 80.8 | 63.3 | 99.2% | 20.1 | 129 | 113,648 |
| this repo | FP8 W8A8, every Linear | 35.53 GB | +5.95 | 97.9% | 89.3 | 80.4 | 63.3 | 99.1% | 35.4 | 225 | 262,448 |
| Mixed NVFP4 / FP8 | NVFP4 W4A4, FP8 W8A8 in 216 modules | 25.30 GB | +38.89 | 95.1% | 88.9 | 79.0 | 63.3 | 99.1% | 48.0 | 308 | 311,488 |
| NVFP4 | NVFP4 W4A4, every Linear | 22.07 GB | +64.28 | 93.7% | 89.0 | 78.8 | 64.2 | 99.1% | 53.8 | 336 | 326,144 |
Drift and agreement come from 400 BF16 traces of the knowledge suite (215,950 completion tokens); the standard error over traces is 0.3 for this build. Tool calling is 3 runs per build; knowledge is 2 runs per quantized build and 1 for BF16.
What this table does not cover:
- Context past 32K. The model's window is 524,288 tokens; we did not measure the quantized layers there.
- Other GPUs. Only the RTX PRO 6000 Blackwell (sm_120) was tested.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above, the
same 1,370 items, auto-scored, no LLM judge.
| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| Laguna-XS-2.1 | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
| Nemotron-3.5-Lightning-30B-A3B | 30 B MoE+Mamba | 19.2 GiB | 87.1 | 87.9 | 85.4 | 70.5 | 97.9% | 1429 |
| Ornith-1.5-35B-A3B | 35 B MoE | 22.6 GiB | 88.7 | 91.7 | 74.4 | 60.0 | 99.3% | 760 |
| Muse-Glimmer-30B | 30 B MoE | 20.4 GiB | 86.6 | 88.8 | 78.6 | 54.5 | 99.7% | 800 |
| Qwen3.8-27B | 27 B dense | 20.7 GiB | 88.8 | 90.4 | 85.5 | 54.5 | 99.7% | 651 |
| Granite-4.2-30B | 30 B dense | 18.1 GB | 85.5 | 86.2 | 85.8 | 60.8 | 98.5% | 1502 |
| Nex-N2.5-mini NVFP4 | 35 B MoE, 3 B active | 23.91 GB | 88.5 | 90.5 | 81.9 | 57.5 | 99.3% | 524 |
| Nex-N2.5-mini mixed | 35 B MoE, 3 B active | 26.04 GB | 88.5 | 90.6 | 80.9 | 58.7 | 99.2% | 504 |
| Nex-N2.5-mini FP8 | 35 B MoE, 3 B active | 38.13 GB | 88.7 | 90.9 | 79.4 | 60.0 | 99.5% | 545 |
| K2-Horizon-MoVA-36B-A4B NVFP4 | 37 B MoE+MoVA, 4 B active | 36.7 GB | 84.1 | 86.5 | 71.8 | 60.4 | 95.6% | 1234 |
| K2-Horizon-MoVA-36B-A4B mixed | 37 B MoE+MoVA, 4 B active | 44.5 GB | 84.9 | 87.3 | 73.5 | 58.9 | 96.3% | 1118 |
| Nemotron-3.5-Bolt-74B-A9B mixed | 75 B MoE+Mamba, 9 B active | 46.70 GB | 89.3 | 90.6 | 87.9 | 58.3 | 98.5% | 1173 |
| Laguna-S-2.1 | 110 B MoE | 64.0 GiB | 84.3 | 87.1 | 64.6 | 81.0 | 97.3% | 995 |
| Qwen3.8-Flash-Next | 180 B MoE, 6 B active | 183.7 GB | 90.3 | 92.2 | 84.8 | 56.7 | 99.5% | 686 |
| MiMo-V2.6-Flash-RL NVFP4 | 309 B MoE, 15 B active | 182.40 GB | 89.3 | 91.1 | 83.0 | 61.0 | 99.0% | 664 |
| Kolibri-1 FP8 | 78 B MoE, 3.5 B active | 78.83 GB | 85.0 | 86.1 | 86.1 | 48.0 | 97.0% | 1824 |
| Kolibri-1 NVFP4 | 78 B MoE, 3.5 B active | 47.70 GB | 84.9 | 86.2 | 84.9 | 48.0 | 96.4% | 1947 |
| Kolibri-1 mixed | 78 B MoE, 3.5 B active | 45.81 GB | 83.9 | 85.0 | 85.5 | 47.5 | 96.6% | 1848 |
| LFM2.5-24B-A2B NVFP4A16 | 24 B MoE, 2.45 B active | 14.95 GB | 83.9 | 85.4 | 82.4 | 44.5 | 98.8% | 1603 |
| MiniMax-M3.1-Flash (release, 4 cards) | 427 B MoE, 23 B active | 250.12 GB | 82.9 | 85.0 | 81.7 | 25.0 | 99.2% | 513 |
| MiniMax-M3.1-Flash-pruned-272B | 272 B MoE, 23 B active | 162.90 GB | 82.5 | 84.4 | 80.4 | 32.5 | 98.8% | 707 |
| Nanbeige4.3-33B (BF16 release) | 33 B dense | 66.24 GB | 87.9 | 89.7 | 80.8 | 63.3 | 99.2% | 846 |
| Nanbeige4.3-33B FP8 (this repo) | 33 B dense | 35.53 GB | 87.5 | 89.3 | 80.4 | 63.3 | 99.1% | 871 |
| Nanbeige4.3-33B mixed | 33 B dense | 25.30 GB | 87.0 | 88.9 | 79.0 | 63.3 | 99.1% | 905 |
| Nanbeige4.3-33B NVFP4 | 33 B dense | 22.07 GB | 87.1 | 89.0 | 78.8 | 64.2 | 99.1% | 940 |
overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. Read it with finished: overall scores an answer that overran the token budget as
wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in
overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.
What's in it
Nanbeige4.3-33B is a dense Llama-architecture model: 33.12 B parameters, 52 layers, hidden size 7168, 56 query and 8 KV heads, MLP width 22,016.
| params | share | |
|---|---|---|
| MLP gate + up + down, 52 layers | 24.62 B | 74.3% |
| attention q + k + v, 52 layers | 3.44 B | 10.4% |
| attention o, 52 layers | 2.67 B | 8.1% |
| embeddings and LM head (untied) | 2.39 B | 7.2% |
| modules | count | format |
|---|---|---|
| every decoder Linear (q, k, v, o, gate, up, down) | 364 | FP8 E4M3, one scale per output channel; activations FP8, one scale per token |
| embeddings, LM head, norms | all | BF16, byte-identical to the release |
The checkpoint is compressed-tensors and runs on vLLM's stock kernels. q/k/v and gate/up share one format per layer because vLLM fuses them.
Licence
Apache-2.0, the base model's licence. This is a modified version of Nanbeige/Nanbeige4.3-33B: the decoder's linear layers were quantized and config.json carries a quantization_config. Everything else is unchanged.
primitive ·
more models ·
inference economics for production LLM systems
- Downloads last month
- 499