---
license: apache-2.0
base_model: Nanbeige/Nanbeige4.3-33B
base_model_relation: quantized
tags:
  - fp8
  - compressed-tensors
  - llm-compressor
  - agentic
  - tool-calling
  - sglang
  - vllm
language:
  - en
  - zh
pipeline_tag: text-generation
---

# Nanbeige4.3-33B — FP8 (W8A8, dynamic) by GetCortex27

An FP8 quantisation of [Nanbeige/Nanbeige4.3-33B](https://huggingface.co/Nanbeige/Nanbeige4.3-33B) (Apache 2.0, 33B dense, 512K native context, agentic-coding model, thinking on by default) produced with [llm-compressor](https://github.com/vllm-project/llm-compressor) and published with our own fidelity and throughput measurements. We serve this checkpoint in production; the numbers below are what we measured on it, not the base model's self-reported benchmarks.

## Recipe

```python
QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])
```
- Weights: FP8 E4M3, per-channel scales. Activations: FP8, dynamic per-token. No calibration data (data-free).
- llm-compressor 0.15.0, transformers 5.21.0. Format: `compressed-tensors` — loads natively in vLLM and SGLang.
- Size: 34 GB safetensors (from 66 GB BF16). Fits one 96 GB GPU with 516K tokens of fp8 KV cache at 32K context.
- `use_cache` is set to `true` in `config.json` and `generation_config.json` (the upstream repo ships `false`, which makes plain `transformers.generate` recompute the prefix every step).

## Fidelity vs BF16

Teacher-forced next-token distributions on 16 × 512-token chunks (6,000 scored positions) of a fixed mixed corpus — Python, English prose, a claims-adjudication JSON record, and a chat-templated agentic turn containing an XML tool call. Reference = BF16 top-64 log-probs; KL computed on that support, renormalised. Script: `kld_eval.py` in [our repo](https://github.com/GetCortex27) (to be published).

| Metric | Value |
|---|---|
| Mean KL(BF16 ‖ FP8), top-64 | 0.0139 |
| p95 KL | 0.0505 |
| Top-1 agreement | 96.6% |
| Mean \|Δ log p\| of the actual next token | 0.092 nats |

The base model has been reported as unusually sensitive to 4-bit quantisation (plain Q4_K_M ≈ 90% top-1 agreement). At 8-bit the drift is small; we did not find a need for calibration-based FP8.

Caveat: the comparison ran the FP8 weights dequantised in transformers, so it measures weight error. The serving-path smoke tests below ran through SGLang's FP8 kernels.

## Throughput (SGLang 0.5.23, one RTX PRO 6000 Blackwell 96 GB, no speculative decoding, thinking off)

Launch: `python -m sglang.launch_server --model-path GetCortex27/Nanbeige4.3-33B-FP8 --attention-backend flashinfer --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 2048 --max-running-requests 96 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 32768`

| Concurrency | Aggregate out tok/s | Per-stream tok/s | TTFT p95 |
|---|---|---|---|
| 1 | 29 | 29 | 0.07 s |
| 4 | 104 | 26 | 0.11 s |
| 16 | 355 | 22 | 0.16 s |
| 32 | 607 | 19 | 0.11 s |
| 64 | 1,178 | 18 | 0.21 s |
| 96 | 1,669 | 17 | 0.19 s |

~137-token prompts, ~165-token outputs, 48–192 requests per level, zero errors in 448 requests. BF16 on the same card: 22 tok/s single-stream, 785 aggregate at 64 concurrent.

## Serving notes

- Reasoning parser `qwen3`, tool-call parser `qwen3_coder` (the base model recommends an XML tool-call format; this parser handles it). With thinking on and tools present, a weather-tool prompt returned a structured `tool_calls` entry with the reasoning separated into `reasoning_content` and counted in `usage`.
- Thinking is on by default via the chat template; disable per request with `chat_template_kwargs: {"enable_thinking": false}`.
- Recommended sampling (from the base model card): temperature 1.0 / top_p 0.95 for agentic use; 0.6 / 0.95 for chat.
- vLLM: `vllm serve GetCortex27/Nanbeige4.3-33B-FP8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder`.

## Licence

Apache 2.0, inherited from the base model. Quantisation and measurements by GetCortex27 (Minimorsel Enterprises Pvt. Ltd.). No fine-tuning; weights are a numerical transform of the originals.