FP8_DYNAMIC quant of Nanbeige4.3-33B (llm-compressor), with fidelity + throughput card
324f2c1 verified |
Download README.md from GetCortex27/Nanbeige4.3-33B-FP8: direct link, hf CLI and curl.
- Browser
- Download file 4.12 kB
-
https://huggingface.co/GetCortex27/Nanbeige4.3-33B-FP8/resolve/main/README.md
- Command line
-
hf download hf://GetCortex27/Nanbeige4.3-33B-FP8/README.md
-
curl -L -o README.md https://huggingface.co/GetCortex27/Nanbeige4.3-33B-FP8/resolve/main/README.md
4.12 kB
| license: apache-2.0 | |
| base_model: Nanbeige/Nanbeige4.3-33B | |
| base_model_relation: quantized | |
| tags: | |
| - fp8 | |
| - compressed-tensors | |
| - llm-compressor | |
| - agentic | |
| - tool-calling | |
| - sglang | |
| - vllm | |
| language: | |
| - en | |
| - zh | |
| pipeline_tag: text-generation | |
| # Nanbeige4.3-33B — FP8 (W8A8, dynamic) by GetCortex27 | |
| An FP8 quantisation of [Nanbeige/Nanbeige4.3-33B](https://huggingface.co/Nanbeige/Nanbeige4.3-33B) (Apache 2.0, 33B dense, 512K native context, agentic-coding model, thinking on by default) produced with [llm-compressor](https://github.com/vllm-project/llm-compressor) and published with our own fidelity and throughput measurements. We serve this checkpoint in production; the numbers below are what we measured on it, not the base model's self-reported benchmarks. | |
| ## Recipe | |
| ```python | |
| QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]) | |
| ``` | |
| - Weights: FP8 E4M3, per-channel scales. Activations: FP8, dynamic per-token. No calibration data (data-free). | |
| - llm-compressor 0.15.0, transformers 5.21.0. Format: `compressed-tensors` — loads natively in vLLM and SGLang. | |
| - Size: 34 GB safetensors (from 66 GB BF16). Fits one 96 GB GPU with 516K tokens of fp8 KV cache at 32K context. | |
| - `use_cache` is set to `true` in `config.json` and `generation_config.json` (the upstream repo ships `false`, which makes plain `transformers.generate` recompute the prefix every step). | |
| ## Fidelity vs BF16 | |
| Teacher-forced next-token distributions on 16 × 512-token chunks (6,000 scored positions) of a fixed mixed corpus — Python, English prose, a claims-adjudication JSON record, and a chat-templated agentic turn containing an XML tool call. Reference = BF16 top-64 log-probs; KL computed on that support, renormalised. Script: `kld_eval.py` in [our repo](https://github.com/GetCortex27) (to be published). | |
| | Metric | Value | | |
| |---|---| | |
| | Mean KL(BF16 ‖ FP8), top-64 | 0.0139 | | |
| | p95 KL | 0.0505 | | |
| | Top-1 agreement | 96.6% | | |
| | Mean \|Δ log p\| of the actual next token | 0.092 nats | | |
| The base model has been reported as unusually sensitive to 4-bit quantisation (plain Q4_K_M ≈ 90% top-1 agreement). At 8-bit the drift is small; we did not find a need for calibration-based FP8. | |
| Caveat: the comparison ran the FP8 weights dequantised in transformers, so it measures weight error. The serving-path smoke tests below ran through SGLang's FP8 kernels. | |
| ## Throughput (SGLang 0.5.23, one RTX PRO 6000 Blackwell 96 GB, no speculative decoding, thinking off) | |
| Launch: `python -m sglang.launch_server --model-path GetCortex27/Nanbeige4.3-33B-FP8 --attention-backend flashinfer --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 2048 --max-running-requests 96 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --context-length 32768` | |
| | Concurrency | Aggregate out tok/s | Per-stream tok/s | TTFT p95 | | |
| |---|---|---|---| | |
| | 1 | 29 | 29 | 0.07 s | | |
| | 4 | 104 | 26 | 0.11 s | | |
| | 16 | 355 | 22 | 0.16 s | | |
| | 32 | 607 | 19 | 0.11 s | | |
| | 64 | 1,178 | 18 | 0.21 s | | |
| | 96 | 1,669 | 17 | 0.19 s | | |
| ~137-token prompts, ~165-token outputs, 48–192 requests per level, zero errors in 448 requests. BF16 on the same card: 22 tok/s single-stream, 785 aggregate at 64 concurrent. | |
| ## Serving notes | |
| - Reasoning parser `qwen3`, tool-call parser `qwen3_coder` (the base model recommends an XML tool-call format; this parser handles it). With thinking on and tools present, a weather-tool prompt returned a structured `tool_calls` entry with the reasoning separated into `reasoning_content` and counted in `usage`. | |
| - Thinking is on by default via the chat template; disable per request with `chat_template_kwargs: {"enable_thinking": false}`. | |
| - Recommended sampling (from the base model card): temperature 1.0 / top_p 0.95 for agentic use; 0.6 / 0.95 for chat. | |
| - vLLM: `vllm serve GetCortex27/Nanbeige4.3-33B-FP8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder`. | |
| ## Licence | |
| Apache 2.0, inherited from the base model. Quantisation and measurements by GetCortex27 (Minimorsel Enterprises Pvt. Ltd.). No fine-tuning; weights are a numerical transform of the originals. | |