bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

FP8 quantization of bottlecapai/ThinkingCap-Qwen3.6-27B — capability of Qwen3.6-27B with 50% less thinking tokens on average, achieved by finetuning Qwen3.6-27B (Qwen Team, 2026) while preserving the original answer quality and style.

➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at bottlecapai/ThinkingCap-Qwen3.6-27B.

About this quantization

Native block-wise FP8 (E4M3): weights are quantized to FP8 with 128×128 block scales (weight_scale_inv) and activations are quantized dynamically per token — quant_method: "fp8", weight_block_size: [128, 128], the same DeepSeek-V3-style format that Qwen/Qwen3.6-27B-FP8 ships. vLLM and SGLang load it natively, and the model's MTP (multi-token-prediction) head is preserved, so self-speculative decoding works (≈3.2 accepted tokens/step). ≈29 GB instead of ≈55 GB bf16 — near-lossless quality at half the memory, with FP8 tensor-core throughput on Hopper / Blackwell (Ada / Ampere serve it via weight-only FP8-Marlin kernels).

Kept in bf16: lm_head, the MTP head, the vision tower, and the Gated-DeltaNet input gates (linear_attn.in_proj_a/b, whose 48-wide output doesn't tile a 128 block).

Usage

# vLLM
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

# SGLang
python -m sglang.launch_server --model-path bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 --trust-remote-code

Speculative decoding (MTP)

The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add --speculative-config when serving (requires vLLM ≥ 0.24.0):

vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.35 drafts per verify step here, for a further ≈2.4× on top of the finetune's token savings (see below).

For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF.

Expected performance

Measured on our internal serving harness on 8× RTX PRO 6000 (Blackwell) with vLLM 0.25.0, one configuration per GPU. Each cell is N=200 questions/dataset × 3 generation seeds over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); acc is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA).

FP8 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding ≈55% faster (36.4 vs 23.4 tok/s, MMLU-Pro standard), and MTP self-speculative decoding (≈3.1–3.3 accepted tokens per verify step) adds a further ≈1.8–2.0× — stacking with the finetune's token savings to ≈6× faster per task than the unquantized base (MMLU-Pro 13.7 s vs 97.3 s; RealWorldQA 6.8 s vs 34.2 s).

All timings are measured per request during the eval. median tokens = median completion length (the finetune's lever); tok/s = per-request steady-state decode rate (prefill excluded via time-to-first-token); task s = measured end-to-end wall-clock per request under the batch-size-16 concurrency; speedup = task s of the base model in standard decoding ÷ task s of the row. Qwen3.6-27B-FP8 (official) is Qwen/Qwen3.6-27B-FP8, the official FP8 quant of the base, shown for comparison.

MMLU-Pro (reasoning)

config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base · standard	0.902 ± 0.019	2186	22.6	97.3	1.00×	—
Qwen3.6-27B base · MTP	0.892 ± 0.031	2117	47.7	45.6	2.13×	3.23
Qwen3.6-27B-FP8 (official) · standard	0.888 ± 0.007	2250	35.5	63.7	1.53×	—
Qwen3.6-27B-FP8 (official) · MTP	0.907 ± 0.044	2174	64.4	34.5	2.82×	3.21
ThinkingCap bf16 · standard	0.890 ± 0.022	963	23.4	41.6	2.34×	—
ThinkingCap bf16 · MTP	0.895 ± 0.012	910	51.1	18.8	5.18×	3.26
ThinkingCap-FP8 · standard	0.908 ± 0.026	1000	36.4	27.9	3.49×	—
ThinkingCap-FP8 · MTP	0.900 ± 0.050	960	70.8	13.7	7.10×	3.26

RealWorldQA (vision)

config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base · standard	0.802 ± 0.019	718	21.7	34.2	1.00×	—
Qwen3.6-27B base · MTP	0.798 ± 0.064	752	42.2	18.7	1.83×	3.09
Qwen3.6-27B-FP8 (official) · standard	0.803 ± 0.014	703	33.3	22.2	1.54×	—
Qwen3.6-27B-FP8 (official) · MTP	0.790 ± 0.054	714	56.8	13.8	2.48×	3.09
ThinkingCap bf16 · standard	0.818 ± 0.047	344	21.5	17.4	1.97×	—
ThinkingCap bf16 · MTP	0.805 ± 0.045	372	42.5	9.9	3.45×	3.09
ThinkingCap-FP8 · standard	0.807 ± 0.026	346	33.3	11.6	2.95×	—
ThinkingCap-FP8 · MTP	0.810 ± 0.045	334	58.6	6.8	5.03×	3.09