Qwen3.8-27B-Quant (SLQ b3.6 mixed precision)

Mixed-precision quantized build of Qwen/Qwen3.8-27B produced with the SLQ method ported to Ascend NPU, then deployed and validated through the vLLM-Ascend OpenAI-compatible HTTP API.

  • Average bitwidth: 3.600 bpp (parameter-weighted, including the g128 scale/zero overhead)
  • Storage-equivalent packing estimate: 17.26 GB (3.17ร— vs BF16) once the weights are INT-packed; the files in this repo are fake-quant BF16, so their on-disk size is ~51 GB (same as BF16)
  • Targeted deployment: Ascend 910B (A2), vLLM-Ascend v0.30.0, tensor-parallel = 2

Quantization recipe

item value
Method SLQ official pipeline (stage โ‘ ) + NPU port: per-layer GPTQ with Hessian second-order compensation
Weight format asymmetric INT, group size 128, scale bf16 + zero int4 accounting (+0.156 bpp)
Bitwidth options {2, 4, 6, 8} produced from a single Hessian per layer
Sequential propagation calibration_bitwidth = 4
Calibration FineWeb-Edu sample-10BT, 65,536 tokens (8 ร— 8192)
KK layer groups 400 groups (qkv / gate+up fused); ILP budget scan over Shapley sensitivity
Budget b3.6 โ†’ 3.600 bpp (predicted top-10 KL 0.0473)

Bitwidth distribution of the b3.6 allocation (496 linear layers, 24.35B quantized parameters):

bits layers share parameters
2-bit 148 29.8% 6.78B (27.9%)
4-bit 312 62.9% 17.56B (72.1%)
6-bit 23 4.6% 5.65M (0.02%)
8-bit 13 2.6% 3.19M (0.01%)

Embedding / lm_head / norms / vision tower stay BF16 (3.01B parameters, ~11%).

What "fake-quant" means here: every quantized weight was rounded to its INT grid and then dequantized back to BF16 before writing the checkpoint. Transformers and vLLM load it as an ordinary BF16 model, so accuracy behaviour matches the quantized model while the file size does not shrink. Real memory/storage gains need an INT packing/loading path (humming_online or equivalent).

Validation

Measured on 8ร— Ascend 910B3, TP=2, vLLM-Ascend v0.30.0 (details in the repository's experiment/slq-official/DEPLOYMENT-VALIDATION.md):

check result
Quantization fingerprint every 128-column group within 2^bits levels (4/16/54/101 vs 4/16/64/256) โ†’ PASS
/health 200
/v1/models qwen3.8-27b-slq-b36, max_model_len 4096
/v1/chat/completions coherent zh/en answers; 128 tokens โ‰ˆ 4.3 s once warm
/v1/completions The capital of China is โ†’ "Beijing, which is also the largest city..."
Streaming SSE works
Memory 25.61 GB weights per card, 24.11 GiB KV cache (320,605 tokens)
Decode throughput ~28โ€“30 tok/s (TP=2, warm)

Cold-start note: the first few requests after the server starts trigger NPU graph compilation and can take minutes; send a warm-up request before benchmarking.

Files

config.json, generation_config.json, chat_template.jinja, tokenizer files
preprocessor_config.json, video_preprocessor_config.json   # VLM preprocessing
model-00001-of-00002.safetensors                            # 47 GB
model-00002-of-00002.safetensors                            # 4.6 GB
model.safetensors.index.json
quantization_config.txt                                     # SLQ allocation record
quantization_info.json                                      # machine-readable summary
validation/                                                 # fingerprint + HTTP logs

Usage

vLLM-Ascend (recommended, matches the validation environment)

vllm serve <this-repo> \
  --served-model-name qwen3.8-27b-slq-b36 \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --trust-remote-code
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen3.8-27b-slq-b36",
  "messages": [{"role": "user", "content": "็”จไธ€ๅฅ่ฏ่ฏดๆ˜Žๆททๅˆ็ฒพๅบฆ้‡ๅŒ–ใ€‚"}],
  "max_tokens": 128}'

On Ascend the container needs non-contiguous device mapping and explicit HCCL interfaces; see the repository's experiment/slq-official/deploy/serve_27b.sh.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
# The architecture recorded in config.json is Qwen3_5ForConditionalGeneration.

Limitations

  • Fake-quant weights: no size/memory reduction on disk or in NPU HBM yet.
  • The 15 mtp.* (multi-token-prediction head) tensors are not included, so MTP-based speculative decoding is unavailable; plain autoregressive serving is unaffected.
  • Calibration used 65,536 tokens instead of the paper recipe's 262,144 (a time-budget trade-off during a full rebuild), so results may differ slightly from the reported 3.600 bpp run.
  • Accuracy numbers reported for the original run (6 task ร— 500-sample zero-shot average recovery 100.0%) refer to the same allocation but a checkpoint built with the full calibration set.

Provenance

  • Base model: Qwen/Qwen3.8-27B
  • Method: SLQ (IST-DASLab), NPU port and pipeline: DemoAuguste/vllm-ascend โ†’ experiment/slq-official/
  • Quantized and validated: 2026-11-18/19 on Ascend 910B3 (A2), CANN 9.2.1
Downloads last month
38
Safetensors
Model size
27B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for DemoAuguste/Qwen3.8-27B-Quant

Base model

Qwen/Qwen3.8-27B
Finetuned
(973)
this model