Instructions to use DemoAuguste/Qwen3.8-27B-Quant with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use DemoAuguste/Qwen3.8-27B-Quant with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="DemoAuguste/Qwen3.8-27B-Quant") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("DemoAuguste/Qwen3.8-27B-Quant") model = AutoModelForMultimodalLM.from_pretrained("DemoAuguste/Qwen3.8-27B-Quant", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use DemoAuguste/Qwen3.8-27B-Quant with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "DemoAuguste/Qwen3.8-27B-Quant" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "DemoAuguste/Qwen3.8-27B-Quant", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/DemoAuguste/Qwen3.8-27B-Quant
- SGLang
How to use DemoAuguste/Qwen3.8-27B-Quant with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "DemoAuguste/Qwen3.8-27B-Quant" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "DemoAuguste/Qwen3.8-27B-Quant", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "DemoAuguste/Qwen3.8-27B-Quant" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "DemoAuguste/Qwen3.8-27B-Quant", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use DemoAuguste/Qwen3.8-27B-Quant with Docker Model Runner:
docker model run hf.co/DemoAuguste/Qwen3.8-27B-Quant
Qwen3.8-27B-Quant (SLQ b3.6 mixed precision)
Mixed-precision quantized build of Qwen/Qwen3.8-27B produced with the SLQ method ported to Ascend NPU, then deployed and validated through the vLLM-Ascend OpenAI-compatible HTTP API.
- Average bitwidth: 3.600 bpp (parameter-weighted, including the g128 scale/zero overhead)
- Storage-equivalent packing estimate: 17.26 GB (3.17ร vs BF16) once the weights are INT-packed; the files in this repo are fake-quant BF16, so their on-disk size is ~51 GB (same as BF16)
- Targeted deployment: Ascend 910B (A2), vLLM-Ascend
v0.30.0, tensor-parallel = 2
Quantization recipe
| item | value |
|---|---|
| Method | SLQ official pipeline (stage โ ) + NPU port: per-layer GPTQ with Hessian second-order compensation |
| Weight format | asymmetric INT, group size 128, scale bf16 + zero int4 accounting (+0.156 bpp) |
| Bitwidth options | {2, 4, 6, 8} produced from a single Hessian per layer |
| Sequential propagation | calibration_bitwidth = 4 |
| Calibration | FineWeb-Edu sample-10BT, 65,536 tokens (8 ร 8192) |
| KK layer groups | 400 groups (qkv / gate+up fused); ILP budget scan over Shapley sensitivity |
| Budget | b3.6 โ 3.600 bpp (predicted top-10 KL 0.0473) |
Bitwidth distribution of the b3.6 allocation (496 linear layers, 24.35B quantized parameters):
| bits | layers | share | parameters |
|---|---|---|---|
| 2-bit | 148 | 29.8% | 6.78B (27.9%) |
| 4-bit | 312 | 62.9% | 17.56B (72.1%) |
| 6-bit | 23 | 4.6% | 5.65M (0.02%) |
| 8-bit | 13 | 2.6% | 3.19M (0.01%) |
Embedding / lm_head / norms / vision tower stay BF16 (3.01B parameters, ~11%).
What "fake-quant" means here: every quantized weight was rounded to its INT grid and then dequantized back to BF16 before writing the checkpoint. Transformers and vLLM load it as an ordinary BF16 model, so accuracy behaviour matches the quantized model while the file size does not shrink. Real memory/storage gains need an INT packing/loading path (
humming_onlineor equivalent).
Validation
Measured on 8ร Ascend 910B3, TP=2, vLLM-Ascend v0.30.0 (details in the repository's
experiment/slq-official/DEPLOYMENT-VALIDATION.md):
| check | result |
|---|---|
| Quantization fingerprint | every 128-column group within 2^bits levels (4/16/54/101 vs 4/16/64/256) โ PASS |
/health |
200 |
/v1/models |
qwen3.8-27b-slq-b36, max_model_len 4096 |
/v1/chat/completions |
coherent zh/en answers; 128 tokens โ 4.3 s once warm |
/v1/completions |
The capital of China is โ "Beijing, which is also the largest city..." |
| Streaming SSE | works |
| Memory | 25.61 GB weights per card, 24.11 GiB KV cache (320,605 tokens) |
| Decode throughput | ~28โ30 tok/s (TP=2, warm) |
Cold-start note: the first few requests after the server starts trigger NPU graph compilation and can take minutes; send a warm-up request before benchmarking.
Files
config.json, generation_config.json, chat_template.jinja, tokenizer files
preprocessor_config.json, video_preprocessor_config.json # VLM preprocessing
model-00001-of-00002.safetensors # 47 GB
model-00002-of-00002.safetensors # 4.6 GB
model.safetensors.index.json
quantization_config.txt # SLQ allocation record
quantization_info.json # machine-readable summary
validation/ # fingerprint + HTTP logs
Usage
vLLM-Ascend (recommended, matches the validation environment)
vllm serve <this-repo> \
--served-model-name qwen3.8-27b-slq-b36 \
--tensor-parallel-size 2 \
--max-model-len 4096 \
--gpu-memory-utilization 0.85 \
--trust-remote-code
curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3.8-27b-slq-b36",
"messages": [{"role": "user", "content": "็จไธๅฅ่ฏ่ฏดๆๆททๅ็ฒพๅบฆ้ๅใ"}],
"max_tokens": 128}'
On Ascend the container needs non-contiguous device mapping and explicit HCCL interfaces;
see the repository's experiment/slq-official/deploy/serve_27b.sh.
Transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
# The architecture recorded in config.json is Qwen3_5ForConditionalGeneration.
Limitations
- Fake-quant weights: no size/memory reduction on disk or in NPU HBM yet.
- The 15
mtp.*(multi-token-prediction head) tensors are not included, so MTP-based speculative decoding is unavailable; plain autoregressive serving is unaffected. - Calibration used 65,536 tokens instead of the paper recipe's 262,144 (a time-budget trade-off during a full rebuild), so results may differ slightly from the reported 3.600 bpp run.
- Accuracy numbers reported for the original run (6 task ร 500-sample zero-shot average recovery 100.0%) refer to the same allocation but a checkpoint built with the full calibration set.
Provenance
- Base model: Qwen/Qwen3.8-27B
- Method: SLQ (IST-DASLab), NPU port and pipeline:
DemoAuguste/vllm-ascendโexperiment/slq-official/ - Quantized and validated: 2026-11-18/19 on Ascend 910B3 (A2), CANN 9.2.1
- Downloads last month
- 38
Model tree for DemoAuguste/Qwen3.8-27B-Quant
Base model
Qwen/Qwen3.8-27B