axteam/Nanbeige4.3-33B-AWQ-8bit

Nanbeige4.3-33B · AWQ INT8 (W8A16)

8-bit quantization of Nanbeige/Nanbeige4.3-33B for vLLM serving. Quantized by axteam.

Base model	Nanbeige/Nanbeige4.3-33B
Method	AWQ INT8 (W8A16)
Format	compressed-tensors pack-quantized
Weights / activations	W8A16 — INT8 sym weights, group 32, BF16 activations
Architecture	Dense (Llama architecture) · 52 layers · text-only
Size on disk	~37.4 GB
Calibration	256 samples
License	apache-2.0

Quantization details

• Scheme — W8A16 — INT8 sym weights, group 32, BF16 activations, stored as compressed-tensors pack-quantized.
• Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts).

Metric	This model	BF16 original
Perplexity (lower = closer)	1.2237	1.2232 (+0.0%)
Top-1 next-token agreement	98.8%	99.1%

Requirements

Runtime	vLLM — the architecture is in vLLM; this build not run here yet.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 37.4 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU.
Transformers (without vLLM)	transformers>=4.56.0 + compressed-tensors>=0.15.0 (same Llama architecture as MiniCPM5; not run here for Nanbeige).
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/Nanbeige4.3-33B-AWQ-8bit --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/Nanbeige4.3-33B-AWQ-8bit",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/Nanbeige4.3-33B-AWQ-8bit", max_model_len=32768)
params = SamplingParams(max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/Nanbeige4.3-33B-AWQ-8bit --kv-cache-dtype fp8

License

Quantized derivative of Nanbeige/Nanbeige4.3-33B (apache-2.0).

---

한국어 요약

Nanbeige/Nanbeige4.3-33B 를 AWQ INT8 (W8A16) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — INT8 대칭 32묶음, 활성화 BF16. 형식 compressed-tensors pack-quantized.
• BF16 유지 — 임베딩·LM 헤드·모든 norm. attention·MLP projection 만 양자화.
• PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
• 품질(실측) — BF16 대비 perplexity 1.2237(원본 1.2232, +0.0%), top-1 98.8%(원본 99.1%).
• 요구사항 — vLLM; GPU Turing(sm75) 이상 — 가중치 전용 INT, 특수 텐서코어 불필요; 가중치 37.4 GB + KV 캐시; Transformers 직접 사용: transformers ≥4.56.0(Nanbeige 는 미실행).
• 실행 — vllm serve axteam/Nanbeige4.3-33B-AWQ-8bit --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/Nanbeige4.3-33B-AWQ-8bit").chat(...) (위 예시). thinking 끄기: chat_template_kwargs={"enable_thinking": False}.
• KV 캐시 절감(런타임) — --kv-cache-dtype fp8.