axteam/Nanbeige4.3-33B-NVFP4

Nanbeige4.3-33B · NVFP4 (W4A4, FP4)

4-bit quantization of Nanbeige/Nanbeige4.3-33B for vLLM serving. Quantized by axteam.

Base model	Nanbeige/Nanbeige4.3-33B
Method	NVFP4 (W4A4, FP4)
Format	compressed-tensors nvfp4-pack-quantized
Weights / activations	W4A4 — FP4 (E2M1) weights group 16 + FP8 scale, FP4 activations
Architecture	Dense (Llama architecture) · 52 layers · text-only
Size on disk	~20.6 GB
Calibration	256 samples
License	apache-2.0

Quantization details

• Scheme — W4A4 — FP4 (E2M1) weights group 16 + FP8 scale, FP4 activations, stored as compressed-tensors nvfp4-pack-quantized.
• Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized.
• Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.

Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts).

Metric	This model	BF16 original
Perplexity (lower = closer)	1.3594	1.2232 (+11.1%)
Top-1 next-token agreement	90.2%	99.1%

⚠️ Note: noticeable 4-bit degradation; prefer 8-bit (FP8 / AWQ INT8).

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/Nanbeige4.3-33B-NVFP4 --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/Nanbeige4.3-33B-NVFP4",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/Nanbeige4.3-33B-NVFP4", max_model_len=32768)
params = SamplingParams(max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/Nanbeige4.3-33B-NVFP4 --kv-cache-dtype fp8

License

Quantized derivative of Nanbeige/Nanbeige4.3-33B (apache-2.0).

---

한국어 요약

Nanbeige/Nanbeige4.3-33B 를 NVFP4 (W4A4, FP4) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.

• 방식 — 가중치·활성화 FP4(W4A4). Blackwell FP4 텐서코어. 형식 compressed-tensors nvfp4-pack-quantized.
• BF16 유지 — 임베딩·LM 헤드·모든 norm. attention·MLP projection 만 양자화.
• PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
• 품질(실측) — BF16 대비 perplexity 1.3594(원본 1.2232, +11.1%), top-1 90.2%(원본 99.1%).
• ⚠️ 4비트 열화 큼 — 8비트 권장.
• 실행 — vllm serve axteam/Nanbeige4.3-33B-NVFP4 --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/Nanbeige4.3-33B-NVFP4").chat(...) (위 예시). thinking 끄기: chat_template_kwargs={"enable_thinking": False}.
• KV 캐시 절감(런타임) — --kv-cache-dtype fp8.