Instructions to use IteraCompute/Qwen3.8-27B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use IteraCompute/Qwen3.8-27B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="IteraCompute/Qwen3.8-27B-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("IteraCompute/Qwen3.8-27B-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("IteraCompute/Qwen3.8-27B-NVFP4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use IteraCompute/Qwen3.8-27B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IteraCompute/Qwen3.8-27B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IteraCompute/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/IteraCompute/Qwen3.8-27B-NVFP4

SGLang

How to use IteraCompute/Qwen3.8-27B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "IteraCompute/Qwen3.8-27B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IteraCompute/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "IteraCompute/Qwen3.8-27B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IteraCompute/Qwen3.8-27B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use IteraCompute/Qwen3.8-27B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/IteraCompute/Qwen3.8-27B-NVFP4
```

IteraCompute Qwen3.8-27B NVFP4 — Deployment Edition

An open-weight deployment release maintained by IteraCompute for efficient inference on NVIDIA Blackwell GPUs, with our RTX 5090 serving configuration and deployment guide.

IteraCompute's deployment optimization brings together FP8 KV cache, chunked prefill, prefix caching, scheduling, and OpenAI-compatible reasoning/tool-call integration. The serving example below describes this deployment approach. The original checkpoint tensors and model configuration remain unchanged; this edition documents our deployment work rather than a new training or quantization run.

中文：IteraCompute 部署优化版 Qwen3.8-27B NVFP4，提供可下载权重和针对 RTX 5090 调整的推理部署说明。我们围绕显存利用、缓存、请求调度以及思考和工具调用接入进行部署优化。底层权重和模型配置保持原样；上游来源与许可说明保留在文末。

Checkpoint

Item	Details
Model	Qwen3.8-27B
Quantization	NVIDIA Model Optimizer NVFP4; see `config.json` and `hf_quant_config.json` for layer coverage
Weight files	Two Safetensors shards, 17,916,112,584 bytes in total (about 17.92 GB)
Configuration architecture	`Qwen3_5ForConditionalGeneration` / `qwen3_5`, as supplied upstream
Configured maximum context	262,144 tokens, input and output combined
IteraCompute deployment	vLLM 0.31.0, one RTX 5090 32 GB per instance, FP8 KV cache
License	Apache 2.0; retain LICENSE and upstream notices

NVFP4 execution requires compatible Blackwell hardware and engine support. Available context and concurrency depend on engine settings and memory. The checkpoint's context configuration is separate from the limits of any hosted API plan. This release does not add a new quality evaluation or benchmark; figures in the preserved source card belong to its authors and their stated test conditions.

Download and verify

hf download IteraCompute/Qwen3.8-27B-NVFP4 --local-dir ./Qwen3.8-27B-NVFP4
cd Qwen3.8-27B-NVFP4
sha256sum -c SHA256SUMS

For reproducible use, add --revision <full-commit-sha> with a commit from this repository's history. SHA256SUMS covers the checkpoint, serving files, and preserved upstream documentation. release.json records the source commit and confirms comparison with the deployed checkpoint.

Serve with vLLM

The following settings reflect IteraCompute's vLLM 0.31.0 deployment. Install an appropriate CUDA/Blackwell build of vLLM in a separate environment, then run:

vllm serve ./Qwen3.8-27B-NVFP4 \
  --served-model-name qwen3.8-27b-nvfp4 \
  --host 127.0.0.1 --port 8000 \
  --quantization modelopt \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.97 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 2048 \
  --kv-cache-dtype fp8 \
  --attention-backend FLASHINFER \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b-nvfp4","messages":[{"role":"user","content":"Hello!"}],"max_tokens":256,"chat_template_kwargs":{"enable_thinking":false}}'

The example binds to localhost. Engine and driver compatibility must be checked for your environment. Use the included chat template for reasoning and tool calls; further upstream details are in SOURCE_MODEL_CARD.md.

Qwen 4 NVFP4 is next

IteraCompute plans to upload a Qwen 4 NVFP4 version as soon as possible after the official model becomes available and licensing, quantization, and inference validation are complete. Follow our Qwen4-27B-NVFP4 repository for the release. That repository is currently a preparation page with no Qwen 4 weights; its name is a planned target, subject to the official model lineup.

中文：我们后续也会第一时间推进并上传 Qwen 4 的 NVFP4 版本。 将在官方模型可用、确认许可证并完成量化及推理验证后尽快发布。预告仓库已准备好：IteraCompute/Qwen4-27B-NVFP4。目前尚未提供 Qwen 4 权重，最终型号以官方发布为准。

About IteraCompute

IteraCompute provides OpenAI-compatible model inference. Visit our website, API documentation, and current free Qwen API offering for hosted access and current terms.

This is an independent community distribution, unaffiliated with Qwen or NVIDIA. Original model and quantization credit remains with the respective upstream authors.

Source and license

Provenance: the original model is from Qwen, and this NVFP4 quantization is from Gittensor Model Hub. IteraCompute hosts this verified checkpoint and contributes deployment documentation. The weights and serving files are unchanged from the pinned source; we do not claim authorship of the original model or quantization. See release.json for the exact source revision and SOURCE_MODEL_CARD.md for the preserved source documentation.

Downloads last month: 84

Safetensors

Model size

15B params

Tensor type

BF16

Model tree for IteraCompute/Qwen3.8-27B-NVFP4

Base model

Qwen/Qwen3.8-27B

Quantized

(1,689)

this model