Instructions to use Beling86/Qwen3.8-Flash-Next-3.48BPW with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Beling86/Qwen3.8-Flash-Next-3.48BPW with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Beling86/Qwen3.8-Flash-Next-3.48BPW")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Beling86/Qwen3.8-Flash-Next-3.48BPW")
model = AutoModelForCausalLM.from_pretrained("Beling86/Qwen3.8-Flash-Next-3.48BPW", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Beling86/Qwen3.8-Flash-Next-3.48BPW with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Beling86/Qwen3.8-Flash-Next-3.48BPW"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beling86/Qwen3.8-Flash-Next-3.48BPW",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Beling86/Qwen3.8-Flash-Next-3.48BPW

SGLang

How to use Beling86/Qwen3.8-Flash-Next-3.48BPW with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Beling86/Qwen3.8-Flash-Next-3.48BPW" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beling86/Qwen3.8-Flash-Next-3.48BPW",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Beling86/Qwen3.8-Flash-Next-3.48BPW" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beling86/Qwen3.8-Flash-Next-3.48BPW",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Beling86/Qwen3.8-Flash-Next-3.48BPW with Docker Model Runner:
```
docker model run hf.co/Beling86/Qwen3.8-Flash-Next-3.48BPW
```

Qwen3.8-Flash-Next-3.48BPW

Mixed-precision AutoRound quantization of Qwen/Qwen3.8-Flash-Next for vLLM. Text-only (vision and MTP removed). Non-PLE weights: 51.07 GiB, 3.48 bits per weight on average (packed weights, scales, zero points and BF16 parts). The 51.2 B-parameter per-layer embedding (PLE) is Qwen's own FP8 table with its global scale (~47.7 GiB, offloaded to host memory by vLLM).

Precision layout

Component	Format
Routed experts, all 48 layers	W3A16, group 128, symmetric
GDN linear attention, QSA self-attention projections	W8A16, group 128, symmetric
Indexer, decay/gate controls, hyper-connections, shared experts, routers, embeddings, LM head, norms	BF16
Per-layer embedding (PLE)	FP8 E4M3 + global scale (official Qwen tensors)

AutoRound 0.16.1 (SignRound), 250 iterations/block, 104 x 2048-token calibration samples (chat/code/reasoning/long-context, production chat template). Format auto_round:auto_gptq.

Fidelity vs. BF16

Teacher-forced full-vocabulary KL(P_bf16 ‖ Q_quant) on a held-out set (24 × 1,024 tokens; sources and splits disjoint from calibration). Both models run without the PLE injection (the BF16 PLE table does not fit the quantization host), so this measures the quantization of the transformer blocks only:

Domain	Positions	Mean KL	Median KL	P99 KL	Top-1 agreement	PPL bf16 → quant
all	24552	0.11572	0.01821	1.5207	88.78%	5.431 → 5.677
chat	10230	0.13859	0.03363	1.6373	85.61%	10.235 → 11.257
code	5115	0.08184	0.00106	1.3862	94.43%	2.773 → 2.773
math	5115	0.07272	0.00463	1.2121	92.98%	2.675 → 2.612
prose	4092	0.15467	0.05358	1.5946	84.38%	6.255 → 6.622

P99.9 KL 4.5617, max 12.482. vLLM load/generation was not validated for this release (smoke test skipped).

Serving (vLLM)

Status: not validated on vLLM for this release (the quantization host has a 64 GB GPU, too small for the ~50 GiB of weights plus vLLM's startup; only the KL evaluation and the structural gates below were run). The checkpoint ships the official FP8 per-layer-embedding (PLE) shards with their global scale (ple-*.safetensors) and ple_embedding_dtype: float8_e4m3fn in config.json, so vLLM's FP8 PLE loader should be selected (from reading vLLM's source; not tested).

python vllm-patch.py              # run with the python of your vLLM environment (see below)
vllm serve Beling86/Qwen3.8-Flash-Next-3.48BPW --trust-remote-code --dtype bfloat16 --engram-config '{"cpu_offload": true, "use_thp": true}'

vllm-patch.py (in this repo; idempotent, keeps .orig backups, --check / --revert) fixes a vLLM bug we hit while serving a sibling checkpoint with per-expert mixed bit-widths: vLLM's INC parser reads the per-layer bits of a fused MoE layer only if the class name contains fusedmoe, but the class is now RoutedExperts, so every MoE layer falls back to the global bits: 4 and the 2/3-bit experts are allocated with 4-bit containers (CUDA OOM while the model is constructed, before any weight is read). This checkpoint has the same layout (W2/W3 experts under a global bits: 4), so expect the same failure on stock vLLM without the patch. The Ling-specific KDA patch in that script does not apply to this model and reports "not applicable".

2/3-bit experts are meant to run on the Humming kernels, 8-bit projections on Marlin. If you serve this model, please report what happens.

Validation gates passed

Coverage, packed layout vs. config, zero points vs. sym, fused-group consistency, size ≤ 52.0 GiB (non-PLE weights), plan conformance, strict reload (no missing keys) for the KL evaluation, PLE shards and global scale.

Downloads last month: -

Safetensors

Model size

15B params

Tensor type

I32

BF16

F16

I64

F8_E4M3

Model tree for Beling86/Qwen3.8-Flash-Next-3.48BPW

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

(639)

this model