Instructions to use Beling86/Ling3.0-Flash-3.44BPW with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Beling86/Ling3.0-Flash-3.44BPW with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Beling86/Ling3.0-Flash-3.44BPW", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Beling86/Ling3.0-Flash-3.44BPW", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Beling86/Ling3.0-Flash-3.44BPW", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Beling86/Ling3.0-Flash-3.44BPW with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Beling86/Ling3.0-Flash-3.44BPW"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beling86/Ling3.0-Flash-3.44BPW",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Beling86/Ling3.0-Flash-3.44BPW

SGLang

How to use Beling86/Ling3.0-Flash-3.44BPW with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Beling86/Ling3.0-Flash-3.44BPW" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beling86/Ling3.0-Flash-3.44BPW",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Beling86/Ling3.0-Flash-3.44BPW" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Beling86/Ling3.0-Flash-3.44BPW",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Beling86/Ling3.0-Flash-3.44BPW with Docker Model Runner:
```
docker model run hf.co/Beling86/Ling3.0-Flash-3.44BPW
```

Ling3.0-Flash-3.44BPW

Mixed-precision AutoRound quantization of inclusionAI/Ling-3.0-flash for vLLM. Text-only; the MTP (next-token-prediction) layer is removed. Checkpoint size 49.82 GiB, 3.44 bits per weight on average over all tensors (packed weights, scales, zero points and BF16 parts).

Precision layout

Component	Format
Routed experts, layers 2, 41	W4A16, group 128, symmetric
Routed experts, other 38 MoE layers	W3A16, group 128, symmetric
Attention (KDA + MLA) projections	W8A16, group 128, symmetric (KDA `f_proj` kept BF16)
Dense MLP (layers 0–1)	W8A16, group 128, symmetric
Routers, shared experts, embeddings, LM head, norms	BF16

AutoRound 0.16.1 (SignRound), 200 iterations/block, 128 × 2048-token calibration samples (chat / code / reasoning / long-context, production chat template). Format auto_round:auto_gptq.

Fidelity vs. BF16

Teacher-forced full-vocabulary KL(P_bf16 ‖ Q_quant) on a held-out set (24 × 1,024 tokens; sources and splits disjoint from calibration):

Domain	Positions	Mean KL	Median KL	P99 KL	Top-1 agreement	PPL bf16 → quant
all	24552	0.08767	0.00311	1.2216	92.88%	2.535 → 2.539
chat	10230	0.07438	0.01194	0.8137	91.54%	3.665 → 3.616
code	5115	0.08211	0.00003	1.4230	96.54%	1.674 → 1.660
math	5115	0.09923	0.00147	1.4297	93.55%	2.206 → 2.195
prose	4092	0.11341	0.00838	1.3177	90.81%	2.016 → 2.139

P99.9 KL 7.0989, max 19.787. vLLM decode-agreement check: not run.

Serving (vLLM)

Tested on vLLM 0.32.0 (single 64 GB GPU, ~49.6 GiB of weights). Stock vLLM 0.32/0.33 needs two small patches for this checkpoint; they are not quantization problems but vLLM bugs, and vllm-patch.py in this repo applies them (idempotent, keeps .orig backups, --check / --revert):

python vllm-patch.py              # run with the python of your vLLM environment
vllm serve Beling86/Ling3.0-Flash-3.44BPW --trust-remote-code --dtype bfloat16 --max-model-len 262144

#	Where	Bug	Symptom without the patch
1	`quantization/inc/config_parser.py`	The per-layer bit-width of a fused MoE layer is only read when the class name contains `fusedmoe`; the class is now `RoutedExperts`. Every MoE layer silently falls back to the global `bits: 4`, so the W3 experts get 4-bit containers.	CUDA OOM at model construction (at `lm_head`, ~63 GiB allocated for a 49.8 GiB checkpoint), before any weight is read
2	`models/bailing_moe_v3.py`	`bailing_v3_kda_attention` (host-syncing KDA state update) is not registered through `eager_break_during_capture`, so it runs inside CUDA-graph capture.	`cudaErrorStreamCaptureUnsupported` / "operation not permitted when stream is capturing" at `state_indices[~has_initial_state]`

Without patch 2 you can also run with --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' or --enforce-eager.

config.json must contain "model_type": "bailing_hybrid". vLLM detects MLA from it. The first export of this checkpoint lost the key (Ling's remote-code config class has an empty model_type, so transformers 5 dropped it when AutoRound re-saved the config); vLLM then sized the KV cache as plain 32-head attention and died at the first MLA layer with "Layer model.layers.5.self_attn.attn: page size is not divisible by the maximum page size and cannot be padded". The config.json in this repo is fixed; if you downloaded an older copy, fetch it again. The quantization pipeline now restores the key automatically (v3/fix_config.py).

2/3-bit experts run on the Humming kernels, 4/8-bit layers on Marlin. Loading takes ~100 s (49.62 GiB).

Validation gates passed

Coverage, packed layout vs. config, zero points vs. sym, fused-group consistency, size ≤ 50.0 GiB, plan conformance, strict reload (no missing keys) for the KL evaluation.

Downloads last month: 24

Safetensors

Model size

15B params

Tensor type

F32

I32

BF16

F16

Model tree for Beling86/Ling3.0-Flash-3.44BPW

Base model

inclusionAI/Ling-3.0-flash

Quantized

(50)

this model