Qwen3.6-35B-A3B-abliterated-NVFP4-MTP

The fastest abliterated Qwen3.6 for NVIDIA Blackwell / DGX Spark — NVFP4 quantized with MTP (Multi-Token Prediction) layers for speculative decoding.

95.2 tok/s decode on a single DGX Spark (GB10) with MTP=3.

What Makes This Different

No other model on HuggingFace combines all three:

Feature	This Model	sakamakismile NVFP4	RedHatAI NVFP4
NVFP4 weights	✅	✅	✅
MTP layers	✅ BF16	❌	✅
Abliterated	✅	✅	❌
Speculative decode	✅ 95 tok/s	❌ ~70 tok/s	✅ but censored

Without MTP layers, you're limited to ~70 tok/s on GB10. MTP enables speculative decoding which gives a +35% speedup.

Model Details

• Architecture: Qwen3 MoE (Mixture of Experts), 35B total / 3B active parameters
• Main weights: NVFP4 quantized (21.9 GB) — optimized for NVIDIA Blackwell CUTLASS kernels
• MTP layers: BF16 (1.6 GB, 19 tensors in model-mtp.safetensors)
• Total size: ~23.5 GB
• Context length: 131,072 tokens
• Abliteration: Refusal direction removed via representation engineering (from huihui-ai)

How It Was Made

1. Started with sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4 (NVFP4 main weights, no MTP)
2. Extracted BF16 MTP layers from huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated (full BF16 model)
3. Injected MTP tensors into a separate model-mtp.safetensors file
4. Updated model.safetensors.index.json with MTP weight mappings
5. Verified with vLLM speculative decoding (MTP=1 through MTP=4)

Benchmark Results

Hardware: NVIDIA DGX Spark (GB10 Blackwell, sm_121, ~128GB unified memory) Engine: vLLM v0.23.1 with FlashInfer + CUTLASS NVFP4 MoE kernels Benchmark: 256 generated tokens at various context lengths

Config	1K ctx	4K ctx	16K ctx	32K ctx	64K ctx
MTP=3, seqs=1 (best)	92.2	95.2	90.7	89.7	84.4
MTP=2, seqs=4	87.2	75.5	82.8	81.6	74.4
MTP=4, seqs=1	75.6	71.3	74.0	77.6	74.4
No MTP (NVFP4 only)	~71	~70	~74	~68	~70

MTP=3 is the sweet spot. MTP=4 regresses due to speculation overhead exceeding acceptance rate.

Recommended vLLM Launch Config

docker run -d --gpus all --network host --ipc host --shm-size=16g \
  --name vllm-qwen36 \
  -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  -e TORCH_MATMUL_PRECISION=high \
  -v /path/to/models:/models \
  ghcr.io/spark-arena/dgx-vllm-eugr-nightly:latest \
  vllm serve /models/Qwen3.6-35B-A3B-abliterated-NVFP4-MTP \
    --host 0.0.0.0 --port 8000 --max-model-len 131072 \
    --max-num-batched-tokens 32768 --max-num-seqs 1 \
    --trust-remote-code --gpu-memory-utilization 0.7 \
    --reasoning-parser qwen3 --kv-cache-dtype fp8 \
    --load-format instanttensor --attention-backend flashinfer \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
    --enable-prefix-caching -tp 1