Refusal in Language Models Is Mediated by a Single Direction
Paper • 2406.11717 • Published • 16
Swift-Qwen3.8-27B-Uncensored-NVFP4
NVFP4 checkpoint of ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP, an abliterated Swift-Qwen3.8-27B (UkisAI's reasoning-efficient fine-tune of Qwen3.8-27B). For vLLM and SGLang. GGUFs for llama.cpp: Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF.
- Uncensored: 15/100 refusals against Swift's 98/100, KL divergence 0.0634 (measured on the BF16 source).
- Swift's own NVFP4 recipe, unmodified, from ukisai/Swift-Qwen3.8-27B-NVFP4, calibrated with NVIDIA ModelOpt.
- W4A4: runs on native FP4 kernels on Blackwell.
- MTP head and vision tower in BF16, bit-identical to the source.
Precision
| Component | Precision |
|---|---|
192 MLP projections and lm_head |
NVFP4 (block size 16, weights and activations) |
144 Gated DeltaNet projections (in_proj_qkv, in_proj_z, out_proj) |
FP8 E4M3 |
64 attention projections (q_proj, k_proj, v_proj, o_proj) |
FP8 E4M3 |
| Embeddings, norms, small DeltaNet tensors, vision tower, MTP head | BF16 |
| KV cache | not quantized |
21.9 GB, NVIDIA ModelOpt mixed-precision format.
Usage
vllm serve ajgazin/Swift-Qwen3.8-27B-Uncensored-NVFP4 \
--max-model-len 32768 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
# self-speculative decoding with the built-in MTP head: add
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
Needs a vLLM with ModelOpt mixed-precision support (tested on 0.29.0). No --quantization flag.
Sampling, as for Swift and Qwen: temperature 1.0, top_p 0.95, top_k 20, min_p 0. The model thinks before answering by default.
Tested on an RTX 5090 (32 GB) with vLLM 0.29.0: NVFP4 layers on FlashInfer's CUTLASS FP4 kernel (W4A4), FP8 layers on FlashInfer's FP8 kernel, text, MTP decoding and image input all working.
--gpu-memory-utilization 0.88 |
Weights | KV cache |
|---|---|---|
| Without MTP | 18.9 GiB | 91,629 tokens |
| With MTP | 19.7 GiB | 43,194 tokens |
The model
The source applies the refusal direction of orcarouter/Qwen3.8-27B-Uncensored (Arditi et al. 2024, one direction) to Swift's weights. Method, direction and scripts are on the source model's card.
| Model | Refusals | KL divergence |
|---|---|---|
| BF16 source (against Swift) | 15/100 | 0.0634 |
| Swift-Qwen3.8-27B | 98/100 | 0 |
Measured with Heretic on the BF16 source. Not re-measured on this checkpoint: Heretic cannot load a ModelOpt export.
Quantization
- ModelOpt
0.47.0rc0at commit022767c, the one Swift used. One RTX PRO 6000 Blackwell. - Recipe:
modelopt-recipe.json, Swift's. Local-Hessian calibration with the FP8 scale sweep for NVFP4, max calibration for FP8. - Calibration: 2,048 samples (6,450,474 tokens, up to 4,096 each) from NVIDIA's seven Nemotron
post-training v3 datasets, rendered with the chat template, seed 42. Swift's sample selection
script is not published, so the samples differ from theirs;
calib-manifest.jsonrecords ours.
Checks
| Held-out NLL (64 samples, disjoint from calibration) | |
|---|---|
| BF16 | 2.93182 |
| Quantized (ModelOpt simulation, weights and activations) | 2.93752 (+0.19%) |
- 193 NVFP4 and 208 FP8 modules, exactly the recipe's; tensor and scale counts equal Swift's checkpoint.
- All 798 BF16 tensors, the 15 MTP tensors included, are bit-identical to the source
(
precision-inventory.csv).
Reproduce
quantization/ has the recipe, manifests, scripts, package list and run log.
git clone https://github.com/NVIDIA/Model-Optimizer && cd Model-Optimizer
git checkout 022767c7ab3d7d36211affd85e5c496770cde768
pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cu130
pip install ".[hf]" transformers==5.14.1 datasets==5.0.1 accelerate==1.14.0 safetensors==0.8.0 \
flash-linear-attention==0.5.2
cd ..
# from a download of quantization/; <source dir> is the source repo at revision f39e20d
python scripts/check_recipe.py --recipe modelopt-recipe.json \
--index <source dir>/model.safetensors.index.json
python scripts/build_calib.py --manifest calibration-manifest.json --tokenizer <source dir> --out calib
python scripts/quantize_nvfp4.py --model <source dir> --recipe modelopt-recipe.json \
--calib calib/calib.jsonl --heldout calib/heldout.jsonl --out out \
--modelopt-examples Model-Optimizer/examples/hf_ptq
python scripts/check_export.py --out out --source <source dir> --recipe modelopt-recipe.json
About 60 GB of GPU memory, about an hour.
Not evaluated
Refusals and KL divergence of the quantized model, general benchmarks, and FP8 KV cache (the checkpoint has no KV scales).
License
Derivative of Swift-Qwen3.8-27B, under the Swift Open License v1.0 (Swift model card): free for individuals and organizations up to US$1,000,000 annual recurring revenue, above that commercial use needs a Swift Enterprise License from UkisAI. Qwen3.8-27B and orcarouter/Qwen3.8-27B-Uncensored are Apache 2.0.
Intended use
The model answers requests the original declines. You are responsible for how you use it and for complying with applicable law and the license.
Credits
- Qwen for Qwen3.8-27B.
- UkisAI for Swift-Qwen3.8-27B and its NVFP4 recipe.
- OrcaRouter for Qwen3.8-27B-Uncensored and its refusal direction.
- NVIDIA for ModelOpt and the Nemotron datasets.
- Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).
- Heretic, used to evaluate the source model.
- Downloads last month
- 605
Model tree for ajgazin/Swift-Qwen3.8-27B-Uncensored-NVFP4
Base model
Qwen/Qwen3.8-27B Finetuned
ukisai/Swift-Qwen3.8-27b Finetuned
ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP