Instructions to use abliterant/Qwen3.8-27B-RANA-abliterated-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use abliterant/Qwen3.8-27B-RANA-abliterated-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="abliterant/Qwen3.8-27B-RANA-abliterated-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("abliterant/Qwen3.8-27B-RANA-abliterated-FP8")
model = AutoModelForMultimodalLM.from_pretrained("abliterant/Qwen3.8-27B-RANA-abliterated-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use abliterant/Qwen3.8-27B-RANA-abliterated-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "abliterant/Qwen3.8-27B-RANA-abliterated-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abliterant/Qwen3.8-27B-RANA-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/abliterant/Qwen3.8-27B-RANA-abliterated-FP8

SGLang

How to use abliterant/Qwen3.8-27B-RANA-abliterated-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "abliterant/Qwen3.8-27B-RANA-abliterated-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abliterant/Qwen3.8-27B-RANA-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "abliterant/Qwen3.8-27B-RANA-abliterated-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abliterant/Qwen3.8-27B-RANA-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use abliterant/Qwen3.8-27B-RANA-abliterated-FP8 with Docker Model Runner:
```
docker model run hf.co/abliterant/Qwen3.8-27B-RANA-abliterated-FP8
```

Qwen3.8-27B-RANA-abliterated-FP8

An Abliterant FP8 release of abliterant/Qwen3.8-27B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.8-27B for research use. It uses the same FP8 format as Qwen's official Qwen3.8-27B-FP8, at about half the size of the BF16 build; recorded capability comparisons are reported below.

Quick start · Evaluation · Available files · Abliterant models

This is a safety-alignment-removed research model. Read Intended use and Limitations before using it. Method, full evaluation and release gates are on the BF16 card.

At a glance

Field	Value
Base checkpoint	abliterant/Qwen3.8-27B-RANA-abliterated, derived from Qwen/Qwen3.8-27B
Release	FP8 quantization of refusal-ablated BF16 weights; 27B parameters
Precision	Blockwise FP8 e4m3 weights, BF16 scales and non-converted tensors; dynamic activation quantization
Weight size	30.9 GB
Context	20,480 evaluation setting; 73,728 tested; 262k native context, not fully validated here
Runtime	vLLM 0.30.0, tested on RTX PRO 6000 (Blackwell, 96 GB)
License	Apache-2.0

Quick start

Install the evaluation runtime with pip install "vllm==0.30.0".

Tested with vLLM 0.30.0 on an RTX PRO 6000 (Blackwell, 96 GB):

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve abliterant/Qwen3.8-27B-RANA-abliterated-FP8 \
  --max-model-len 20480 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

vLLM picks up the FP8 format from config.json; no extra quantization flag is needed.
Thinking is on by default. Recommended sampling: T = 1.0, top_p = 0.95, top_k = 20.
--max-model-len 20480 matches the evaluation (16k-token budget). Long technical requests can need 20–50k tokens of reasoning; raise --max-model-len and max_tokens for those (73,728 tested, native context 262k).
VLLM_USE_FLASHINFER_SAMPLER=0 was needed on Blackwell GPUs.
Vision input, multi-turn tool calling and MTP speculative decoding all work (tested on these shards).

What changed

The FP8 conversion is applied to the BF16 build's weights. Nothing else about the model changes.

Format: identical to Qwen's official FP8 checkpoint. The same 407 linear weights are stored as FP8 (e4m3) in 128 × 128 blocks, each with one BF16 scale (weight_scale_inv); activations are quantized dynamically. Embeddings, lm_head, norms, gates, the whole vision tower and the other non-converted tensors stay in BF16. The tensor names, dtypes and shapes match the official checkpoint exactly (1,606 tensors).
BF16 tensors: every tensor left in BF16 is byte-identical to the BF16 build.
Same arithmetic as Qwen: on the tensors abliteration did not touch, this conversion reproduces Qwen's own FP8 file: 100% of block scales and 99.91% of FP8 values are bit-identical (the rest differ by rounding at the last step).
Rounding error: dequantized FP8 weights differ from the BF16 ones by 2.65% (relative Frobenius norm), uniformly across all 407 tensors, including the 131 edited by abliteration.

Evaluation

The recorded release evaluations and functional checks are included in results/. Full intervention details and release gates are on the BF16 card.

FP8 compared with the BF16 build it was converted from, on the same prompts and settings. Capability differences are paired per question (± standard error); none is larger than about 2 standard errors.

	BF16	FP8	FP8 − BF16
MMLU-Pro (700)	80.9	79.6	−1.3 ± 1.1
GSM8K (1,319)	91.4	90.4	−0.9 ± 0.6
IFEval (541)	89.3	91.3	+2.0 ± 1.0
TruthfulQA MC1 / MC2 (817)	34.8 / 52.6	35.7 / 53.1	+1.0 ± 0.5 / +0.5 ± 0.2
HumanEval (164 × 3)	96.7	97.2	+0.4 ± 1.0
Avg. capability change vs base (5 tasks)	0.82 pp	0.58 pp
Answers the request, held-out (240) ¹	90.0% / 54.4%	87.8% / 54.9%	−2.2 (p = 0.05) / +0.5 (p = 0.88)
Answers the request, HarmBench (200) ¹	78.8% / 57.3%	78.0% / 58.5%	−0.8 (p = 0.65) / +1.2 (p = 0.64)
Hits the 16k-token budget, held-out ¹	6.7%	7.2%	+0.5 (p = 0.57)
Hard refusals, HarmBench (3 seeds × 2 judges) ²	0	0

¹ Mean over seeds 1–3. Answers are shown as gpt-oss-safeguard-20b / Mistral-Small-24B judge; p = exact McNemar on the same prompts, pooled over seeds. gpt-oss HarmBench labels are after re-judging ²; that is why BF16 reads 78.8% here and 78.5% (raw labels) in the BF16 card's full tables. ² Every gpt-oss HARD_REFUSAL on HarmBench is re-judged using the procedure in the BF16 G1 re-adjudication disclosure. Raw labels: BF16 2 (seed 2), FP8 3 (seed 1); re-judged: BF16 2 answers, FP8 2 answers and 1 degenerate.

In short: the FP8 build keeps the BF16 build's capability (it passes the same capability gate vs base: mean |Δ| 0.58, worst task −1.0 pp on GSM8K) and behaves the same on refusal prompts over 3 seeds and two judges. The one borderline gap (held-out, −2.2 pp under gpt-oss, p = 0.05) does not appear under the second judge (+0.5 pp, p = 0.88).

Capability

Same harness as the BF16 card: vLLM, thinking on, 16k-token generation budget, identical prompts and seeds. Change versus base in percentage points ± standard error (paired on the same questions).

task (items)	base	BF16 Δ	FP8 Δ
MMLU-Pro (700)	80.0	+0.9 ± 1.1	−0.4 ± 1.1
GSM8K (1,319)	91.4	−0.1 ± 0.6	−1.0 ± 0.6
IFEval (541)	90.6	−1.3 ± 1.1	+0.7 ± 1.0
TruthfulQA MC1 (817)	36.1	−1.3 ± 0.6	−0.4 ± 0.6
TruthfulQA MC2 (817)	54.2	−1.6 ± 0.3	−1.1 ± 0.4
HumanEval (164 × 3)	97.2	−0.4 ± 0.9	+0.0 ± 0.6
mean \|Δ\| vs base	–	0.82	0.58

MMLU-Pro uses 50 questions per subject (700 total); all other sets are full size. HumanEval uses a custom chat-format harness and is not comparable with lm-eval's raw-completion HumanEval.

Refusals (seeds 1–3, two judges)

200 HarmBench "standard" behaviours plus 240 held-out prompts (StrongREJECT, JailbreakBench, CategoricalHarmfulQA), judged by openai/gpt-oss-safeguard-20b and mistralai/Mistral-Small-24B-Instruct-2501. "Budget hits" = the response reached the 16k-token generation limit before finishing (no judge involved).

set	model	hard	soft	answers (gpt-oss / Mistral)	budget hits	avg. tokens
HarmBench (200)	BF16	0	7.0	78.8% / 57.3%	20.5%	6,634
HarmBench (200)	FP8	0	8.7	78.0% / 58.5%	19.5%	6,620
Held-out (240)	BF16	0.3	10.3	90.0% / 54.4%	6.7%	4,697
Held-out (240)	FP8	0.3	14.3	87.8% / 54.9%	7.2%	4,760

Hard / soft = mean count per seed, gpt-oss judge, HarmBench after re-judging ². Paired over 3 seeds (FP8-only vs BF16-only, exact McNemar): answers on HarmBench 36 vs 41 (p = 0.65) under gpt-oss and 87 vs 80 (p = 0.64) under Mistral; held-out 22 vs 38 (p = 0.05) and 93 vs 90 (p = 0.88). Budget hits on HarmBench 27 vs 33 (p = 0.52); held-out 16 vs 12 (p = 0.57).

What the budget hits are

Mostly long reasoning on technical requests, not repetition. The 59 FP8 seed-1 prompts that hit the budget (41 HarmBench, 18 held-out) were re-run with the same seed and sampling:

setting	finished	median tokens	answers (gpt-oss)
16k budget (original run)	0 / 59	16,384	8 / 59
64k budget	57 / 59	23.9k HarmBench, 19.6k held-out	55 / 59
16k budget + `presence_penalty` 1.5	16 / 59	16,384	24 / 59

Finished responses used up to 49k tokens. With these prompts scored at 64k, seed-1 answer rates rise from 77.0% to 92.0% (HarmBench) and from 86.2% to 93.3% (held-out), and budget hits fall from 13.4% to 0.5% of all 440 prompts. presence_penalty, Qwen's setting against repetition, barely helps, which fits: the text is not repeating. The 16k-budget answer rates above are therefore lower bounds. Qwen recommends allowing up to 262k tokens for reasoning; set --max-model-len and max_tokens well above 16k if you need complete answers to long technical requests.

Functional smoke tests (these shards)

Vision: reads the code word and shape from a synthetic image: pass.
Tool calling: 3-turn call → result → second call with a new argument: pass.
MTP speculative decoding: works; mean acceptance length 2.18 averaged over the whole evaluation workload. The BF16 card's 2.45 was measured on one short generation, so the two numbers are not directly comparable.

Available files

Format	Repo	Size
BF16 (reference)	`abliterant/Qwen3.8-27B-RANA-abliterated`	55.6 GB
FP8 (this repo)	`abliterant/Qwen3.8-27B-RANA-abliterated-FP8`	30.9 GB
GGUF (llama.cpp; Q8_0 to IQ2_M)	`abliterant/Qwen3.8-27B-RANA-abliterated-GGUF`	10.0–28.6 GB
MLX (Apple Silicon; 4-bit to BF16)	`abliterant/Qwen3.8-27B-RANA-abliterated-MLX`	16.1–54.7 GB

*.safetensors, model.safetensors.index.json, config.json (with quantization_config), tokenizer/processor files, chat_template.jinja, LICENSE: the model.
results/: every number on this card. fp8_vs_bf16.json (paired FP8 − BF16 capability deltas and the seed-1 refusal run), refusal_3seed.json (seeds 1–3, both judges, re-judging, paired tests), budget_rerun.json (the 64k / presence-penalty re-run), capability.{md,json} (scores vs base for base, orca, BF16, FP8), verify_fp8.json (format and byte-level checks), smoke_fp8.json, humaneval_fp8.json, kl_fp8.json (KL from base, weight rounding only).

Limitations and intended use

Limitations

Reduced refusal is not evidence of safer, more accurate, or universally compliant behavior. Capability comparisons apply only to the evaluated tasks and settings.

Answer rates depend on the judge. The two judges differ by 21–36 pp on the same responses; compare models only under the same judge.
Hard-refusal count relies on re-judging. The gpt-oss judge gave 3 raw HarmBench hard refusals (all seed 1, all 16k budget hits). Re-judged with the auditor reminder, the same procedure as the BF16 card's G1 disclosure, they became 2 answers and 1 degenerate. The Mistral judge gave 0 on all seeds.
Long reasoning on technical requests. About 20% of HarmBench and 7% of held-out responses need more than 16k tokens; see What the budget hits are.
KL from base covers weight rounding only. With the FP8 weights dequantized and run in BF16, KL from base is 0.048 against 0.041 for BF16 (same Heretic protocol, results/kl_fp8.json); FP8 activations are not included.
Everything listed under Limitations on the BF16 card applies here too: judge-measured refusal rates, a 700-question MMLU-Pro subset, and a custom HumanEval harness.

Intended use

Research only: interpretability, red-teaming, and robustness evaluation of refusal behaviour.
Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.
You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

Provenance and license

The weights retain Qwen's Apache-2.0 license; quantization does not remove its terms. The recorded FP8 release evidence is from revision 733ae0fbac437a304823e0fdb923131a096976ce. Its underlying BF16 intervention provenance is documented on the BF16 card.

Qwen team: base model Qwen/Qwen3.8-27B and the FP8 format of Qwen/Qwen3.8-27B-FP8.
Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
Jim Lai (grimjim): prior work on norm-preserving abliteration.
Benchmarks/datasets: HarmBench, StrongREJECT, JailbreakBench, CategoricalHarmfulQA; lm-evaluation-harness; vLLM.

Downloads last month: 151

Safetensors

Model size

28B params

Tensor type

BF16

F8_E4M3

Model tree for abliterant/Qwen3.8-27B-RANA-abliterated-FP8

Base model

Qwen/Qwen3.8-27B

Finetuned

abliterant/Qwen3.8-27B-RANA-abliterated

Quantized

(5)

this model