Qwen3.8-Flash-Next-FP4-Local-Abliterated-v14

繁體中文摘要：這是 DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated 的 V14 轉換版。權重與原模型逐位元相同（FP4-Local 格式不變），多的是記憶的設定、兩個程式檔，以及一份鏡像補丁 vllm-v14-patch/：在 local-inference-lab 的 vLLM 鏡像（ghcr.io/local-inference-lab/vllm:release-10152026，能跑 FP4-Local 格式的 b12x 版 vLLM）上加入記憶；建好的鏡像在 Docker Hub：lovewin10/vllm-v14:release-10152026-v14。用這個鏡像服務時，模型帶有跨請求保存的情節記憶：在一段對話裡說過的事，之後在另一段全新的對話裡問得到；CUDA graph、前綴快取、MTP 推測解碼與非同步排程照常開著。實測結果與已知的限制寫在下面。

DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated converted to the V14 episodic memory architecture (Qwen4ExpV14ForConditionalGeneration).

• The weights are the original's, byte for byte, in its FP4-Local format (ModelOpt mixed precision, NVFP4 experts with compressed scales). Nothing was trained, merged or re-quantized here.
• Added: the architecture name, an auto_map and the memory options (qwen4_exp_v14_memory_config) in config.json, two code files (configuration_qwen4_exp_v14_reinstatement.py, modeling_qwen4_exp_v14_reinstatement.py) that hold the memory, and the image patch vllm-v14-patch/. SHA256SUMS lists this repository's files; the weights' lines are the original's.
• Served by the image lovewin10/vllm-v14:release-10152026-v14 (built from vllm-v14-patch/), the model has a memory that persists across requests: what was said in one chat is answered in a later, separate chat. The store is on disk and survives restarts.
• The image is ghcr.io/local-inference-lab/vllm:release-10152026 (the vLLM that serves this checkpoint's format with b12x kernels) with the memory added; it is not an official vLLM or local-inference-lab image.

How the memory works

While a prompt is read, the keys and values of its 12 indexed-attention (QSA) layers are cut into events of up to 128 tokens and written to a store on disk, each with a low-rank engram of the state it leaves in the 36 Gated DeltaNet layers. A later prompt is read once to see what it asks for; the stored events whose keys would take its attention are recalled by one vote for all layers and heads; their engrams are composed into the request's recurrent state and the prompt is read again with the recalled keys and values merged into the attention (by the local attention's log-sum-exp, which the patched b12x QSA kernels return). Generated tokens attend the recalled events too, inside the decode CUDA graphs. Nothing is retrieved as text and nothing is added to the prompt. While generating, the request also attends the event the memory's world model expects next, predicted from the store's own transitions. A question with nothing in the store to ask for is answered right, though not always in the plain model's words (see Measured).

Serve

Pull the image from Docker Hub (12.9 GB compressed), or build it from this repository's folder (the build pulls ghcr.io/local-inference-lab/vllm:release-10152026, about 44 GB, and stops if any file the patch replaces differs from the original it was made from):

docker pull lovewin10/vllm-v14:release-10152026-v14
# or: docker build -t lovewin10/vllm-v14:release-10152026-v14 vllm-v14-patch

It starts as NVIDIA's command for one RTX PRO 6000 Blackwell starts its base (PROFILE=qwen38-flash-next); only the image and the model folder change, and a folder for the store is mounted at /memory. On one 96 GB GPU (measured on an RTX PRO 6000 Blackwell under Docker Desktop, where the port is published instead of --network host):

docker run -d --name flash-next-v14 --init --gpus device=0 `
  --restart on-failure --ipc host --shm-size 32g --stop-timeout 30 `
  --ulimit memlock=-1 --ulimit stack=67108864:67108864 -p 127.0.0.1:8000:8000 `
  -e PROFILE=qwen38-flash-next -e MODEL=/models/qwen -e PORT=8000 `
  -v <this repository>:/models/qwen:ro -v flash-next-cache:/cache -v <a folder for the store>:/memory `
  lovewin10/vllm-v14:release-10152026-v14

The profile carries the rest of NVIDIA's settings: a 262,144-token context, 16 sequences, FP8 KV cache, MTP with 3 draft tokens, b12x kernels, GPU memory utilization 0.96, the qwen3 reasoning parser and the qwen3_xml tool parser. The model is served as Qwen3.8-Flash-Next, with the sampling defaults of generation_config.json (temperature 1.0, top-p 0.95, top-k 20); thinking is on unless a request sets chat_template_kwargs.enable_thinking to false.

• VLLM_V14_MEMORY=0 serves the folder as the plain checkpoint (no memory), exactly as the base image does.
• VLLM_V14_MAX_SLOTS (default 4): requests that read the memory at once; each holds about 100 MiB of decode buffers taken from the KV cache. Other requests of the same step run without memory, the ones with a slot keep theirs.
• VLLM_V14_INDEX_GB (unset by default): the store's index rows are reserved at start-up for the payloads it holds and 8,192 more (about 0.53 GiB for a new store, at most 2 GiB), and the log warns when they are 80% full (a restart sizes them anew); a number pins the size. Beyond the rows a vote reads the rest back from the store.
• VLLM_V14_WORK_GB (default 1.25): GPU memory held back from the KV cache for the memory's work the vLLM profile does not see (a prefill's recurrent states, the keys of recent votes, the far contexts of the requests that recall).
• VLLM_V14_KV_CACHE_GB (default 0.25): GPU memory for the decoded keys of recently recalled events; VLLM_V14_LEARN_CPU=0 fits the world model on the GPU instead of the CPU.
• VLLM_V14_TRACE=1 logs every recall decision.
• With the memory on the KV cache is smaller (measured: 656,518 tokens instead of 766,241 with the defaults above). The GPU still shows nearly full, as with the base image: the profile hands everything below --gpu-memory-utilization 0.96 to the KV cache. A lower -e GPU_MEMORY_UTILIZATION (or a fixed -e KV_CACHE_MEMORY_BYTES=..., the base image's options) leaves GPU memory free, all of it taken from the KV cache: about 0.96 GiB, some 52,000 tokens, for every 0.01. Below about 0.90 one 262,144-token request no longer fits and vLLM stops after loading the weights, so add -e MAX_MODEL_LEN=-1 as well (vLLM serves the longest context that fits and logs it as Auto-fit max_model_len) or a smaller MAX_MODEL_LEN. At 0.85 that is roughly 50,000–85,000 tokens, fewer with a larger store index. These figures are worked out from the start-up log at 0.96, not measured at the lower settings.
• Other images do not know the V14 architecture name, including the DGX Spark image of the original's serve-spark.sh: there, serve the original repository (or set architectures in config.json back to Qwen4ExpForConditionalGeneration). vllm-v14-patch/README.md lists what the patch changes.

Measured

One RTX PRO 6000 Blackwell (96 GB) under Docker Desktop, 2026-11-12, with the profile above and the image built from vllm-v14-patch/, starting from an empty store. Every statement and every question is a chat of its own with no shared history; temperature 0, thinking off (enable_thinking: false). A handful of questions: a functional check, not a benchmark.

	Memory on	Memory off (VLLM_V14_MEMORY=0)
A fact told in one chat, asked in a new chat (6 facts)	6 / 6	0 / 6
The same six questions asked again after four 20,000-token prompts (vLLM answers a prompt it has read before from its checkpoint, without a forward)	6 / 6
Unrelated questions asked afterwards (4)	4 / 4	4 / 4
Decode speed, three answers of 240 to 400 tokens	99.0 tokens/s (85.4, 125.5, 86.2)	122.4 tokens/s (114.8, 145.9, 106.4)
First token of those three short prompts	1.06 s (0.69, 1.24, 1.24)	0.15 s (0.13, 0.12, 0.19)
First token of a 1,944-token prompt; the same prompt again 15 s later	2.67 s; 0.20 s	0.50 s; 0.10 s
MTP mean acceptance length during the speed prompts	2.12	2.10
KV cache	656,518 tokens	766,241 tokens
Device memory the memory takes at start-up	2.47 GiB (decode buffers 0.41, index 0.71, held back for its work 1.25)

Against the image first published on 2026-11-12: KV cache 564,006 → 656,518 tokens (the memory's own device memory 3.77 → 2.47 GiB), decode 96.7 → 99.0 tokens/s; recall unchanged.

• First token. With the memory a prompt is read without a CUDA graph and, when its vote recalls something, twice.
• Decode. Decoding and MTP verify keep their CUDA graphs with the merge captured in them: two Triton kernels per memory layer, about 0.75 ms of a step measured alone (1.4 ms with the earlier kernel and torch ops). Most of the rest are eager pauses of about a tenth of a second each: the request votes again at its 32nd, 96th and 224th generated token and forms the expected event anew at 32, 96 and 224 tokens, both over every memory layer.
• The answers are not the plain model's word for word. Unrelated answers stay right but are often worded differently from the memory-off ones: a stored event that resembles a question can pass the vote, and while generating the request attends the world model's expected event.
• Long prompts. Four rounds of 20,000-token prompts: the largest prefill took 2.27 GiB of device memory above what was allocated when it began (2.01 GiB of it vLLM's own activations, which its profile budgets), the container's memory went from 34.5 to 35.7 GiB, and the facts were still answered 6 of 6 afterwards.
• Host memory stays level as the store grows. 35 new chats of 24,500 tokens each, every event written to an empty store on a hard disk (6,752 events, three world model fits): once the read caches were full (decoded keys 0.25 GiB, stored bytes 0.5 GiB, 256 state engrams), the engine process's own memory stayed at 6.6 to 7.1 GiB, up to 1.2 GiB of it events waiting for the disk. The image pushed before the evening of 2026-11-12 kept about 0.2 MB for every stored event and a fit's copy of the index until a later garbage collection; a long session could exhaust Windows' commit under WSL 2 and stop the virtual machine. Pull the image again.
• A request sent right behind another waits for that one's events to be written before it is matched against the store (the same 1,944-token prompt sent at once: 1.19 s to the first token).

Limits

• It recalls what a question asks for in similar words. The vote compares the question's attention queries with stored keys; a question that shares little with the note that answers it can miss it.
• Two-step questions are the weak case, as on the other V14 models: the second fact is recalled only if its vote reaches half of the first one's (recall_ratio).
• A corrected fact is not retired; recalled events carry no time stamp in this image (time headers are off: they need a text encoder of the backbone the engine does not have).
• Decoding rows beside a prefill read no memory in that step. vLLM runs prefill chunks and decoding requests in one batch; a decoding row in such a batch attends only its own context for that step.
• A prefill with memory runs without a CUDA graph (it votes and recalls on the host); with this profile prefills above 64 tokens do anyway. Decode and MTP verify keep their CUDA graphs, the merge captured in them.
• Requests beyond VLLM_V14_MAX_SLOTS (4) run without memory (requests that have a slot keep theirs in the same steps), and requests given as prompt embeddings do too.
• A prompt the engine has read before (a regenerated reply) is answered from its checkpoint without a forward; it recalls what its first reading recalled when that was one of the last 64 requests, and otherwise only from the vote at its 32nd generated token.
• Pictures and videos of a prompt are stored as their placeholder tokens; they are not identified, so a picture is not recognized again later.
• No separation between users. Every request reads and writes the same store, which keeps what the prompts contained (token ids, attention keys and values). Run one server with its own /memory for each user or tenant whose memories must stay apart; delete the directory to forget.
• GPU memory. The memory's buffers and index are taken from the KV cache at start-up (see Serve).
• Host memory under Docker Desktop (WSL 2). The virtual machine's page cache, the 27 GB PLE table the profile keeps in host memory and the GPU memory all count against the host's commit limit; on the machine this was measured on the start failed until the page cache was dropped while the weights loaded. Keep memory in .wslconfig well below the host's RAM plus page file minus the GPU memory.
• The recall thresholds (recall_ratio, recall_floor) were set on Qwen3.5-0.8B and not re-tuned for this model.
• The trained weights are untouched, so everything the original card says about the model itself still holds, including its own limitations.

Credits and license

• The model is DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated (from nvidia/Qwen3.8-Flash-Next-FP4-Local, from Qwen3.8-Flash-Next); its card follows below, unchanged, and its licenses apply to this repository (the NVIDIA Open Model Agreement in LICENSE, the Qwen Community License in upstream/, see NOTICE).
• The serving engine is vLLM (Apache-2.0) as built by local-inference-lab with b12x kernels in FlashInfer (Apache-2.0). vllm-v14-patch/ changes five of the image's files (each marked as changed, the originals in vllm-v14-patch/pristine/) and adds the memory, under Apache-2.0 (vllm-v14-patch/LICENSE); the image it builds is not an official image of vLLM, FlashInfer or local-inference-lab.
• Conversion, memory code and image patch: LTM-Lab / lovewin10.

---

Original model card (DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated)

Qwen3.8-Flash-Next FP4-Local — Abliterated

Fewer refusals, the same compact weight formats, and similar single-Spark decode speed to NVIDIA's FP4-Local base.

Quick start · Evaluation · Two-Spark recipe · Method

A standalone derivative of NVIDIA's Qwen3.8-Flash-Next-FP4-Local, modified through abliteration: a weight edit that reduces the model's learned refusal behavior. The edit is baked into the checkpoint, so no adapter or runtime hook is needed.

The weights occupy approximately 96 GiB on disk and run on one DGX Spark using the supplied launcher. A separate two-Spark serving recipe is linked below.

This checkpoint keeps NVIDIA's FP4-Local storage format, including its compressed NVFP4 scales. Like the base, it requires a compatible B12X vLLM container. Use the pinned image in serve-spark.sh; a standard Transformers or stock vLLM loading example is not a supported path for this checkpoint.

Highlights

• Substantially reduced refusals: 0 of 100 development and 6 of 100 holdout prompts were detected as refusals, against 99 and 100 for the NVIDIA base.
• Base-compatible weight layout: every tensor retains the base checkpoint's name, dtype and shape. Routed experts remain NVFP4; attention and shared-expert projections remain MXFP8.
• Similar single-Spark speed: median decode rates of 38.0 tok/s for prose and 54.3 tok/s for code, against 39.4 and 50.7 for the base under the same serving profile.
• Functional checks passed: image input, tool calling, thinking controls and MTP speculative decoding passed the checks described below.
• Self-contained: download the repository and run the included DGX Spark launcher.

These results describe the MXFP8 release dated October 22, 2026. They are bounded measurements, not a claim that all capabilities are unchanged.

Model overview

Original model	Qwen/Qwen3.8-Flash-Next by Qwen
Quantized base	nvidia/Qwen3.8-Flash-Next-FP4-Local by NVIDIA, revision b9ce18a
Parameters	125B language-model parameters, 6B active, plus a 51B n-gram embedding table and a 4B MTP layer, as described in NVIDIA's card
Input / output	Text and images in, text out
Weight storage	Approximately 96 GiB / 102 GB across 21 shards
Precision	Mixed precision: NVFP4 routed experts, MXFP8 attention and shared-expert projections, and other tensors in their original base formats
What changed	Refusal-direction ablation in the 48 attention output projections and the routed/shared expert down projections
Single-Spark context	Configured for 262,144 tokens; retrieval checks covered approximately 8K and 30K tokens
Tested hardware	DGX Spark, GB10, 128 GB unified memory; separate single-node and two-node profiles

Weight storage is the size on disk, not the complete runtime memory requirement. The single-Spark launcher uses disk-backed n-gram embeddings and a 104 GiB container memory cap.

Quick start (DGX Spark)

You need:

• One DGX Spark with Docker and NVIDIA GPU support.
• The Hugging Face CLI, hf. See the installation guide.
• Space for approximately 102 GB of model files, plus the Docker image and compilation caches.
• Enough free unified memory to run the 104 GiB container. Avoid running another large model alongside it.

Download the complete repository, preserving the tensors/ directory, then start the server:

hf download DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated \
  --local-dir ./qwen-abliterated

bash ./qwen-abliterated/serve-spark.sh ./qwen-abliterated

The launcher uses this pinned runtime image:

eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595

First start: compilation and autotuning took about 23 minutes on the tested Spark. Later starts reused the cache and took about 5 minutes. Startup time varies by host and cache state.

The launcher stays attached to the server logs. In another terminal, check readiness:

curl --fail http://127.0.0.1:8000/health

Once that succeeds, the OpenAI-compatible API is available at http://127.0.0.1:8000/v1, with model ID qwen-trial117.

Send a request

This example disables thinking and requests a direct answer:

curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-trial117",
    "messages": [
      {
        "role": "user",
        "content": "Explain how a heat pump works in three sentences."
      }
    ],
    "max_tokens": 256,
    "chat_template_kwargs": {
      "enable_thinking": false
    }
  }'

For Python, install the client in your Python environment:

python3 -m pip install openai

This example enables thinking explicitly:

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="none",
)

reply = client.chat.completions.create(
    model="qwen-trial117",
    messages=[
        {"role": "user", "content": "Write a haiku about unified memory."}
    ],
    max_tokens=2048,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True}
    },
)

message = reply.choices[0].message
reasoning = getattr(message, "reasoning", None)
if reasoning:
    print(reasoning)
print(message.content)

• Thinking is on by default. Set chat_template_kwargs.enable_thinking per request. Reasoning is returned in the message's reasoning field.
• Tools use the OpenAI-compatible tools field, with the qwen3_xml parser.
• Sampling defaults come from generation_config.json: temperature 1.0, top-p 0.95 and top-k 20. The greedy evaluation settings below are separate from these defaults.

To stop the server from another terminal:

docker stop -t 90 qwen-trial117

Rerun the launcher to start it again; compiled caches persist.

Serving profile

The launcher follows NVIDIA's published DGX Spark profile for the base:

Setting	Default
Tensor parallelism	1
Maximum context	262,144 tokens
Active sequences	Up to 8
KV cache	FP8
Prefix caching / chunked prefill	Enabled
Speculative decoding	MTP, 4 draft tokens
Kernels	B12X MoE, linear and GDN
N-gram embeddings / PLE	Disk-backed
Container memory cap	104 GiB

Eight active sequences do not mean eight simultaneous maximum-length contexts; available KV cache limits the total.

Variable	Default	Effect
PORT	8000	API port
HOST	127.0.0.1	Bind address; use an authenticating proxy before exposing the API beyond the host
MTP_TOKENS	4	Speculative draft tokens; 0 disables MTP
CACHE_DIR	~/.cache/qwen-trial117	Compiled kernels and autotune results

Evaluation

The following results cover the October 22, 2026 MXFP8 release on one DGX Spark, using serve-spark.sh unless otherwise stated. NVIDIA's base checkpoint was served with the same profile.

The base revision, runtime image, frozen intervention inputs and validation summary are recorded in provenance.json. File hashes and sizes are recorded in SHA256SUMS and release-files.json.

Refusals and drift

Refusal tests used 100 development and 100 holdout prompts from mlabonne/harmful_behaviors. Answers were greedy, seed 3741, thinking off and capped at 100 tokens. A refusal regex classified the responses.

Checkpoint	Development refusals	Holdout refusals
NVIDIA base	99 / 100	100 / 100
This model	0 / 100	6 / 100

These are detected refusals in short responses. They do not measure answer correctness, usefulness or universal compliance. Raw responses to refusal-test prompts are not published.

Distribution drift was measured separately on benign prompts from mlabonne/harmless_alpaca, using 100 prompts per split:

Metric, this model versus NVIDIA base	Development	Holdout
First-token KL, nats	0.207	0.219
First-token top-1 agreement	81%	84%

KL measures a difference between probability distributions. Here it uses the first answer token and renormalized top-20 candidates. It is not a percentage, a full-vocabulary KL measurement or a general capability score.

Capabilities

• Functional checks: 11 of 11 passed. Model listing, streamed usage, thinking on and off, automatic and forced tool calls, a tool-result follow-up returning exact JSON, image colour recognition, and exact retrieval of three planted markers from documents of about 8,000 and 30,000 tokens.
• Vision: 10 of 10 solid-colour images named correctly. This is a basic image-input check, not a broad vision benchmark.
• Speculative decoding: observed MTP acceptance was comparable to the base. A greedy audit found no emitted-token mismatches against the target model's top choice on the audited output.
• Writing: 13 of 20 editing and composition cases passed all scoring dimensions, against 14 for the base. An AI reviewer scored the fixture; it was not human-adjudicated. The two changed cases also varied between runs of the other checkpoints.

Two writing behaviors repeated across runs: short arithmetic questions could trigger step-by-step explanations that exhausted a tight output budget, and requests to tighten a long note received lighter edits than the base.

These checks establish specific working paths. They do not establish unchanged overall capability.

Speed

Single request, greedy decoding, thinking off, 512 output tokens, MTP 4. Decode rates are medians of six probes over two rounds; ranges are shown in parentheses.

Metric	This model	NVIDIA base
Prose decode, tok/s	38.0 (37.7–39.6)	39.4 (39.1–40.9)
Code decode, tok/s	54.3 (48.5–56.0)	50.7 (49.3–52.4)
Speculative decode step	67.7–67.9 ms	66.8–67.2 ms
KV cache capacity	947,989 tokens	902,668 tokens
Time to first token, short prompt	0.05–0.27 s	—
Time to first token, ~30,000-token prompt, cold	12.7 s	—

Decode rate measures generation after the first content token; it excludes prefill. Speed depends strongly on content because speculative decoding accepts more draft tokens on predictable text such as code.

On a separately tuned single-Spark stack with MTP 2, based on diffbot's DGX Spark recipe, prose reached 45.3–45.7 tok/s against 46.1–47.1 for the base. That stack is not included in this repository, and those figures do not describe the supplied launcher's defaults.

Two DGX Sparks

For two-node serving, see XtraSaltyDev's TP=2 recipe and evaluation. It serves this checkpoint without modifying the weights.

The qualified profile uses 262K context, 16 active sequences, BF16 KV cache, MTP4 and disk-backed PLE, with a 104 GiB container cap on each Spark. It requires working RoCE networking and the model files on both nodes.

Measured on October 22, 2026:

Workload	Original eight-slot profile	Qualified sixteen-slot profile
16 concurrent short requests: aggregate output	229.0 tok/s	313.4 tok/s (+37%)
Same workload: p95 first token	11.89 s	0.77 s
16 concurrent ~8K-input requests: aggregate output	38.64 tok/s	40.98 tok/s (+6%)

Both columns use the same abliterated checkpoint and runtime image. This comparison measures serving-profile changes, not performance against NVIDIA's unmodified model.

The short workload used 62–64 input tokens and 256 forced output tokens, greedy decoding and thinking off. Aggregate throughput includes queue and prefill time across the request wave; it is not a single user's decode rate. The samples are small and do not establish production latency guarantees. Serial prose decode was about 8% slower than the eight-slot reference.

The selected profile also passed 10/10 strict JSON retrieval checks at 261,623 input tokens. This result belongs to the two-Spark BF16-KV profile; it does not extend the single-Spark FP8-KV validation above. Sixteen scheduler slots do not imply sixteen simultaneous 262K contexts.

See the GitHub repository for setup, full results and tradeoffs, and the pinned checkpoint revision and raw evidence.

How it was made

1. Find the refusal direction. The method follows Arditi et al. (2024) and the parameterization of Heretic, reimplemented for this checkpoint's native vLLM backend. Per-layer directions come from attention-output activations on harmful and harmless prompt sets. An Optuna TPE search balanced refusals against KL, and trial 117 was selected.

2. Bake the intervention into the weights. The selected projection was applied to the 48 attention output projections and the routed/shared expert down projections. Expert weights were re-rounded in NVFP4 and MXFP8 while preserving their scale encodings.

3. Restore attention to MXFP8. The intermediate bake stored the edited attention output weights in BF16. Constrained rounding re-encoded them into the base's MXFP8 format while preserving the intended directional edit.

The selected trial uses the accepted continuation recalibration. Its parameters, directions and row norms are included in bake-selection.json, frozen-directions.json and frozen-row-norms.json.

Direction-preserving re-encode

Plain MXFP8 rounding of the edited attention weights would introduce error along the removed direction equal to about a tenth of the intervention's size.

The re-encode script instead rounds each weight to one of its two nearest E4M3 values, using the base's scale rule of one E8M0 exponent per 32 inputs. A greedy pass chooses roundings that cancel error along the removed direction while minimizing added squared error.

Across the 48 attention output projections:

• Error along the removed direction is at most 0.002% of the intervention.
• Overall relative weight error is 1.47–1.93%.
• The fraction of the direction removed matches the BF16 intermediate to within 0.00002.

Per-layer measurements are in reencode/reencode-report.json. These are weight-space measurements, separate from the behavioral tests above.

The tokenizer, chat template, vision encoder and built-in MTP layer are NVIDIA's. The model configuration is unchanged apart from a standalone_provenance metadata entry. This is an edited derivative of NVIDIA's quantized checkpoint, not a repeat of its quantization-aware distillation.

Files

File	Contents
tensors/, model.safetensors.index.json	Weights: 21 shards in the base layout
config.json, tokenizer, chat_template.jinja, preprocessor configs	Base configuration and processing files, with the provenance addition noted above
serve-spark.sh	Single-Spark launcher
bake-selection.json, frozen-directions.json, frozen-row-norms.json	Frozen trial 117 intervention inputs
reencode/	MXFP8 re-encode script and per-layer report
provenance.json	Build information and validation summary
SHA256SUMS, release-files.json	Release file hashes and sizes
LICENSE, NOTICE, upstream/	Governing terms, notices and NVIDIA's original card

Limitations

• Narrow evaluation: broad capability benchmarks and video input were not evaluated for this derivative. The vision checks were limited to solid colours.
• Profile-specific evidence: single-Spark retrieval checks reached about 30K tokens. The separate two-Spark profile passed the near-262K checks described above. Neither proves reliable performance on every task at those lengths.
• Original targets not met: the project aimed for zero refusals and a KL ceiling of 0.20 and did not meet its original qualification targets. The top-20 KL reported here is distinct from the original full-vocabulary metric.
• Refusals remain: 6 of 100 holdout prompts were detected as refusals.
• Hardware and runtime scope: validation used DGX Spark and pinned B12X runtimes. Other hardware and runtime versions are not qualified here.
• Outputs can be wrong or inconsistent: reduced refusal behavior does not establish correctness, factual reliability or preserved general capability.

Responsible use

Abliteration reduces learned refusal behavior. The model may attempt requests the original declines and can produce harmful, offensive, false or illegal content.

• You are responsible for how you use it and for complying with applicable law and the governing licenses.
• Add your own safeguards before deploying it to end users.
• Do not use it for activities prohibited by the Qwen or NVIDIA terms.

There is no warranty of any kind.

License

This checkpoint is derived from NVIDIA's quantized checkpoint and Qwen's original model. Their governing terms continue to apply; this release does not relicense the base weights.

• NVIDIA Open Model Agreement: the governing download terms for NVIDIA's checkpoint. A copy is included in LICENSE.
• Qwen Community License 1.0: additional terms for the original model, included in upstream/QWEN-LICENSE.

Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement.

Read the full license texts before use or redistribution, including their commercial-use conditions. Runtime containers and external software have their own licenses.

Citation

@misc{developingdad2026qwen38abliterated,
  title        = {Qwen3.8-Flash-Next FP4-Local -- Abliterated},
  author       = {DevelopingDad},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated}}
}

@article{arditi2024refusal,
  title   = {Refusal in Language Models Is Mediated by a Single Direction},
  author  = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
  journal = {arXiv preprint arXiv:2406.11717},
  year    = {2024}
}

@misc{heretic,
  author       = {Weidmann, Philipp Emanuel},
  title        = {Heretic: Fully automatic censorship removal for language models},
  year         = {2025},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/p-e-w/heretic}}
}

Credits

This release stands on other people's work.

Model

• Qwen: Qwen3.8-Flash-Next, the original model, under the Qwen Community License 1.0.
• NVIDIA: the FP4-Local checkpoint, quantized with Model Optimizer (Apache-2.0), under the NVIDIA Open Model Agreement. NVIDIA's published DGX Spark profile is the basis of serve-spark.sh.

Method

• Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee and Neel Nanda: the refusal-direction research behind abliteration.
• Philipp Emanuel Weidmann (p-e-w) and the Heretic contributors: Heretic (AGPL-3.0). Its search parameterization was reimplemented here, and no Heretic code is included. The Heretic-DGX fork was consulted while adapting it.
• Maxime Labonne (mlabonne): the abliteration write-up and the two prompt sets used to find directions and measure refusals. Neither prompt set is redistributed here.
  • mlabonne/harmful_behaviors, revision 01cead0, comes from AdvBench (llm-attacks, MIT).
  • mlabonne/harmless_alpaca, revision 02c6a92, comes from Stanford Alpaca (CC BY-NC 4.0).
• Open Compute Project: the Microscaling (MX) format specification that MXFP8 follows.

Software and serving

• vLLM (Apache-2.0): the serving engine.
• FlashInfer (Apache-2.0) and B12X by Local Inference Lab (Apache-2.0): the kernels.
• eugr/spark-vllm-docker (MIT): the DGX Spark vLLM image used for building and serving.
• Optuna (MIT), PyTorch (BSD-3-Clause) and safetensors (Apache-2.0).
• diffbot: the tuned-stack recipe behind the optional single-Spark speed figure.
• XtraSaltyDev: the separate two-Spark serving recipe, evaluation harnesses and deployment results.

External projects retain their own licenses. This release is not endorsed by Qwen, NVIDIA or any of the projects above.