RTX 5090 running Flash Next Abliterated EXL3 2.50bpw: ~42% faster decode in a matched maintenance test
I run Qwen3.8 Flash Next Abliterated EXL3 2.50bpw locally on an RTX 5090 + Ryzen 9 9950X3D + 64 GiB RAM. Using the existing checkpoint, a maintenance-specific static hotset improved decode from 42.80 to 60.76 token/s (+41.98%) in a matched 16K-prompt experiment. The original 50-question direct-answer battery scored v2 42/50 vs maintenance-v3 41/50. Both the speed gain and the observed quality difference are published.
Our files now require Hugging Face login and access consent; approval is automatic. The model card remains public. Hugging Face shares the requesting account name and email with the repository owner through its standard access form. No discussion reply is required. This gate applies to our configuration/evidence files, not the upstream author's weights. Access details.
Downloads: Original author's model weights · Configuration + evidence ZIP · What's included / what's still missing
This is a documentation, configuration-reference and evidence release. It does not include weights, runtime executable source, compiled extensions, an installer, or a one-click launcher. A profile and rank file alone do not install the tested hooks. The upstream model license still applies; this release does not relicense the checkpoint or engine.
What changed in the architecture?
All 512 experts in each of 48 MoE layers remain available. For the tested CPU344 configuration, 168 experts are GPU-resident and 344 are CPU-handled, with 10 routed experts per token per layer. Static hotsets prioritize resident experts and avoid dynamic expert swapping for the selected workload. No expert pruning, training, re-abliteration or re-quantization was performed. CPU RAM and disk streaming are part of the design; the entire checkpoint is not resident in VRAM.
The tested baseline settings are CPU worker threads=8, FUSED=1, static SWAP=0, MEMOPS=0, PLE row cache=OFF, band-swizzle=ON, and K4/V4 KV cache, separate from the EXL3 2.50-bpw weights. The context capacity used in long tests was 262144; actual long prompts were approximately 250K tokens. The sampled card peak minus startup ambient was around 22.8 GiB, not an exact model allocation or a new v3 VRAM reduction over v2.
Measured decode speed
| Separate matched A/B/A matrix | Baseline → candidate token/s | Gain | Opening-to-closing baseline drift |
|---|---|---|---|
| Dynamic v2 → maintenance-v3; 16384-token maintenance prompt, 32768 capacity | 42.80 → 60.76 | +41.98% | +3.58% |
| Dynamic v2 → balanced static KG; 249965-token engineering prompt, 262144 capacity | 42.22 → 58.58 | +38.77% | +10.24% |
| Dynamic v2 → balanced static KG; 249955-token maintenance prompt, 262144 capacity | 46.94 → 54.45 | +16.00% | +2.76% |
| Balanced static → maintenance-only static; same 249955-token maintenance prompt, 262144 capacity | 59.90 → 65.19 | +8.83% | +3.28% |
Each phase used three actual 512-token decode samples. The pooled baseline contains all six opening/closing samples. These are selected-prompt results; the percentages refer to different comparisons and must not be added or merged. Decode throughput is not end-to-end request throughput. For the first row, 512-token decode time is approximately 12.0 s → 8.4 s, excluding prefill. Cold/cached/unknown prefill states remain separate. Baseline drift is a conservative screen, not a significance test.
All 15 comparisons and negative results · CSV · Chinese full report. Several attempted changes were slower or inconclusive, including legacy handoff, threads6/10, PLE caching and CPU328 placement; they are retained rather than presented as improvements. Static profiles had a RAM and startup-time tradeoff. Three samples above 50 token/s do not prove all long prompts remain above 50.
Did answer quality get worse?
The original 50 distinct direct questions scored v2=42/50 (84%) and maintenance-v3=41/50 (82%). That is one fewer correct answer in this battery, not an established universal 2% capability loss or proof of quality equivalence. The differing question was binary_power (expected 1024); the logic category scored 4/6 in both versions. A matched repeat of graph/binary questions later reproduced the binary error in both runtimes, including v2 baselines. The original failures are preserved; causality remains unresolved.
Eight actual two-step tool tasks and eight short document version/conflict tasks each passed all three phases. One selected 249985-token supersedes document also passed. These are separate tasks, not extra repetitions added to the 50-question denominator. Gas-moles reasoning truncation and numeric-string type failures remain. Native schema format success and five-code NIAH retrieval do not prove across-task reasoning quality.
Balanced KG has not completed its own full 50-question battery. It cannot inherit maintenance-v3's quality score. The user selected maintenance-v3 for current use and retains v2 as the comparison/rollback reference. This is a speed/quality tradeoff decision, not universal promotion. Quality results and retained negatives.
Is it uncensored?
The checkpoint author labels this build Abliterated / Uncensored. This project reuses that build; it did not perform the abliteration. No additional DSH content filter was added for this deployment. The label is not a guarantee that every request will be answered, nor a guarantee of correctness. See the upstream model card.
Download, inspect and discuss
- Original weights from ghost-actual: download subject to the original model license.
- Small configuration + evidence archive: reference profile, 48-layer maintenance rank, architecture, 15 speed comparisons, quality results and 71 historical evidence documents. No model or executable runtime included.
- Individual files and evidence index with separate original/published SHA256. Host paths and process identities are redacted where present; measurements and failed results remain unchanged.
- English Community post ready to paste, Chinese post, longer English article draft, and sharing instructions.
Prior GitHub runtime/evidence links returned anonymous HTTP 404 during this release review. Evidence and configuration references are now downloadable directly here. Public runtime source and a portable installer are still missing; this archive is not complete end-to-end reproduction material.
Optimization remains paused. Deployment receipt records a point-in-time maintenance-v3 / DSH default-route check at 2026-10-02T17:48:53.905254+00:00, not a current health claim or new speed test. Automatic restart/reboot routing and unload/reload were not certified.
Credits: Qwen base model, OrcaRouter abliteration, ghost-actual EXL3 quantization, and ExLlamaV3 / the CPU-offload runtime fork. Our contribution is CPU/GPU placement experiments, static hotset evaluation and evidence reporting, distinct from upstream training, abliteration and quantization.