Qwen3.8-Flash-Next Uncensored INT8 (QEngine)
INT8-converted checkpoint for Qwen3.8-Flash-Next, derived from orcarouter/Qwen3.8-Flash-Next-Uncensored at revision da29941aee972c7eeab497e05a292c569da1f1d6.
Compatible implementation
Native INT8 inference is implemented in QEngine PR #26, on the integration branch, tested at commit bfb4ce0b8d307e8b1cb707786c028540502afcd3 (based on QEngine 1.3.5.9). Build this implementation to run these weights; an upstream release without the INT8 changes is not sufficient.
The implementation supports native group-128 INT8 routed/shared experts, body and MTP loading, W8 PLE projections, and explicit five-GPU tensor parallelism (FLASH_PAR=tp). See the branch README for build instructions and Dockerfile.flash-int8.
External dependency: the checkpoint manifest references an external INT8 n-gram table. Supply that table at the path recorded in qfn-int8-manifest.json, also accessible inside the container; downloading this repository alone does not supply it. Validation used the 51.84 GB INT8 n-gram table from the converted QEngine Flash-Next model directory.
Inference validation and performance
Measured on five RTX 3090 24 GB at 230 W/card, PCIe Gen3 (three x16 and two x8), approximately 189 GiB system RAM, CUDA 13.1.1. TP5, host-staged transport (QE_P2P=0, FLASH_COMM=auto, QE_DECODE_COMM=auto), two sequence slots, KV in RAM, 131072-token maximum context, MTP k=3, INT8 n-gram table pinned, expert-cache margin 256 MiB/card. Text-only (--vision 0); thinking disabled.
Canonical club-3090 benchmark: three warmups and five measured narrative/code runs; three measured fresh prompts per prefill depth; sampler temperature 0.6, top_p 0.95, top_k 20, min_p 0.
| Workload | Mean throughput | Mean TTFT |
|---|---|---|
| Narrative decode | 95.57 tok/s (94.10 wall tok/s) | 163 ms |
| Code decode | 54.41 tok/s (52.92 wall tok/s) | 253 ms |
| Prefill approximately 10K | 1165.97 tok/s | 8462 ms |
| Prefill approximately 90K | 1349.67 tok/s | 66685 ms |
Narrative decode CV: 3.7%; code: 10.4%. Narrative completions were 1000 tokens each; code completions were 333/480/800/414/646 tokens. Measured prefill prompts were 10003/9646/9646 and 89998/86665/93331 tokens. Concurrent streaming/nonstreaming arithmetic smoke checks passed; final session recorded 27 requests and zero errors. Physical TP5 CUDA fixtures and compute-sanitizer memcheck passed (zero reported errors).
These measurements establish working inference on this setup, not quality equivalence against BF16/INT4 or a general hardware performance guarantee. Vision, final-build P2P/copy-engine transport, and long-duration stability were not evaluated.
Quantization
The accompanying qfn-int8-manifest.json records the conversion: symmetric INT8, group size 128, nearest-even rounding, BF16 scales, zero point 0, RTN with per-group BF16 absmax divided by 127; zero groups use scale 1.
Files
model.safetensors: converted model weights (130,580,392,128 bytes).model_mtp.safetensors: converted MTP weights (2,670,277,592 bytes).config.json, tokenizer files, chat template, and image/video preprocessing configs.qfn-int8-manifest.json: source revision and conversion details.LICENSE: Qwen Community License 1.0.
Download with hf download born2bewild/Qwen3.8-Flash-Next-Uncensored-INT8-QEngine --local-dir ./Qwen3.8-Flash-Next-Uncensored-INT8-QEngine. The complete checkpoint is about 133 GB; ensure sufficient disk space.
License
Use is subject to the included Qwen Community License 1.0, including its conditions for commercial products and hosted model services. Read the license before use.
- Downloads last month
- 114
Model tree for born2bewild/Qwen3.8-Flash-Next-Uncensored-INT8-QEngine
Base model
Qwen/Qwen3.8-Flash-Next