Qwen3.8-Flash-Next Uncensored INT8 (QEngine)

INT8-converted checkpoint for Qwen3.8-Flash-Next, derived from orcarouter/Qwen3.8-Flash-Next-Uncensored at revision da29941aee972c7eeab497e05a292c569da1f1d6.

Compatible implementation

Native INT8 inference is implemented in QEngine PR #26, on the integration branch, tested at commit bfb4ce0b8d307e8b1cb707786c028540502afcd3 (based on QEngine 1.3.5.9). Build this implementation to run these weights; an upstream release without the INT8 changes is not sufficient.

The implementation supports native group-128 INT8 routed/shared experts, body and MTP loading, W8 PLE projections, and explicit five-GPU tensor parallelism (FLASH_PAR=tp). See the branch README for build instructions and Dockerfile.flash-int8.

External dependency: the checkpoint manifest references an external INT8 n-gram table. Supply that table at the path recorded in qfn-int8-manifest.json, also accessible inside the container; downloading this repository alone does not supply it. Validation used the 51.84 GB INT8 n-gram table from the converted QEngine Flash-Next model directory.

Inference validation and performance

Measured on five RTX 3090 24 GB at 230 W/card, PCIe Gen3 (three x16 and two x8), approximately 189 GiB system RAM, CUDA 13.1.1. TP5, host-staged transport (QE_P2P=0, FLASH_COMM=auto, QE_DECODE_COMM=auto), two sequence slots, KV in RAM, 131072-token maximum context, MTP k=3, INT8 n-gram table pinned, expert-cache margin 256 MiB/card. Text-only (--vision 0); thinking disabled.

Canonical club-3090 benchmark: three warmups and five measured narrative/code runs; three measured fresh prompts per prefill depth; sampler temperature 0.6, top_p 0.95, top_k 20, min_p 0.

Workload Mean throughput Mean TTFT
Narrative decode 95.57 tok/s (94.10 wall tok/s) 163 ms
Code decode 54.41 tok/s (52.92 wall tok/s) 253 ms
Prefill approximately 10K 1165.97 tok/s 8462 ms
Prefill approximately 90K 1349.67 tok/s 66685 ms

Narrative decode CV: 3.7%; code: 10.4%. Narrative completions were 1000 tokens each; code completions were 333/480/800/414/646 tokens. Measured prefill prompts were 10003/9646/9646 and 89998/86665/93331 tokens. Concurrent streaming/nonstreaming arithmetic smoke checks passed; final session recorded 27 requests and zero errors. Physical TP5 CUDA fixtures and compute-sanitizer memcheck passed (zero reported errors).

These measurements establish working inference on this setup, not quality equivalence against BF16/INT4 or a general hardware performance guarantee. Vision, final-build P2P/copy-engine transport, and long-duration stability were not evaluated.

Quantization

The accompanying qfn-int8-manifest.json records the conversion: symmetric INT8, group size 128, nearest-even rounding, BF16 scales, zero point 0, RTN with per-group BF16 absmax divided by 127; zero groups use scale 1.

Files

  • model.safetensors: converted model weights (130,580,392,128 bytes).
  • model_mtp.safetensors: converted MTP weights (2,670,277,592 bytes).
  • config.json, tokenizer files, chat template, and image/video preprocessing configs.
  • qfn-int8-manifest.json: source revision and conversion details.
  • LICENSE: Qwen Community License 1.0.

Download with hf download born2bewild/Qwen3.8-Flash-Next-Uncensored-INT8-QEngine --local-dir ./Qwen3.8-Flash-Next-Uncensored-INT8-QEngine. The complete checkpoint is about 133 GB; ensure sufficient disk space.

License

Use is subject to the included Qwen Community License 1.0, including its conditions for commercial products and hosted model services. Read the license before use.

Downloads last month
114
Safetensors
Model size
126B params
Tensor type
BF16
·
I32
·
I64
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for born2bewild/Qwen3.8-Flash-Next-Uncensored-INT8-QEngine

Quantized
(55)
this model