voodooquant/Flint-DFlash2-GPTQ-INT8-PTQR

Flint-DFlash2 GPTQ INT8 PTQR

This DFlash2 draft is designed to run as the speculative-decoding companion to the matching Flint PTQR target model. It is not a standalone model.

PTQR-retrained INT8 quantization of incoai/Flint-DFlash2, the DFlash 2 draft model for RecursiveSI/flint, in the same GPTQ INT8 W8A8 GS128 format and with the same structure as the Qwen3.8-27B DFlash2 PTQR draft.

This is not a standalone language model. It is a five-layer block-diffusion drafter used by a speculative-decoding server alongside the matching target model linked above. The checkpoint is intended for the custom W8A8 serving path in curvedinf/int8-vllm.

What is PTQR?

PTQR (post-training quantization with rounding-dither retraining) fine-tunes the model weights directly under the deployed quantization: a student whose forward pass replicates the exact INT8 serving kernels (the G128 weight grid, dynamic per-token activation quantization, INT8 KV cache) is distilled against a frozen BF16 teacher, with geometrically annealed dither on each rounding decision so that early training explores neighboring grid points and late training is bit-identical to serving. The retrained weights are re-quantized and exported through the same GPTQ GS128 wrapper, so the checkpoint stays format- and config-identical to a one-shot GPTQ export.

Quantization details

Parameter	Value
Format	GPTQ (qweight/qzeros/scales, no g_idx)
Weight precision	INT8
Intended serving mode	W8A8
Group size	128 (GS128)
Symmetric	true
desc_act	false
Draft layers	5
Quantized matrices	35 attention/MLP linears
Float surfaces	the selector codebooks, draft context-KV projection and conv/state surfaces keep the trained grid in bf16 — the identical exception set as the Qwen PTQR draft
Checkpoint	151 tensors, 518 MB (bf16 source 805 MB)

All 35 packed GPTQ matrices were checked against their scale tensors; every matrix uses contiguous groups of exactly 128 weights.

Dequant audit vs the trained masters: dense linears 0.66% mean (0.70% worst), the same class as the target export; bf16-grid surfaces 0.16%; untouched tensors byte-identical to the bf16 draft.

Fidelity gate: speculative-decoding acceptance is unchanged by quantization — 2.25 with this draft versus 2.27 for the bf16 drafter (TP4/C8, int8 KV, NS=7, random 32/128), with identical TPOT (30.0 ms).

Serving

Use this draft with the linked PTQR target model and a DFlash2-capable build of the custom vLLM fork:

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
VLLM_GFX908_INT8_LM_HEAD=1 \
VLLM_GFX908_ACT_QUANT=round \
VLLM_GFX908_DECODE_BLOCKSCALE=1 \
VLLM_G8_INT8_DOT=1 VLLM_G8_INT8_PV=1 \
VLLM_UA_PREFILL_BLOCKM=128 \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve voodooquant/Flint-GPTQ-INT8-PTQR \
  --tensor-parallel-size 4 \
  --max-num-seqs 12 \
  --dtype bfloat16 \
  --max-model-len 524288 \
  --kv-cache-dtype int8_block_g128 \
  --speculative-config '{
    "method": "dflash",
    "model": "voodooquant/Flint-DFlash2-GPTQ-INT8-PTQR",
    "num_speculative_tokens": 7,
    "kv_cache_dtype": "int8_block_g128"
  }'

DFlash 2 drafts candidate tokens; the target model verifies them, so speculative decoding preserves the target model's output distribution. The top-level KV flag configures the target and the nested field configures this draft; both use INT8 block KV with 128-wide groups. This is one fixed TP4/C12 contract: AITER W8A8 INT8 GEMMs at every M, AITER unified attention, vLLM custom all-reduce, and the two linked GS128 checkpoints. --dtype bfloat16 is the residual dtype; the AITER GEMM path dynamically quantizes its inputs to INT8.

The draft consumes EAGLE3 auxiliary hidden states from target layers 6, 16, 27, 37 and 47 (0-based 5, 15, 26, 36, 46; dflash_config.target_layer_ids), uses block size 8, a selector of rank 256 and top-k 16, and a 2048-token sliding window.

Source

• DFlash 2 draft model: incoai/Flint-DFlash2
• Target model: RecursiveSI/flint
• Custom serving stack: curvedinf/int8-vllm — see docs/recipes/flint.md
• PTQR export and training: scripts/ptqr_export_flint_draft.py, scripts/ptqr_train_draft_flint_ptqr.py