voodooquant/Flint-GPTQ-INT8-PTQR

Flint GPTQ INT8 PTQR

This target model is designed to run with the matching Flint-DFlash2 draft model for speculative decoding.

PTQR-retrained quantization of RecursiveSI/flint, produced and validated on 4x AMD Instinct MI100 (gfx908 / CDNA1) over XGMI. It uses the same GPTQ INT8 W8A8 GS128 checkpoint format and serving methodology as Qwen3.8-27B PTQR: every dense and expert GEMM is true int8xint8 (W8A8), KV cache is int8, and there is no 16-bit fallback in the serving recipe.

What is PTQR?

PTQR (post-training quantization with rounding-dither retraining) fine-tunes the model weights directly under the deployed quantization: a student whose forward pass replicates the exact INT8 serving kernels (the G128 weight grid, dynamic per-token activation quantization, INT8 KV cache) is distilled against a frozen BF16 teacher, with geometrically annealed dither on each rounding decision so that early training explores neighboring grid points and late training is bit-identical to serving. The retrained weights are re-quantized and exported through the same GPTQ GS128 wrapper, so the checkpoint stays format- and config-identical to a one-shot GPTQ export.

This release is rung 1 at step 40 (r1s40): 60 steps, lr 1e-5, weights-only SGD with the group scales trained alongside the weights, on serving-shaped windows (2048-4096 tokens). The corpus was extended with eight 512k-token needle documents (7.07M tokens total) so that long-context behaviour is trained, not just extrapolated: the checkpoint natively documents a 262,144 context and is served to 512k by extending the rope tables to the runtime length.

Quantization details

Parameter	Value
Format	GPTQ (qweight/qzeros/scales, no g_idx)
Bits	8
Group size	128
Symmetric	true
desc_act	false
lm_head	false (untied; the serving stack runs a separate int8 per-channel head)
Quantizer	PTQR retraining on the GPTQ GS128 grid (scripts/ptqr_export_flint.py)
Checkpoint	35 shards, 36,439 tensors, 38.5 GiB

Why group_size 128: the accompanying runtime dispatches AITER true int8xint8 compute (W8A8) for every decode and prefill GEMM shape, which requires one weight scale per 128-wide K block.

Quality was verified at the serving gate (teacher-forced NLL and KL-divergence probes, speculative-decoding acceptance). Measured at TP4/C8 with int8 KV:

Gate	Value
Serving NLL	4.6588 @64 tokens, 1.3898 @1024 (bf16 reference 4.6590 / 1.39, teacher 4.6586)
KLD (1024-token val window)	0.0095 (bf16 floor 0.0055); 0.0103 with int8 KV; 0.0110 with int8 KV + W8A8 experts + graphs
Draft acceptance (paired with the DFlash2 draft, NS=7)	2.25 (bf16 draft 2.27)
Throughput, C8, 32 in / 256 out, 64 requests	261.1 output tok/s, TTFT 627 ms, TPOT 27.2 ms

On the current recipe (TP4, C12, NS=7, 512k max length) the same pair measures 532/535 tok/s on a controlled 12x256 shared-prefix load, a 26.4-27.3 ms single-stream step, and 8.05 s cold TTFT for a 42k-token prompt.

Serving (vLLM, gfx908 fork)

Use with the companion fork stack:

• vLLM: curvedinf/int8-vllm — branch main (AITER W8A8 INT8 GEMMs everywhere, AITER unified attention with int8 QK/PV dots, INT8 KV, vLLM custom all-reduce)
• AITER: curvedinf/int8-aiter — branch main (gfx908 kernels and tuning)

VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
VLLM_GFX908_INT8_LM_HEAD=1 \
VLLM_GFX908_ACT_QUANT=round \
VLLM_GFX908_DECODE_BLOCKSCALE=1 \
VLLM_GFX908_O_PROJ_I8=1 \
VLLM_G8_INT8_DOT=1 VLLM_G8_INT8_PV=1 \
VLLM_UA_PREFILL_BLOCKM=128 \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve voodooquant/Flint-GPTQ-INT8-PTQR \
  --tensor-parallel-size 4 \
  --max-num-seqs 12 \
  --dtype bfloat16 \
  --max-model-len 524288 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype int8_block_g128 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser glm47 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[16,32,48,64,96,128]}' \
  --speculative-config '{
    "method": "dflash",
    "model": "voodooquant/Flint-DFlash2-GPTQ-INT8-PTQR",
    "num_speculative_tokens": 7,
    "kv_cache_dtype": "int8_block_g128"
  }'

This is one fixed TP4/C12 contract: both GS128 models, AITER W8A8 INT8 GEMMs at every M, AITER unified attention, vLLM custom all-reduce, and INT8 target/draft KV. --dtype bfloat16 is the model's residual/native dtype; GEMM inputs are dynamically quantized to INT8 by the AITER W8A8 path. The top-level KV flag configures this target, the nested field configures the DFlash2 draft.

The full recipe (KV-tier settings, extra envs, and the exact systemd unit) lives in scripts/serve_recipe_flint.sh in the fork. Do not substitute W8A16, float KV, another TP size, or no speculation in the intended recipe.

Model architecture

Flint is a sparse MoE language model (model_type flint): 49 layers — layers 0-1 dense, layers 2-48 MoE with 256 routed experts top-6 plus a shared expert (intermediate 1024) — hidden 2048, MoE intermediate 512, dense intermediate 6144, 32 attention heads / 4 KV heads, head_dim 128, sliding-window attention of 513 on 35 of the 49 layers, vocab 150,144, native context 262,144, released and served to 512k.

Reproduction

The PTQR trainer, the export vehicle, and the full experiment ledger (KLD gates, acceptance batteries, A/B measurements) live in the vLLM fork: curvedinf/int8-vllm — see docs/recipes/flint.md and docs/recipes/surface_experiments_ledger.jsonl.