Flint-AWQ (INT4 W4A16, g64, calibrated)

4-bit AWQ-format export of RecursiveSI/flint (39.8B total / ~2.5B active MoE for coding agents) built on 2×RTX 3090 (Ampere, sm_86) for single-user 256K-context serving under SGLang with the AWQ-Marlin fused-MoE kernels. Text in / text out; reasoning (reasoning_content) and multi-turn tool calls are preserved and probe-verified. Weights are 20.9 GiB (10.6 GB per card at TP=2), leaving a 589K-token full-attention KV pool at --context-length 262144.

Revision 2 (2026-10-22, current): calibrated build. Revision 1 was a data-free round-to-nearest export of the same shapes; it looped in long agentic turns (see below) and is superseded — same file names, same bytes per tensor, pull the current revision.

How it was made

transformers has no Flint modeling file, so the usual llmcompressor AWQ/GPTQ path does not exist for this architecture, and the vendor's MLX int4 export carries a free per-group bias that cannot be re-encoded as AWQ (AWQ zero-points are integers on the quantization grid). The checkpoint is therefore produced by a self-contained calibrator that runs the model's own reference forward, scripts/quantize/quantize_flint_awq_calib.py (~13 GPU-minutes on one 3090, layer by layer against the 79.6 GB BF16 base):

• Calibration set: 96 × 2,048 tokens (138,680 tokens) — code reasoning traces (AM-Thinking, rendered through the Flint chat template), raw code (the-stack-smol, 5 languages), an opencode agentic session, SWE-bench problem statements, Open-Platypus.
• AWQ scale search (α grid on the activation-magnitude heuristic, per projection family) folded into the preceding norms / router / up_proj rows, so the served tensors are standard AWQ (qweight / qzeros / scales), group size 64, asymmetric on the integer-zero-point grid.
• GPTQ error feedback on the activation Grams of that forward. The always-active path (attention, the shared expert, the two dense MLPs) sees every calibration token and gains 20–36 % held-out error over the clipped round-to-nearest. A routed expert sees only 0.5–12K rows for a 2,048-dim Hessian, and GPTQ at the standard 1 % damping fits the noise (expert down_proj came out worse than the clip) — so each expert's Gram is shrunk toward the layer-wide Gram and the thin down_proj Hessians are damped hard (--expert-gram-shrink 1 --expert-percdamp 0.1 --expert-down-percdamp 1). Receipt and probe in the repo.
• Kept in BF16: lm_head, embed_tokens, every norm, the MoE router mlp.gate, the per-head attention sinks (softmax_offset). Every expert, the shared expert, the dense MLPs and all attention projections are INT4.
• calibration-log.json ships alongside the weights (per-layer AWQ α / loss ratios, sample mix, timings).

Fidelity to the BF16 base, measured with a serving-independent fp32 reference forward over the first 2,048 tokens of a real 15K-token agentic prefix (next-token distribution vs the base):

build	top-1 agreement	mean KL	runaway-repetition replays (68K poisoned prefix N=24 / 15K prefix N=48)
revision 1 (data-free RTN)	64.4 %	0.727	11/24, 1–3/48
revision 2 (this checkpoint)	76.4 %	0.331	0/24, 0/48

Scale audit: 0 of 36,439 scales / qweight tensors flagged (no zero / NaN / Inf / extreme scales; dead-channel comparison against the BF16 base).

Validation (SGLang v0.5.22 + local patches, TP=2, RTX 3090 ×2, --context-length 262144)

Probe	Result
Thinking / tool-call (non-stream, streaming, multi-turn) / developer role	pass
Needle-in-haystack, natural-text haystack, greedy	found at server-verified 131,041 (depths 0.5 / 0.9) and 243,522 (0.5 / 0.9) prompt tokens
Needle, repeated-sentence filler (The quick brown fox… × 29K)	fails from 131K@0.5 — a NoPE repetition attractor, not a quantization or serving defect (reproduced greedy, cold-cache, and on an attention-BF16 variant); do not evaluate Flint with a repeated-sentence haystack
TP=2 vs TP=1	final answers byte-identical on 5/6 greedy prompts, identical tool call on the sixth
AWQ scale integrity	0 / 36,439 flagged

Single-user decode (M=1, 100 output tokens, fresh prefill, bf16 KV, --triton-attention-num-kv-splits 64):

Prompt tokens (server-verified)	tok/s	TPOT
1,024	164.2	6.1 ms
32,768	148.1	6.8 ms
261,916	89.4	11.2 ms

Receipts: benchmarks/quality/flint/ (runaway-repetition root cause + calibrated ladder, needle discriminator, TP check, tool-schema probe), benchmarks/regression/flint-2026-10-22-gptq.json. A full-300 SWE-bench Lite cell under the isolated opencode harness is queued; the number lands in the repo README.

Serving

Flint needs the vendor's flint_sglang model package (learned sinks, NoPE full-attention layers, sliding-window KV pool, Flint reasoning / tool-call parsers and grammar backend). The 3090 repo ships a v0.5.22 port under plugins/flint_sglang; scripts/launch.sh flint is the tuned preset. The load-bearing flags:

SGLANG_EXTERNAL_MODEL_PACKAGE=flint_sglang FLASHINFER_USE_CUDA_NORM=1 FLINT_STRICT_TOOLS=1 \
python -m sglang.launch_server --model-path mattbucci/Flint-AWQ \
  --tp 2 --context-length 262144 --max-running-requests 1 \
  --attention-backend triton --kv-cache-dtype auto --triton-attention-num-kv-splits 64 \
  --reasoning-parser flint --tool-call-parser flint --grammar-backend flint \
  --sampling-defaults model --chunked-prefill-size 8192 --mem-fraction-static 0.85 --enable-fp32-lm-head

• MoE runner must stay auto — the AWQ-Marlin fused-MoE path asserts it; the vendor's --moe-runner-backend triton is for BF16/FP8 weights.
• Triton attention is the only Ampere backend with learned sinks + the SWA KV pool; it rejects FP8 E4M3 KV on sm_86, so KV stays bf16 (14 full-attention layers × 4 KV heads make it cheap: 28 KB/token). --triton-attention-num-kv-splits 64 matters at M=1 / TP=2 (2 KV heads per rank): the default 8 leaves deep decode at 36 tok/s.
• Tool-call arguments: the model ignores JSON-schema required keys on its own (opencode's bash without description, read with path instead of filePath); FLINT_STRICT_TOOLS=1 makes the plugin's grammar enforce each tool's argument schema (20/20 valid vs 1/24 without).
• Small-context boots: keep swa_layer_tokens ≥ chunked_prefill_size + 513 (raise --swa-full-tokens-ratio), or a prompt longer than the SWA pool stalls forever at 0 % GPU. The 262K ship (29K SWA pool) is unaffected.
• Sampling defaults come from generation_config.json (T 1.0 / top-p 0.95 / top-k 20); give hard problems ≥ 32K output tokens including reasoning, and keep reasoning_content in the conversation history.

License

Apache 2.0, inherited from the base model.