romanerm/flint-GGUF

Flint — GGUF (IQ4_XS / IQ4_NL / Q4_K_M, imatrix)

Quantized GGUFs of RecursiveSI/flint — Recursive's 39.8B mixture-of-experts coding-agent model (~2.5B active parameters per token, 256 routed experts, 49 layers, 262k context).

All three types were produced from the same BF16 checkpoint with a single shared importance matrix, and evaluated on the same text with the same chunking, so the numbers below are directly comparable to each other.

Files

File	Type	Size	BPW	Notes
flint-IQ4_XS.gguf	IQ4_XS	21.28 GB (19.81 GiB)	4.25	smallest; 24 GB cards with partial offload
flint-IQ4_NL.gguf	IQ4_NL	22.51 GB (20.96 GiB)	4.50	fastest generation of the three
flint-Q4_K_M.gguf	Q4_K_M	24.12 GB (22.46 GiB)	4.85	best quality of the three
flint.imatrix.gguf	importance matrix	430 MB	—	reuse it to make your own quants

All files are single-file GGUF with tokenizer, chat template and original metadata embedded. For reference: the BF16 base checkpoint is 79.5 GB (16.0 BPW).

Quality: perplexity and KL-divergence

Reference text: WikiText-2 raw test split, 64 chunks × 512 tokens = 32,768 scored tokens, context 512, llama.cpp b11815. Base log-probabilities were saved once from the BF16 checkpoint (--save-all-logits) and every quant was measured against them (--kl-divergence), so BF16 and all quants share identical tokens and chunking.

Model	Mean PPL	ΔPPL vs BF16	Mean KLD	Median KLD	Same top-p
BF16 (base)	21.0072 ± 0.6319	—	—	—	100 %
Q4_K_M	22.5250 ± 0.7028	+7.22 %	0.2585 ± 0.0067	—	81.38 ± 0.31 %
IQ4_NL	23.5211 ± 0.7461	+11.97 %	0.3291 ± 0.0070	0.1149	78.19 ± 0.32 %
IQ4_XS	23.7839 ± 0.7558	+13.22 %	0.3442 ± 0.0070	—	78.03 ± 0.32 %

Additional details for the two files where the full distribution was recorded:

Metric	IQ4_NL	IQ4_XS
ln(PPL(Q)/PPL(base))	0.1130 ± 0.0095	0.1184 ± 0.0110
90th percentile KLD	0.7003	—
99th percentile KLD	3.6939	—
99.9th percentile KLD	11.6462	—
RMS Δp	13.32 ± 0.20 %	—
Mean Δp	−0.597 ± 0.104 %	—

How to read it:

• Q4_K_M is clearly the best of the three here: 7.2 % perplexity overhead and the highest same-top-p agreement, at +2.8 GB over IQ4_XS.
• IQ4_NL and IQ4_XS are within noise of each other on this evaluation (Δ mean KLD 0.015, Δ PPL 0.26 against per-chunk CIs of ±0.7). IQ4_XS is smaller (21.3 vs 22.5 GB) and quantizes the same checkpoint with a different block structure (256 with 6-bit scales vs 32 with one scale), but on this 32k-token budget it did not come out ahead. Do not read the ordering as a property of the formats in general.
• For all types the KL is concentrated in the tail: median KLD is 0.11 while the 99.9th percentile reaches 11.6 — most tokens are essentially unaffected, a small set of high-surprise tokens diverges.
• The evaluation budget is modest (32k tokens, single run). Second decimal of the mean and percentiles beyond the 99.9th are noisy.

Speed

llama-bench, AMD Radeon AI PRO R9700 (gfx1201, 32 GB), ROCm, full offload (-ngl 99), -t 16, 3 repetitions:

Type	pp512 (tok/s)	tg128 (tok/s)
IQ4_XS	4812.02 ± 246.57	95.79 ± 1.64
IQ4_NL	4733.75 ± 224.47	99.70 ± 1.42
Q4_K_M	4450.02 ± 184.71	97.79 ± 1.76

Prompt processing and generation differ by only a few percent between the three — pick on size/quality, not speed.

Which file should I take?

• Q4_K_M — default choice. Fits a 32 GB card with short-to-medium context, best measured quality.
• IQ4_NL — if you need to shave ~1.6 GB and want the fastest generation; quality is close to IQ4_XS, speed slightly ahead.
• IQ4_XS — smallest 4-bit file; the best pick when memory is the hard constraint (24 GB cards with partial offload, or tight unified-memory devices).

Requirements

llama.cpp build b11815 or newer is required. The flint architecture landed upstream in October 2026 (commit a37cc46104). Older builds fail with unknown model architecture: 'flint'.

• llama.cpp b11815+ (ROCm, CUDA, Vulkan, Metal or CPU)
• VRAM: ~21–24 GB of weights; full offload needs a 32 GB card at short context, 24 GB cards need -ngl tuning
• The model's documented context is 262,144 tokens (prompt + output); KV cache grows accordingly

Usage

# single prompt
llama-cli -m flint-Q4_K_M.gguf -ngl 99 -c 32768 -t 16 \
  --temp 1.0 --top-p 0.95 --top-k 20

# OpenAI-compatible server
llama-server -m flint-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 32768 --n-gpu-layers 99 --threads 16 \
  --temp 1.0 --top-p 0.95 --top-k 20 --jinja

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="unused")
r = client.chat.completions.create(
    model="flint-Q4_K_M.gguf",
    messages=[{"role": "user", "content": "Write a Go function that reverses a UTF-8 string."}],
    max_tokens=32768,
)
print(r.choices[0].message.content)

Sampling (from the base model card)

Setting	Value
Temperature	1.0
Top-p / top-k	0.95 / 20
Output budget for hard problems	at least 32,768 tokens, including reasoning

Flint is a reasoning model: it emits a <think> block before the answer. Preserve the reasoning content in multi-turn history and give it a generous output budget, otherwise generation can stop mid-reasoning.

Quantization details

Parameter	Value
Base checkpoint	RecursiveSI/flint BF16, 3 shards, 79.5 GB
Quantizer	llama.cpp b11815, llama-quantize --imatrix
Threads	16 (quantization is CPU-side in llama.cpp; there is no GPU GGUF quantizer)
Time per type	~12–13 min
Importance matrix	64 chunks × 512 tokens, -ngl 16, 531 entries, 430 MB
Calibration corpus	15.9 MB (~4.4M tokens): Go source after gofmt / golangci-lint fmt, plus agent-harness documentation
Calibration PPL	4.2659

The imatrix is used by all three quantizers in current llama.cpp (quantize_iq4_nl, quantize_iq4_xs and the K-quant functions take per-channel weights), so the same file improves every type listed here.

Rare routed experts receive only partial coverage (save_imatrix: entry 'blk.*.ffn_*_exps.weight' has partial data (99.2–99.6%)). That is expected for a 256-expert MoE with a 32k-token calibration set and does not prevent the file from being useful.

Reproducing

# importance matrix (llama.cpp b11815+)
llama-imatrix -m flint-BF16-00001-of-00003.gguf -f corpus.txt -o flint.imatrix.gguf \
  -c 512 -ngl 16 -t 32 --chunks 64

# any of the three quant types
llama-quantize --imatrix flint.imatrix.gguf flint-BF16-00001-of-00003.gguf \
  flint-Q4_K_M.gguf Q4_K_M 16

# evaluation: save base logits once, then measure each quant against them
llama-perplexity -m flint-BF16-00001-of-00003.gguf -f wiki.test.raw \
  -c 512 --chunks 64 --save-all-logits wikitext-bf16.logits -ngl 16 -t 16

llama-perplexity -m flint-Q4_K_M.gguf -f wiki.test.raw \
  -c 512 --chunks 64 --kl-divergence --kl-divergence-base wikitext-bf16.logits -ngl 99 -t 16

Evaluation runs must share -f, -c and --chunks — the logits file stores the token stream and the run aborts if it does not match.

Notes and caveats

• llama-cli / llama-completion echo the prompt and continue generation; the model starts with <think>, so a small -n returns reasoning only.
• These are plain single-type quants. No per-tensor or per-layer type mixing was applied beyond what llama-quantize does by default (e.g. Q4_K_M keeps some tensors at higher precision, IQ4_NL/IQ4_XS promote a few tensors to Q5_K/Q6_K).
• Quantized on Windows with a ROCm build of llama.cpp; the GGUF files themselves are platform-independent.
• Agentic benchmarks (Terminal-Bench, SWE-bench) were not re-run on these files — only perplexity, KL-divergence and throughput.

Limitations

Inherited from RecursiveSI/flint: the model can produce incorrect or harmful content, buggy code and unsafe tool calls. Test generated code and review consequential changes before applying them. Tool-using applications must enforce their own permission and execution boundaries. Quantization adds approximation on top of the base model's behaviour; the effect is largest on rare experts and high-surprise tokens, so validate the specific file on your own workload before deploying it.

License and attribution

• Base model: RecursiveSI/flint by Recursive — Apache 2.0.
• These quantizations: distributed under the same Apache 2.0 license with attribution to the original authors. Architecture support, training data and evaluation methodology belong to Recursive; this repository only provides GGUF conversions.