terelabs/flint-GGUF

Flint GGUF (Tere Labs)

imatrix GGUF quantizations of RecursiveSI/flint, Recursive's 39.8B-A2.5B mixture-of-experts coding model, made by Tere Labs. This is an independent community quantization, not an official Recursive release.

Three files, one imatrix, and every number below was measured with the same method on the same machine against the original BF16 weights.

Files

File	Bytes	GB	GiB	bpw	KLD mean vs BF16	top-1 agreement
flint-Q5_K_M.gguf	28,976,821,344	28.98	26.99	5.83	0.369	87.3%
flint-Q4_K_M.gguf	25,017,398,368	25.02	23.30	5.03	0.440	85.8%
flint-IQ4_XS.gguf	22,242,866,272	22.24	20.72	4.47	0.473	85.3%
imatrix.gguf	450,750,144	0.45

Memory guide (file size plus KV cache): IQ4_XS fits a 24 GB GPU with a small context or partial offload; Q4_K_M and Q5_K_M are meant for 32 GB+ of VRAM or GPU+RAM split.

Running

Flint support in llama.cpp lives in PR #30928. A build from main without that PR will not load the model. These files were built and tested with that PR at commit 1b44e30.

llama-server -hf terelabs/flint-GGUF:Q4_K_M --jinja -c 32768
llama-cli    -hf terelabs/flint-GGUF:IQ4_XS --jinja

Flint has no BOS token (add_bos_token: false); llama.cpp picks this up from the GGUF metadata.

Chat template that llama.cpp can parse

The chat template shipped in the original checkpoint contains a replace("\x00", "") filter. llama.cpp's Jinja engine does not understand the \x escape, so llama-server --jinja fails to load that template; without --jinja the server falls back to a generic chat format and Flint's tool-call syntax is lost.

The template embedded in these files has that one filter removed and nothing else changed. Rendering a dialogue with tools through it is byte-identical to HF transformers with the original template. The same template is also provided as chat_template_llamacpp.jinja if you want to pass it explicitly with --chat-template-file.

Verified on llama.cpp PR #30928 @ 1b44e30. If llama.cpp later learns \x escapes, this difference disappears.

Recipe

llama-quantize --imatrix imatrix.gguf with base types Q5_K_M / Q4_K_M / IQ4_XS and these overrides:

• token_embd and output: Q8_0
• all attention (attn_q, attn_k, attn_v, attn_output): Q8_0
• dense FFN of the first layers (ffn_gate, ffn_up, ffn_down): Q8_0
• shared expert (*_shexp): Q8_0
• routed experts: the file's base type. This is almost all of the model's weight.

The trade-off is deliberate: everything that runs on every token is stored nearly lossless, and the routed experts carry the compression. Our Q4_K_M is 0.85 GB larger than the lmstudio Q4_K_M for this reason.

imatrix

About 890k tokens, ctx 512, 1732 chunks:

• ~300k tokens of tool-calling dialogues from NousResearch/hermes-function-calling-v1 (Apache-2.0), rendered through Flint's own chat template so the calibration sees exactly the tokens the model generates: <tool_call>name<arg_key>…</arg_key><arg_value>…</arg_value></tool_call>
• ~500k tokens of code from six permissive repositories at pinned commits: requests, flask, express, ripgrep, gin, nlohmann/json
• ~150k tokens of prose from wikitext

No overlap with the KLD evaluation text or with BFCL. Expert coverage: in some layers one expert out of 256 received no calibration data ("partial data 99.61%"); llama.cpp uses default values for it.

Checks before quantization

• Tokenizer: llama.cpp matches the HF tokenizer on code (3k and 60k characters, 13,359 tokens) and on a dialogue with tools.
• Weights: the BF16 converted from the final release is byte-identical to unsloth's BF16 (built from flint-preview) on every tensor checked. Comparisons with unsloth files below are therefore comparisons of quantization, not of weights.
• BOS forced off for every file in every measurement.

Comparison with other Flint GGUFs

All rows measured by us against the same BF16 reference. Lower KLD is better; higher agreement is better.

File	GB	KLD mean	KLD median	KLD p99	top-1	same tool call as BF16	BFCL correct
BF16 (reference)	79.63	0	0	0	100%	100%	79.7%
ours Q5_K_M	28.98	0.369	0.0072	7.18	87.3%	78.7%	81.0%
official int4 Q4_1 (QAT)	25.99	0.675	0.0342	9.81	80.4%	77.0%	81.0%
unsloth UD-Q4_K_XL	25.05	0.389	0.0098	7.24	86.5%	79.7%	80.0%
ours Q4_K_M	25.02	0.440	0.0116	8.11	85.8%	80.0%	78.3%
lmstudio Q4_K_M	24.17	0.668	0.0273	10.39	82.1%	76.0%	80.3%
ours IQ4_XS	22.24	0.473	0.0134	8.62	85.3%	78.3%	78.7%

Measurement error: KLD mean ±0.007–0.010; top-1 about ±0.2 pp; tool-call and BFCL ±4.5 pp (95% CI, 300 tasks).

What the table supports:

• IQ4_XS is the smallest published Flint GGUF (22.24 GB): 1.9 GB smaller than lmstudio Q4_K_M with 29% lower KLD (0.473 vs 0.668) and higher top-1 (85.3% vs 82.1%); 3.75 GB smaller than the official int4.
• Q4_K_M is 1.5x closer to BF16 than the lmstudio Q4_K_M of the same name, for 3.5% more size (KLD 0.440 vs 0.668).
• Q4_K_M is at the top of the 4-bit group on reproducing BF16's tool calls: 80.0% vs 79.7% (unsloth), 77.0% (official) and 76.0% (lmstudio). The gap to unsloth is within noise. unsloth UD-Q4_K_XL and our Q4_K_M are the two strongest 4-bit files: ours leads on tool-call agreement, theirs on KLD.
• Q5_K_M is the closest file to BF16 of everything measured: KLD 0.369, median 0.0072, top-1 87.3%, at 36% of the BF16 size. On the tool-call bench it lands at 78.7% same-call agreement and 81.0% BFCL correct, both within the ±4.5 pp noise of the 4-bit group: the extra bits show up in the logits, not in this 300-task bench.
• All our files are closer to the original BF16 than the official int4. The official file comes from quantization-aware training, so its distance from BF16 is by design; this is a statement about closeness to BF16, not about which is better.
• BFCL accuracy is statistically indistinguishable from BF16 for every file, including the other publishers' files: everything lands in 78–81% and no pairwise difference is significant (exact McNemar, p 0.29–1.0; all 1.0 after Bonferroni). 73 of the 300 tasks are unstable under greedy decoding for BF16 itself.
• Only these files work with llama-server --jinja out of the box. The official, unsloth and lmstudio GGUFs embed the original template with the \x00 filter described above.

Method

KLD and top-1:

• llama-perplexity --kl-divergence against BF16 logits
• text: code from four other permissive repositories (httpx, fd, cobra, axios), disjoint from the calibration set
• ctx 2048, 40 chunks; BF16 perplexity on this text is 7.737

Tool calling:

• 300 BFCL v3 tasks (simple, multiple, parallel, parallel_multiple, live_simple, live_multiple, irrelevance, live_irrelevance)
• prompts rendered with Flint's official chat template; greedy, up to 4096 tokens, one server slot, BOS off. A repeat BF16 run matched 100%.
• "same tool call": function name and all arguments match BF16 (call order ignored)
• "BFCL correct": a simplified BFCL AST checker against the reference answers

Per-file raw numbers (31 columns, including per-category BFCL accuracy) are in comparison.csv in this repository.

Changes from the original (Apache-2.0, section 4b)

These files are derivatives of RecursiveSI/flint (Apache-2.0, copyright Recursive). Changes made by Tere Labs:

1. converted from safetensors to GGUF;
2. weights quantized with llama.cpp using the imatrix and recipe above;
3. embedded chat template modified to remove the replace("\x00", "") filter that llama.cpp's Jinja engine cannot parse.

The license text is in LICENSE. "Flint" and "Recursive" are names of Recursive and its model; this repository is not affiliated with or endorsed by Recursive.