AtomicChat/Flint-GGUF

How to Run Flint Locally

Built from RecursiveSI's original weights with our own importance matrix. The calibration corpora behind our builds are public.

• Flint is RecursiveSI's 39.8B mixture-of-experts coding model. About 2.5B parameters are active per token.
• See our measurements below, including how these files compare to other builds of Flint.

---

Pick a file

Every number below is measured, not estimated. How we measured it is at the bottom, and the raw logs are in the metrics repo so you can check any of it yourself.

KL divergence is how far the quantized model's predictions drift from the original weights. Lower is better, and zero means identical. top-1 is how often it picks the same next word the original would have picked. We measure on two held-out texts: general text in 30 languages, and code.

File	Size on disk	Bits per weight	KL divergence	top-1	KL divergence, code	top-1, code
Flint-AD-Q8_0.gguf	42.9 GB	8.63	0.0266	95.19%	0.156	92.32%
Flint-AD-Q6_K.gguf	34.7 GB	6.98	0.0314	94.67%	0.166	91.81%
Flint-AD-IQ4_XS.gguf	24.5 GB	4.93	0.0610	91.71%	0.212	89.32%
Flint-AD-IQ3_S.gguf	19.1 GB	3.84	0.1807	85.37%	0.403	84.07%
Flint-AD-IQ2_M.gguf	16.4 GB	3.29	0.3084	80.53%	0.525	80.77%
Flint-AD-IQ2_XXS.gguf	13.3 GB	2.67	0.5492	73.38%	0.738	75.74%

Flint is unusually sensitive to quantization. Even an 8-bit copy matches the original's next word only about 95% of the time on general text, where most models reach 98% or more. Read the table against that ceiling, and against the other builds below, which were measured the same way.

AD- marks an Atomic Dynamic layout. The name gives the type of the routed experts, which hold almost all of the weights. Attention, the shared expert and the two dense layers stay at q8_0. The output head is f16, and the token embeddings are q8_0 (f16 in AD-Q6_K and AD-Q8_0).

AD-IQ4_XS was rebuilt on 23 October. Flint's early blocks turned out to be its most sensitive, so the experts of MoE blocks 2 to 19 are now q5_k and the rest iq4_xs, and the output head is q8_0. At the same size it has 11% lower KL divergence than the first build, on both texts.

Which one fits your card

Flint has full attention in only 14 of its 49 layers. The other 35 use a sliding window of 513 tokens. The cache is therefore small: 28 KB per token, which is 0.9 GB at 32k context and 7.5 GB at the full 262k.

Your card	File	Leaves room for
16 GB	AD-IQ2_XXS	around 95k context
24 GB	AD-IQ3_S	around 190k context
32 GB	AD-IQ4_XS	full context
48 GB and up	AD-Q8_0	full context

If you are choosing between two neighbouring files, take the larger one.

Running it

llama-server -m Flint-AD-IQ4_XS.gguf -ngl 99 -c 32768 --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20

The sampling settings are the ones RecursiveSI ships in generation_config.json. The chat template is inside every file, and llama.cpp applies it on its own. Flint reasons in a <think> block before it answers, and its template has no switch to turn that off.

How these compare to other builds

We downloaded other publishers' files and measured them against the same reference, on the same machine and llama.cpp build. Their own published numbers are not used. General text:

Build	Size on disk	KL divergence	top-1
unsloth UD-IQ1_M	11.4 GB	0.499	73.90%
unsloth UD-IQ2_XXS	11.9 GB	0.445	75.29%
ours AD-IQ2_XXS	13.3 GB	0.549	73.38%
unsloth UD-Q2_K_XL	13.6 GB	0.335	78.81%
unsloth UD-IQ3_XXS	14.5 GB	0.319	80.07%
bartowski IQ2_M	14.9 GB	0.361	78.80%
ours AD-IQ2_M	16.4 GB	0.308	80.53%
bartowski IQ3_XXS	17.5 GB	0.215	83.67%
unsloth UD-Q3_K_XL	18.5 GB	0.161	86.04%
ours AD-IQ3_S	19.1 GB	0.181	85.37%
bartowski IQ4_XS	22.4 GB	0.082	90.46%
RecursiveSI flint-nvfp4	22.9 GB	0.167	85.48%
ggml-org Q4_K_M	23.4 GB	0.084	90.27%
ours AD-IQ4_XS	24.5 GB	0.061	91.71%
unsloth UD-Q4_K_XL	25.1 GB	0.056	92.06%
RecursiveSI flint-int4-q4_1	26.0 GB	0.157	85.88%

At every size we measured, Unsloth's files are ahead of ours on this reference. Ours measure better than bartowski's, ggml-org's and RecursiveSI's own GGUF files, though bartowski's are 1.5 to 2 GB smaller at the same names. We are working on the layout and will update the files here when a new one measures better.

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe flint-moe. It is 4,626,695 tokens across 3,164 documents:

Part	Share
agentic and tool use	23.8%
multilingual	19.6%
code	17.3%
reasoning	13.0%
long context	11.1%
vocabulary sweep	7.0%
graphics	4.4%
structured data	3.9%

Every conversation is rendered through this model's own chat template, so the chat markup appears as the single tokens the model actually sees.

If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out general text is 5.1960 plus or minus 0.0269, and 5.6551 on the code text.

Held-out text: eval_neutral (131 chunks at 4096 context) and eval_code (92 chunks) from our calibration dataset, never used for calibration.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement. llama.cpp stores the reference's probabilities in 16 bits with a floor, which puts a floor of 0.00017 under every KL divergence here (0.00005 on code). It is the same for every file.

Hardware: 2x H100-SXM5-80GB, llama.cpp 19ab5a5.

The raw logits of the reference are published in the metrics repo, split into parts because of the file size limit. With them you can measure your own build against exactly the same point we did:

cat base-neutral.kld.*.part > base-neutral.kld

llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99 --no-escape

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, in one run:

llama-imatrix -m Flint-BF16.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special

Then, for AD-IQ4_XS:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.([2-9]|1[0-9])\.ffn_(gate|up|down)_exps\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_(gate|up|down)_exps\.weight$=iq4_xs' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)_shexp\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.ffn_(gate|down|up)\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_\w+\.weight$=q8_0' \
  --output-tensor-type q8_0 \
  --token-embedding-type q8_0 \
  Flint-BF16.gguf Flint-AD-IQ4_XS.gguf IQ4_XS

The rules for every other file are in ladder/ in the metrics repo, next to its quantization log. The first rule that matches a tensor wins.

Model details

39.8B parameters, about 2.5B active per token. 49 layers: the first 2 are dense, and the other 47 have 256 routed experts (6 active per token) plus one shared expert. 35 layers use sliding-window attention and 14 use full attention. Hidden size 2048, vocabulary 150,144, context 262,144.

You need a llama.cpp recent enough to know the flint architecture, which was added in ggml-org/llama.cpp#30928 on 21 October 2026. These files were built and checked with llama.cpp 19ab5a5 from 22 October 2026.

The original model is RecursiveSI/flint, released under Apache 2.0. Its license file is included here.