AtomicChat/Spark-X2.6-4B-GGUF

How to Run Spark-X2.6 4B Locally

GGUF builds of XHToken's Spark-X2.6 4B, made from the original BF16 weights with our own importance matrix, and measured against every other build we could find. The calibration corpora behind our builds are public.

• You can now run Spark-X2.6 in Atomic Chat with thinking on by default (llama.cpp engine b11946 or newer).
• AD-Q4_K_M, 2.79 GB on disk, matches the original's token choice 90.6% of the time; the official Q4_K_M (3.54 GB) 88.1%.
• Every other GGUF build we could find is measured on the same stand below, including where it beats ours.

---

Every Spark-X2.6 4B GGUF we could find, ours and others', measured against the BF16 original on the same held-out text. The dashed line is the best file from anyone but us at each size; red is where it is closer to the original than ours.

Pick a file

Every number is measured on one stand; how is at the bottom, and the raw logs are in the metrics repo. KL divergence is how far a file's predictions drift from the original weights (lower is better, zero is identical); top-1 is how often it picks the same next token as the original.

File	Size on disk	KL divergence	top-1
Spark-X2.6-4B-Q8_0.gguf	4.38 GB	0.00236	98.09%	closest to the original; the official Q8_0 is the same file
Spark-X2.6-4B-AD-Q6_K.gguf	3.86 GB	0.00482	97.08%	near Q8_0 for 0.5 GB less
Spark-X2.6-4B-AD-Q5_K_M.gguf	3.15 GB	0.01602	94.25%	28% smaller than Q8_0
Spark-X2.6-4B-AD-Q4_K_M.gguf	2.79 GB	0.04202	90.62%	recommended
Spark-X2.6-4B-AD-IQ4_XS.gguf	2.63 GB	0.05670	89.27%	the size of others' Q4_K_M
Spark-X2.6-4B-AD-IQ3_S.gguf	2.14 GB	0.19229	80.74%	smallest we publish

AD- marks an Atomic Dynamic layout: each tensor group gets its own type, set by measurement, and the name says what the feed-forward tensors got. llama.cpp's default Q4_K_M of the same BF16 (2.60 GB, KL divergence 0.0921, top-1 86.35%) is the baseline every AD- file beats at its size; it is not published here. AD-IQ4_XS-IQ3_S (2.30 GB) lost to bartowski's IQ4_XS (2.34 GB, KL divergence 0.098 against our 0.138), so it is not published; AD-IQ3_XXS (1.95 GB) fell under our 76% top-1 bar.

How these compare to other builds

Their files, downloaded and measured with our reference, text, context and llama.cpp build. Their own published numbers are never used. Each one is set against our best published file at its size or smaller (2% slack for files that are the same size give or take a few megabytes). The chart draws a line between our files; this table only compares files you can download, so a file that falls between two of ours can be closer to the original than the smaller of them.

Their file	Size	KL divergence	top-1	Our best at that size	Size	KL divergence	top-1	Closer to the original
official Q8_0	4.38 GB	0.0024	98.09%	Q8_0	4.38 GB	0.0024	98.09%	same file
bartowski Q6_K	3.61 GB	0.0070	96.27%	AD-Q5_K_M	3.15 GB	0.0160	94.25%	theirs
official Q4_K_M	3.54 GB	0.0700	88.12%	AD-Q5_K_M	3.15 GB	0.0160	94.25%	ours
bartowski Q5_K_M	3.16 GB	0.0228	93.15%	AD-Q5_K_M	3.15 GB	0.0160	94.25%	ours
bartowski Q4_K_M	2.62 GB	0.0697	87.91%	AD-IQ4_XS	2.63 GB	0.0567	89.27%	ours
llama.cpp default Q4_K_M	2.60 GB	0.0921	86.35%	AD-IQ4_XS	2.63 GB	0.0567	89.27%	ours
bartowski IQ4_XS	2.34 GB	0.0977	85.73%	AD-IQ3_S	2.14 GB	0.1923	80.74%	theirs
bartowski IQ3_M	2.23 GB	0.2012	79.92%	AD-IQ3_S	2.14 GB	0.1923	80.74%	ours

Which one fits your card

The file has to fit, and so does the context. With llama.cpp's full-size cache this model keeps 144 KB per token (1.1 GB at 8k context, 4.5 GB at 32k); three of every four blocks attend over a 512-token window only, so the default sliding-window cache needs less. The table plans for the full-size cache and leaves 1 GB for compute buffers.

Your card	File	Leaves room for
6 GB	AD-Q6_K	about 9k context
8 GB	Q8_0	about 20k context
12 GB	Q8_0	about 49k context
16 GB	Q8_0	about 77k context

Running it

hf download AtomicChat/Spark-X2.6-4B-GGUF Spark-X2.6-4B-AD-Q4_K_M.gguf --local-dir .
llama-server -m Spark-X2.6-4B-AD-Q4_K_M.gguf -ngl 99 -c 16384 --jinja

The chat template is inside every file. Thinking is on by default; per request, turn it off or make it shorter (the template takes high, the default, and low only):

{"messages": [...], "chat_template_kwargs": {"enable_thinking": false}}
{"messages": [...], "chat_template_kwargs": {"reasoning_effort": "low"}}

Checked on llama-server (a34de8618) with AD-Q4_K_M: a chat answer with thinking, thinking off, low effort, and a tool call parsed into tool_calls.

Speed

AD-Q4_K_M on one NVIDIA A10 (24 GB), llama-bench, flash attention on, all layers on the GPU, llama.cpp a34de8618:

test	tokens per second
pp512	5,608
pp8192	5,284
tg128	126.3

What we found while building these

• The official GGUF keeps the 131,072-row token embedding in BF16 and writes the tied head again as output.weight in Q8_0, so its Q4_K_M is 3.54 GB against llama.cpp's 2.60 GB for the same preset. Here the embedding is the head (tied), at the head's type.
• The fused attn_qkv and the per-head sigmoid gate on the attention output (attn_gate) see the largest activations in the importance matrix, most of all in the full-attention blocks; they stay one step above the feed-forward tensors, and attn_gate (a few thousand weights per block) stays q8_0.
• This model is sensitive to quantization: llama.cpp's own Q4_K_M keeps 86.3% top-1 here, where our AD-Q4_K_M of Qwen3.8-27B keeps 95.6%.
• AD-IQ4_XS-IQ3_S (2.30 GB) lost to bartowski's IQ4_XS (2.34 GB, KL divergence 0.098 against our 0.138), so it is not published; AD-IQ3_XXS (1.95 GB) fell under our 76% top-1 bar.

The calibration data

Our importance matrix comes from a corpus we built for this model and published: AtomicChat/calib-corpora, recipe spark-x26. It is 4,787,678 tokens across 2,986 documents:

Part	Share
agentic and tool use	25.2%
code	19.3%
reasoning	15.7%
multilingual	14.6%
long context	12.1%
vocabulary sweep	5.4%
structured data	4.6%
graphics	3.1%

Every conversation is rendered through this model's own chat template, so the chat markup appears as the single tokens the model actually sees.

If you build your own importance matrix from this corpus, pass --parse-special to llama-imatrix. Without it the chat markup is read as ordinary punctuation.

How we measured

Reference: the original BF16 weights, converted to GGUF and run unquantized. Perplexity on the held-out set is 9.3342 plus or minus 0.05925.

Held-out text: eval_neutral from our calibration dataset, never used for calibration. 90 chunks at 4096 context.

Metric: per-token KL divergence of each file's predictions against the reference, plus top-1 agreement.

Hardware: 1x A100-SXM4-40GB.

The reference logits (48 GB) are not published. To measure your own build at the same point, make them from the BF16 (convert_hf_to_gguf.py on the original) and the held-out text in the calibration dataset:

llama-perplexity -m Spark-X2.6-4B-BF16.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld -c 4096 -ngl 99
llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
  --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99

Reproducing a file

The importance matrix was collected on the BF16 weights rather than on a quantized copy, in one run:

llama-imatrix -m Spark-X2.6-4B-BF16.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special

Then, for AD-Q4_K_M:

llama-quantize \
  --imatrix imatrix.gguf \
  --tensor-type '^blk\.(0|1|29|30|31|32|33|34|35)\.ffn_(down|gate|up)\.weight$=q5_k' \
  --tensor-type '^blk\.(2|3|4|5|6)\.ffn_(down|gate|up)\.weight$=q5_k' \
  --tensor-type '^blk\.\d+\.ffn_down\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.ffn_gate\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.ffn_up\.weight$=q4_k' \
  --tensor-type '^blk\.\d+\.attn_qkv\.weight$=q6_k' \
  --tensor-type '^blk\.\d+\.attn_gate\.weight$=q8_0' \
  --tensor-type '^blk\.\d+\.attn_output\.weight$=q6_k' \
  --tensor-type '^token_embd\.weight$=q6_k' \
  --override-kv general.file_type=int:15 \
  Spark-X2.6-4B-BF16.gguf Spark-X2.6-4B-AD-Q4_K_M.gguf Q8_0

The rules for every other file are in ladder/ in the metrics repo, next to its quantization log. The first rule that matches a tensor wins.

Model details

36 layers, hidden size 2560, feed-forward size 10240, vocabulary 131,072, context 1,048,576.

Needs a recent llama.cpp with spark2_5 architecture support. Built and tested with llama.cpp b12015 (a34de8618).

License

Apache-2.0, as the original: XHToken/Spark-X2.6-4B. The original's LICENSE file is in this repo. Quantized by AtomicChat.