cloudnathan5/Dory1.0-4B-GGUF

Dory1.0-4B GGUF (imatrix quants)

GGUF quantizations of nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16, with KL-divergence measurements for every file at 1, 2 and 4 loops.

This is an independent, unofficial quantization; it is not affiliated with or endorsed by NVIDIA. NVIDIA's own GGUF is a single Q5_K_M at nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF.

TL;DR

• Pick Q4_K_M (2.43 GB) as the default, Q5_K_M (2.84 GB) or Q6_K (3.27 GB) if you have room, IQ3_M (1.92 GB) if you don't.
• Our Q5_K_M is 23% smaller than NVIDIA's Q5_K_M with the same KLD, and decodes ~2.8× faster on CUDA (see Differences from NVIDIA's GGUF).
• Needs a llama.cpp build with Dory support (ggml-org/llama.cpp#30306, not merged yet).
• The GGUFs default to 4 loops; NVIDIA recommends starting at 2. Set it per run with --override-kv (below).

The base model

Dory1.0-4B is a looped (latent-recurrent) 4B LLM from NVIDIA's Nemotron Labs. A shared middle block of layers is applied repeatedly per token, so one checkpoint covers several compute budgets: more loops means more compute per token and higher accuracy, without changing the weights.

Architecture	normalized-GPT (nGPT) decoder, 36 layers: 8 input → 20 recurrent (looped 1-4×) → 8 output
Attention	3 sliding-window (1024) : 1 full, GQA 16 query / 4 KV heads, head dim 256
Width / FFN / vocab	2560 / 7168 / 131,072 (untied embeddings and LM head)
Context	256K (262,144)
Training	6T tokens, the first ~4.5T in NVFP4, then BF16; plus 300B tokens of context extension. Reasoning model (<think>), tool calling via the chat template
Loop counts	1-4. NVIDIA: 2 is the default, 1 for minimum compute, 4 for the hardest (e.g. agentic) tasks
KV cache modes	per_loop (default: separate KV for each loop) or last_loop (one KV cache per recurrent layer, overwritten each loop; same memory as 1 loop, accuracy close to per_loop per NVIDIA)
License	OpenMDW-1.1

Benchmarks, intended use and training data: see NVIDIA's model card.

Running it

1. Build llama.cpp with Dory support

Dory support is in ggml-org/llama.cpp#30306 (unmerged). These files were made and tested at PR revision 43e865ec059a35d57f1167eaea76ca187077f063. CUDA example:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/30306/head
git checkout 43e865ec059a35d57f1167eaea76ca187077f063   # or FETCH_HEAD for the latest PR head
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server llama-cli

Use -DGGML_METAL=ON on Apple Silicon or leave GPU flags off for CPU. You don't need the patch in llama.cpp/ to run these files, only to reproduce the conversion.

2. Download a quant

hf download cloudnathan5/Dory1.0-4B-GGUF Dory1.0-4B-Q4_K_M.gguf --local-dir .

3. Serve

./build/bin/llama-server -m Dory1.0-4B-Q4_K_M.gguf \
    -ngl 999 -c 32768 --jinja --reasoning on \
    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 \
    --override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"

Web UI at http://127.0.0.1:8080, OpenAI-compatible API at /v1. Sampling settings are NVIDIA's recommendation.

• Loops: dory.num_loops=int:1|2|3|4. Without the override the files run at 4.
• Use -ngl 999, not the layer count. llama.cpp counts virtual layers (attention and FFN separately, recurrent ones once per loop: 192 at 4 loops), so a smaller -ngl silently leaves part of the model on the CPU.
• Long context on small GPUs: dory.recurrent_kv_cache_mode=str:last_loop keeps KV at the 1-loop size at any loop count, and/or add -ctk q8_0 -ctv q8_0 (see the memory table below).

Which file?

Pareto-optimal set, KL divergence vs BF16 at 2 loops (lower is better; top-1 = how often the quant's most likely token matches BF16's):

file	size	KLD	top-1
Q8_0	4.24 GB	0.0030	98.9%
Q6_K	3.27 GB	0.0067	98.2%
Q5_K_M	2.84 GB	0.0150	97.2%
Q4_K_M	2.43 GB	0.0298	96.0%
Q4_K_S	2.35 GB	0.0334	95.7%
IQ4_XS	2.23 GB	0.0369	95.4%
IQ3_M	1.92 GB	0.0711	93.1%
IQ3_XS	1.79 GB	0.0989	91.8%
IQ3_XXS	1.66 GB	0.1252	90.6%
IQ2_M	1.52 GB	0.1902	87.9%
IQ2_S	1.43 GB	0.2365	86.2%
IQ2_XS	1.34 GB	0.2850	84.3%
IQ2_XXS	1.24 GB	0.3940	81.1%

The other stock quants (Q5_K_S, IQ4_NL, Q3_K_L/M/S, IQ3_S, Q2_K, Q2_K_S) and the BF16 GGUF are also in the repo root. They are dominated on KLD per byte but may run faster on backends where I-quant kernels are slow (CPU, some Metal paths). For quality at 2 bits, IQ2_M is the floor we'd consider usable.

Memory

The weights are small; the KV cache is what grows. In the default per_loop mode at 4 loops there are 24 full-attention layer instances (2 input + 5 recurrent × 4 loops + 2 output), each storing 4 KV heads × 256 dims for K and V: ~97 KiB per token with an F16 cache, ~55 KiB with q8_0. Sliding-window layers add a small fixed amount.

Measured total VRAM (nvidia-smi, llama-server, one slot, flash attention auto, RTX PRO 6000):

config	4K	8K	32K	64K	128K	256K
Q4_K_M, F16 KV, 4 loops per_loop	3.6 GiB	3.9 GiB	6.2 GiB	9.2 GiB	15.3 GiB	27.4 GiB
Q4_K_M, q8_0 KV, 4 loops per_loop			4.7 GiB		10.0 GiB
Q4_K_M, F16 KV, 2 loops per_loop			4.8 GiB
Q4_K_M, F16 KV, any loops last_loop			4.1 GiB

At 4K: IQ4_XS 3.4, Q4_K_M 3.6, Q5_K_M 3.9, Q6_K 4.3, Q8_0 5.1 GiB. Rough guide: an 8 GB card runs Q4_K_M/Q5_K_M at ~32K (F16 KV) or ~64K (q8_0 KV), more with last_loop; a 16 GB card runs Q8_0 at 64K, or Q4_K_M at 128K with q8_0 KV. Leave headroom if the GPU also drives a display.

Speed

Decode, batch 1, RTX PRO 6000 Blackwell (CUDA 13.3): Q5_K_M 193.5 tok/s at 1 loop, 80.9 tok/s at 4 loops. Cost per token scales roughly with the number of layer passes (36 at 1 loop, 96 at 4).

How these were made

• Source: BF16 safetensors → BF16 GGUF with the PR's converter, plus a one-line converter patch that stores the nGPT scale vectors as F32 (see Differences). All quants are made from that BF16 GGUF.
• imatrix (imatrix/imatrix-dory-4loop.gguf): ~927K tokens of Nemotron-Post-Training-Dataset-v1 rendered with Dory's chat template (chat, code, math, STEM, tool calling), computed at 4 loops so the recurrent block's statistics include every loop (it collects exactly 4× the samples of the input/output blocks). ctx 4096, 160 chunks.
• Recipes: llama.cpp's stock recipe for each type, with the imatrix. Nothing custom in the root files.
• Evaluation: llama-perplexity --kl-divergence vs the BF16 GGUF on 16 × 4096-token chunks (~32K scored tokens) of held-out data from the same dataset (disjoint from calibration), at 1, 2 and 4 loops. BF16 PPL on this text: 2.340 / 2.284 / 2.291 at 1 / 2 / 4 loops.

Findings

1. Quant error does not compound with loops. KLD at 4 loops ÷ KLD at 1 loop is 0.98-1.17 for everything Q5 and below, even though the recurrent weights run 4× per token. Only the near-lossless quants grow in relative terms (Q8_0 2.1×), and they stay tiny. nGPT's re-normalization onto the hypersphere after each block probably keeps errors from accumulating. One file serves every loop count.
2. The recurrent block isn't special, and embeddings/LM head precision barely matters. We tried _XL variants (recurrent block one quant level higher) and _L variants (Q8_0 embeddings + output). Every _XL was beaten by a smaller stock quant, and every _L was within noise of its base. They're in experimental/ for reference.
3. The imatrix matters a lot: Q4_K_M 0.030 vs 0.074 without it; Q2_K 0.18 vs 0.70 without it (the no-imatrix Q2_K is also the only file that degrades with loops).
4. I-quants dominate below 4 bits: IQ3_M (1.92 GB) beats Q3_K_L (2.14 GB); IQ2_M is the same size as Q2_K_S with lower KLD (0.19 vs 0.24); IQ4_NL is dominated by IQ4_XS.
5. NVFP4: NVIDIA's own NVFP4 checkpoint (ModelOpt, for vLLM) is Q4_K_M quality (KLD 0.032 at 2 loops, measured by dequantizing it exactly). llama.cpp's NVFP4 type is not: 0.111 quantized from BF16, and 0.098 even when repacking NVIDIA's NVFP4 weights into it. ggml's NVFP4 has no per-tensor scale, so NVIDIA's block scales can't be carried over and get re-derived by plain amax, and Dory's small weights push almost all of the FP8 block scales into the subnormal range. In llama.cpp, use Q4_K_M or IQ4_XS instead. Both ggml NVFP4 files are in experimental/.

Differences from NVIDIA's GGUF

NVIDIA's Q5_K_M uses llama.cpp's stock Q5_K_M layer mix, but keeps the embeddings and LM head in BF16 (1.34 GB of its 3.67 GB) and uses no imatrix. Compared with it, our Q5_K_M:

• is 23% smaller with the same quality: 2.84 GB vs 3.67 GB, KLD 0.0150 vs 0.0157 at 2 loops;
• is ~2.8× faster on CUDA. The PR's converter writes the nGPT scale vectors (*_scale, residual_alpha) as BF16 in BF16 exports. They're consumed by GGML_OP_MUL, which CUDA doesn't support for F32×BF16, so llama.cpp keeps them in host memory and runs a BF16→F32 cast on the CPU for every layer (435 graph splits per token). Storing them as F32 (+2 MB) brings it to 2 splits: Q5_K_M 69.7 → 193.5 tok/s at 1 loop, 27.5 → 80.9 tok/s at 4 loops. NVIDIA's file has the same issue. The fix is llama.cpp/llama.cpp-dory-f32-scales.patch (converter only; all files here already have F32 scales).

Our exact reproduction of NVIDIA's recipe (experimental/Dory1.0-4B-Q5_K_M-NVrecipe.gguf: BF16 embeddings/output, no imatrix, identical tensor types) scores 0.028, worse than NVIDIA's file (0.016). NVIDIA quantized from a pre-BF16 export, and that probably explains the gap.

Full results

KLD vs BF16 (mean) and top-1 agreement, at 1 / 2 / 4 loops. ⚗ = in experimental/; † = reference, not in this repo.

quant	GB	KLD @1	KLD @2	KLD @4	top-1 % @1	top-1 % @2	top-1 % @4	KLD ×4/1
Q8_0	4.24	0.002031	0.002972	0.004186	98.934	98.852	98.837	2.06
Q5_K_M-NVrecipe ⚗	3.68	0.027384	0.027678	0.028930	95.646	95.854	95.744	1.06
NVIDIA Q5_K_M †	3.67	0.016181	0.015669	0.017495	96.922	97.118	96.886	1.08
Q6_K_L ⚗	3.44	0.005784	0.006666	0.007942	98.070	98.232	98.165	1.37
Q5_K_XL ⚗	3.36	0.006929	0.007718	0.008857	98.076	98.083	98.110	1.28
Q6_K	3.27	0.005603	0.006730	0.007736	98.101	98.156	98.125	1.38
NVIDIA NVFP4 (ModelOpt, vLLM) †	3.20	0.031416	0.032140	0.033547	95.533	95.878	95.799	1.07
Q5_K_L ⚗	3.05	0.014558	0.014941	0.015241	97.112	97.197	97.313	1.05
Q4_K_XL ⚗	2.91	0.018032	0.017907	0.019432	96.669	96.965	96.950	1.08
Q5_K_M	2.84	0.014475	0.014968	0.016915	97.078	97.191	97.246	1.17
Q5_K_S	2.79	0.015139	0.015913	0.017363	97.130	96.989	97.109	1.15
Q4_K_L ⚗	2.68	0.031653	0.030150	0.031209	95.631	95.942	95.982	0.99
Q3_K_XL ⚗	2.57	0.046206	0.043815	0.046062	94.376	94.584	94.614	1.00
Q4_K_M-noimat ⚗	2.43	0.074402	0.073470	0.077979	92.413	92.520	92.571	1.05
Q4_K_M	2.43	0.032068	0.029759	0.033552	95.493	95.979	95.838	1.05
IQ4_NL	2.35	0.036168	0.037038	0.038818	95.304	95.435	95.347	1.07
Q4_K_S	2.35	0.035575	0.033381	0.035775	95.402	95.695	95.561	1.01
NVFP4-from-ModelOpt ⚗	2.33	0.101023	0.097839	0.100912	91.094	91.677	91.628	1.00
NVFP4 ⚗	2.33	0.114681	0.111083	0.115871	90.544	90.923	90.950	1.01
IQ4_XS	2.23	0.036075	0.036938	0.039557	95.365	95.396	95.267	1.10
Q3_K_L	2.14	0.079714	0.077637	0.080842	92.104	92.196	92.178	1.01
Q2_K_XL ⚗	2.11	0.128571	0.122824	0.126976	90.257	90.743	90.770	0.99
Q3_K_M	2.01	0.087490	0.083816	0.085869	91.628	92.019	92.043	0.98
Q2_K_L ⚗	1.94	0.186909	0.177722	0.188259	88.211	88.532	88.453	1.01
IQ3_M	1.92	0.075553	0.071135	0.075036	92.944	93.081	92.978	0.99
IQ3_S	1.86	0.085719	0.079906	0.084548	92.590	92.745	92.571	0.99
Q3_K_S	1.85	0.153648	0.147320	0.152097	88.639	89.127	89.133	0.99
IQ3_XS	1.79	0.103870	0.098893	0.101376	91.411	91.762	91.796	0.98
IQ3_XXS	1.66	0.131857	0.125189	0.128707	90.178	90.562	90.547	0.98
Q2_K-noimat ⚗	1.61	0.629025	0.699217	0.754254	75.171	73.812	72.747	1.20
Q2_K	1.61	0.192693	0.183708	0.194131	87.937	88.343	88.144	1.01
Q2_K_S	1.52	0.248474	0.243572	0.252833	85.528	85.863	85.741	1.02
IQ2_M	1.52	0.195255	0.190242	0.196398	87.582	87.909	87.869	1.01
IQ2_S	1.43	0.243480	0.236491	0.241845	85.723	86.154	86.004	0.99
IQ2_XS	1.34	0.292258	0.285019	0.296330	83.964	84.315	84.059	1.01
IQ2_XXS	1.24	0.398152	0.394018	0.408323	80.792	81.094	80.728	1.03

Raw numbers incl. 99th-percentile/max KLD and PPL: eval/kld_results.csv.

Experimental files

file	what
*_L	stock recipe + Q8_0 embeddings and output
*_XL	_L + recurrent block (blk.16-55) one level higher (attn_v/ffn_down two levels)
*-noimat	stock recipe without imatrix (baseline)
Q5_K_M-NVrecipe	exact reproduction of NVIDIA's Q5_K_M recipe
NVFP4	linear weights in ggml NVFP4, quantized from BF16 (embeddings Q4_K, output Q6_K)
NVFP4-from-ModelOpt	same layout, but made from NVIDIA's NVFP4 checkpoint (dequantized, then requantized to ggml NVFP4)

Reproducing

Scripts are in tools/:

1. Build the PR (above) and git apply llama.cpp-dory-f32-scales.patch in the llama.cpp tree.
2. tools/adapt_config.py <hf_dir> <out_dir>: the PR's converter expects NVIDIA's internal config keys, and this translates the public config.json (resulting GGUF metadata matches NVIDIA's). Then convert_hf_to_gguf.py <out_dir> --outtype bf16.
3. tools/build_calib.py: builds calibration and held-out text from Nemotron-Post-Training-Dataset-v1.
4. llama-imatrix ... --override-kv dory.num_loops=int:4 (or use the shipped imatrix).
5. tools/quantize_all.sh: the whole lineup.
6. tools/eval_kld.sh + tools/summarize_kld.py: KLD at 1/2/4 loops.
7. tools/dequant_modelopt_nvfp4.py: dequantizes NVIDIA's ModelOpt NVFP4 checkpoint for the comparison above.

Caveats

• Dory support in llama.cpp is an unmerged PR; the GGUF format for this architecture could still change before it lands.
• Calibration and evaluation text come from the same dataset, so the imatrix's advantage may be slightly overstated on out-of-domain text. No task benchmarks were run on the quants.
• Speed and memory numbers come from one GPU (RTX PRO 6000 Blackwell); expect similar memory and proportionally lower speed elsewhere.

License

The model weights are licensed by NVIDIA under the OpenMDW License Agreement, version 1.1. These quantizations are derivatives distributed under the same license. Base model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16.