netprojects/nemotron-dory1.0-4b-mlx-8bit

NVIDIA Nemotron-Labs Dory 1.0 4B: MLX 8-bit

Built and tested on Apple M5 Max (128 GB) · upstream revision Nemotron-Labs Dory 1.0 4B (208065f)

An 8-bit MLX conversion (affine, group size 64) for Apple Silicon, of nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 (NVIDIA, 16 October 2026). 4.2 GB. Released 17 October 2026.

Dory is NVIDIA's looped 4B reasoning model: a shared block of layers runs 1 to 4 times (2 by default), so one set of weights gives you a speed-versus-accuracy dial. This build runs it with MLX through the small dory-mlx package, which adds the dory architecture.

Published by NetProjects (Internet Projects Ltd). Not affiliated with NVIDIA.

Use

pip install "git+https://github.com/internet-projects/dory-mlx"
python -m dory_mlx "Explain why the sky is blue in three sentences." --model netprojects/nemotron-dory1.0-4b-mlx-8bit --loops 2

import dory_mlx
model, tok = dory_mlx.load("netprojects/nemotron-dory1.0-4b-mlx-8bit", loops=2)   # loops: 1-4
for piece in dory_mlx.chat(model, tok, "What is 17 * 23?", max_tokens=1500):
    print(piece, end="")

The model reasons first (inside <think>...</think>) and then answers. The defaults follow NVIDIA's recommendation: temperature 1.0 and top_p 0.95. For NVIDIA's own accuracy results, see the original model card.

What we verified

Check	Result
Logits vs an independent fp32 PyTorch implementation (real weights, 2 loops, 55 tokens)	top-1 agreement 100%, probability drift (KL) 0.001 to 0.003
8-bit vs the bf16 MLX build (5 short texts, 2 loops)	top-1 agreement 98%, KL 0.003, same text-prediction loss (1.278 vs 1.271)
WikiText-2 test perplexity (first 64K tokens, 1,024-token windows), loops 1 / 2 / 4	12.18 / 11.77 / 11.70 (bf16 build: 12.17 / 11.76 / 11.69)
Sliding-window caches (window shrunk to 64; chunked prefill and token-by-token decode vs one full pass)	top-1 agreement at least 98.7%, KL at most 0.0006

More loops lower the perplexity (12.17, 11.76, 11.69 for bf16), so the loop dial works as NVIDIA describes.

Speed and memory

Measured on a MacBook Pro M5 Max (128 GB) on 20 October 2026, with no other models running; medians of repeated runs (run-to-run spread about 2%).

Loops	Generation, short prompt	Generation after a 32K-token prompt	Reading a 32K-token prompt	Peak memory, 32K-token prompt
1	101 tokens/s	82 tokens/s	3,200 tokens/s (10 s)	7.7 GB
2	69 tokens/s	54 tokens/s	1,980 tokens/s (17 s)	9.0 GB
4	39 tokens/s	31 tokens/s	1,070 tokens/s (31 s)	11.5 GB

With the Dory model file proposed for MLX-LM itself (mlx-lm PR 1988), MLX-LM's batch generation serves several requests at once: about 200 tokens/s in total for 4 requests and 450 tokens/s for 16 (8-bit, 2 loops).

With MLX-LM directly. Using the Dory support proposed for MLX-LM (PR 1988, commit 208065f), on the same M5 Max on 23 October 2026: 8-bit, 2 loops, per-loop cache, greedy decoding, medians of 3 runs, with a 1 GiB MLX cache limit (which costs about 5–7% of prompt-reading speed; generation is unaffected):

Prompt → reply	Reading the prompt	Generation	Peak memory
25 → 505 tokens	402 tokens/s	68.0 tokens/s	4.39 GB
4,096 → 192 tokens	2,198 tokens/s (1.9 s)	64.4 tokens/s	8.97 GB
16,384 → 192 tokens	2,101 tokens/s (7.8 s)	59.6 tokens/s	8.97 GB
32,768 → 192 tokens	1,874 tokens/s (17.5 s)	53.0 tokens/s	8.97 GB

On a 16 GB MacBook Pro M1 Pro, the Kapitsa team independently measured MLX-LM with this PR (4-bit, 2 loops) at 27.9 tokens/s with a 1K-token context and 19.3 tokens/s at 32K (nace-ai/kapitsa#296).

Notes

• English text in, text out. Replies include the reasoning chain, so they are longer than a plain chat model's.
• The weights use a fused layout (query, key, value and gate in one matrix; MLP gate and up in one matrix) for faster decoding. scripts/convert.py in dory-mlx rebuilds them from NVIDIA's original.
• Verified with prompts up to 24K tokens.
• Recommended: the 8-bit build. It stays very close to bf16 (KL 0.003, top-1 agreement 98%) and is half the size.

More from NetProjects

• dory-mlx on GitHub and the NetProjects Hugging Face organization
• H2O-Lightning-4B on MLX and the article about it
• How we use AI at NetProjects

Credits and license

Model: NVIDIA (Nemotron-Labs Dory 1.0 4B, OpenMDW-1.1, the same license as the original; full text in LICENSE). Reference implementations used to write and check the port: vLLM (PR 60149) and llama.cpp. Code in dory-mlx: Apache-2.0. Port by Paris Michalas, Partner, NetProjects (Internet Projects Ltd). Not affiliated with NVIDIA.

More NetProjects MLX builds: NetProjects MLX builds collection · all models at huggingface.co/netprojects