keptn/nemotron-3.5-bolt-e4b-bf16

NVIDIA Nemotron 3.5 Bolt E4B · bf16 · for radiance

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 in one 17.0 GiB file for the radiance inference engine on AMD RDNA4. The weights are as NVIDIA ships them, with NVIDIA's DFlash drafter, the vision tower and the audio encoder, on one, two or four cards.

• Close to the original: KL divergence 0.0015 to the bf16 original on wikitext-2; top token the same in 98.8%
• Fast: 155 tok/s BetterBench combined on one card, 241 tok/s on two cards, 359 tok/s on four cards
• Long context: 65,536 tokens; one card holds 8 sequences at full length
• Vision: images through the model's bf16 vision tower
• Audio: speech through the model's bf16 audio encoder; LibriSpeech test-clean WER 3.9%
• Thinking: on or off per request; the benchmarks below show both

File	nemotron-3.5-bolt-e4b-bf16.rad, 17.02 GiB
Needs	radiance 1.12.0 or newer; one card with 32 GB (tested on Radeon AI PRO R9700) and 8 GiB of free host memory. Two cards: PR #239. Four cards: PR #258. Audio: PR #257
Weights	bf16, as in the checkpoint
Activations	bf16
lm_head	bf16 (tied to the token table)
Speculation (DFlash)	NVIDIA's DFlash drafter, 5 layers, FP8 with one bf16 scale per 128×128 block, 7 draft tokens
Vision tower	the original, bf16, in pinned host memory
Audio encoder	the original, bf16, in pinned host memory
Context	65,536 tokens
Version	1.1 (2026-10-25): adds the audio encoder

Quick start

With the compose files from the radiance repository (1.12.0 or newer). Set RADIANCE_MODELS, RADIANCE_API_KEY and the group ids in .env, see radiance's README:

hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --local-dir ~/models
cp -r radiance/deploy/compose ~/radiance-compose && cd ~/radiance-compose
cp .env.example .env
docker compose -f nemotron-3.5-bolt-e4b-bf16.yaml up -d

Or directly:

hf download keptn/nemotron-3.5-bolt-e4b-bf16 nemotron-3.5-bolt-e4b-bf16.rad --local-dir ~/models
docker run -d --init --name nemotron-bolt --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0 \
    stilldeadcode/radiance:1.12.0 --model /models/nemotron-3.5-bolt-e4b-bf16.rad --host 0.0.0.0 --port 8000 \
    --kv-cache-dtype bf16 --max-model-len 65536 --max-num-seqs 8 --num-speculative-tokens 7 \
    --mm-max-patches 2520 --vision-tower-placement ram \
    --temp 1.0 --top-p 0.95 --top-k 64 --chat-template-kwargs '{"enable_thinking":false}'

	change
fp8 KV	--kv-cache-dtype fp8: half the KV memory, KL divergence 0.014 instead of 0.0015
no drafter	--num-speculative-tokens 0
two cards	--tp 2 --p2p on, ROCR_VISIBLE_DEVICES=0,1; needs radiance with PR #239
four cards	--tp 4 --p2p on, ROCR_VISIBLE_DEVICES=0,1,2,3; needs radiance with PR #258
text only	--mm-max-patches 0 (no images and no audio), drop --vision-tower-placement
thinking	off by default, as in NVIDIA's template; per request "chat_template_kwargs": {"enable_thinking": true}

The drafter runs at 7 draft tokens only: it was trained on blocks of 8. Sampling is NVIDIA's (temperature 1.0, top_p 0.95, both thinking modes) and the checkpoint's top_k 64.

Quality

Distance to the bf16 original. KL divergence on wikitext-2 (test), llama.cpp method: 100 chunks of 512 tokens, the second half of each chunk scored, 25,500 tokens. Reference: the bf16 original in HF transformers, fp32 math.

	mean KL	median KL	99th percentile	top token the same	perplexity (reference: 117.18)
this file, bf16 KV	0.0015	0.00023	0.014	98.8%	117.56
this file, fp8 KV	0.0137	0.0021	0.18	96.3%	117.19

On two and four cards (bf16 KV) the mean KL divergence of this file is 0.0014 and 0.0014.

For scale: HF transformers with bf16 math has a KL divergence of 0.0012 to the same reference (first chunk, 255 tokens).

Benchmarks, greedy. The thinking-off column is the one to compare with other cards.

benchmark	thinking off	thinking on
GSM8K (math, 5-shot)	82.3%	89.6%
IFEval (following instructions, strict)	87.1%	91.5%
LiveCodeBench v6 (175 tasks, pass@1)	46.3%	55.4%

• LiveCodeBench v6 by difficulty (off and on): easy 40/43 and 43/43, medium 26/52 and 32/52, hard 15/80 and 22/80.
• Thinking on uses output limits of 8,192 (GSM8K), 16,384 (IFEval) and 65,536 tokens minus the prompt (LiveCodeBench; the context has 65,536 tokens).
• Thinking off, 137 GSM8K answers stop at the task's limit of 256 tokens, 5 IFEval answers at 1,280 and 5 LiveCodeBench answers at 8,192. Thinking on, 1 IFEval answer stops at 16,384. They count as wrong.

Speed

Radeon AI PRO R9700 at 215 W, bf16 KV, 7 draft tokens, thinking off. One card: radiance 1.11.0 + PR #210 (in 1.12.0). Two cards: radiance main 248ecf5 + PR #239 + PR #238 (both now in main), --p2p on. Four cards: the same + PR #257 + PR #258, --p2p on.

BetterBench 0.6.1, full run: 8 categories, 20 runs each after 3 warm-up runs, temperature 0.7, one stream. Then prefill and parallel streams. Decode tok/s (median):

	1 card	2 cards	4 cards
combined	154.9	241.3	358.8
chat	117.7	189.7	259.1
code	167.4	266.3	398.7
file_edit	176.9	272.5	419.1
json	241.6	370.6	566.3
math	208.6	315.3	493.2
prose	77.3	118.6	171.6
reasoning	127.1	191.4	268.5
summarization	137.5	224.6	328.8
prefill at 24k tokens	6,665	10,241	11,605
prefill at 47k tokens	5,581	8,866	10,725
4 streams, total	262	486	772
8 streams, total	303	586	920

The drafter gains most on code and JSON (4.1 to 4.7 tokens a step) and least on prose (2.0 tokens a step).

Two cards split the weights and the KV cache: +56% combined, the most of the three files, because this file reads the most bytes of weights for each token. Four cards give +49% combined over two cards.

Vision

The model's vision tower (bf16, with the clamps the checkpoint stores) reads images in chat requests (image_url).

benchmark (lmms-eval 0.7.3, LMMs-Eval-Lite, greedy, thinking off)	result
DocVQA val lite (500 questions)	79.8 (ANLS)
AI2D lite (500 questions)	74.6%

Audio

The model's audio encoder (bf16) reads speech in chat requests (input_audio, WAV). It needs radiance with PR #257. A clip is cut at 30 s.

LibriSpeech test-clean, 2,620 utterances, greedy, thinking off, prompt "Transcribe this audio clip.", Whisper English text normalizer (as the Open ASR Leaderboard). One card, 8 requests at a time:

file	WER	RTFx
this file	3.87%	64
int8	3.86%	126
4-bit	3.81%	117

Two cards (--tp 2 --p2p on), this file: WER 3.87%, RTFx 94. Four cards (--tp 4 --p2p on), the 12 clips below: KL 0.00059, 8 of 12 with the same text.

Distance to the original. 12 LibriSpeech clips, compared with the bf16 original in HF transformers (fp32 math). KL divergence over the top 20 tokens, up to the first token that is different:

file	same text	mean KL
this file	11/12	0.00055
int8	9/12	0.0015
4-bit	5/12	0.017

Most errors come from the model, not from the encoder: it translates Latin passages, completes quotes from memory and spells names differently. One answer of 2,620 (the same clip in all three files, on one and two cards) repeats spaces after an unknown name and stops at the limit of 160 tokens. It counts as wrong. NVIDIA reports no audio results for this model.

Memory

KV cache, --max-model-len 65536, from radiance's startup report:

	2 sequences	4 sequences	8 sequences
1 card, bf16 KV	131,072 tokens	262,160 tokens	524,304 tokens
1 card, fp8 KV	131,104 tokens	262,160 tokens	524,336 tokens

Tokens of context the KV pool holds in total, shared by all sequences. Every sequence can have the full context. Only 24 of the 42 layers hold KV: the last 18 layers read the cache of layers 22 and 23. With 8 sequences the cache takes 18.2 GiB (bf16 KV) or 9.1 GiB (fp8 KV) of the card. The weights take 9.6 GiB. The token tables (6.5 GiB), the vision tower and the audio encoder (0.9 GiB together) stay in pinned host memory. With two cards each card holds half of the KV cache (9.1 GiB for 8 sequences at bf16 KV). With four cards each card holds 6.6 GiB (8 sequences, bf16 KV).

How it was made

1. Downloaded nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 (revision 49843eed).
2. Converted NVIDIA's drafter from GGUF to safetensors with drafter/gguf_to_safetensors.py. The GGUF is nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash (revision 0949a2eb); config.json comes from nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 (revision 913d13c4).
3. Compared every drafter tensor with the NVFP4 release: cosine 0.989 to 0.994 for the matrices (4-bit noise), 1.000 for the norms.
4. Built the container. The trunk, the vision tower and the audio encoder stay as the checkpoint holds them; the drafter goes to block FP8.

python drafter/gguf_to_safetensors.py Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf \
    NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4 dflash-bf16
rad-convert NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 --recipe nemotron-3.5-bolt-e4b-bf16.recipe \
    --draft-model dflash-bf16 --set quantization_config.quant_method=compressed-tensors \
    -o nemotron-3.5-bolt-e4b-bf16.rad

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance 1.12.0 or newer. radiance 1.11.0 and older refuse the E4B form of Gemma 4.
• Two cards need PR #239 (in radiance main, not yet in a release). Four cards need PR #258.
• NVIDIA lists text as the only input of this model. The vision tower and the audio encoder are those of the checkpoint.
• Audio needs radiance with PR #257. Older radiance serves text and images from this file and answers audio requests with HTTP 501.
• The context is 65,536 tokens, the length NVIDIA trained.
• The wikitext-2 perplexity of the original is 117. Compare the KL divergence with the bf16 math value (0.0012), not with other cards.

Credits

• Model and drafter: NVIDIA, post-trained from Google's Gemma 4 E4B (Apache 2.0).
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Test sets: GSM8K, IFEval, LiveCodeBench, LMMs-Eval-Lite, LibriSpeech, wikitext-2.

License

OpenMDW License Agreement 1.1, the license of the base model. See LICENSE. The Gemma 4 base model is under the Apache License 2.0, see LICENSE.Apache-2.0. NOTICE names the sources and NVIDIA's terms.