mlboydaisuke/gemma-4-E4B-it-Foundry-Local

Gemma 4 E4B for Foundry Local (ONNX Runtime GenAI, int4)

google/gemma-4-E4B-it as a Foundry Local 0.13.0 local model: the int4 package with an 8K-token context decodes at 58.8 tokens/s on an RTX 5080 Laptop GPU (Windows 11 build 26200.6899, 2026-11-06; 32 tokens/s after a Windows update the same day, see Speed). The packages were exported with Mobius 0.1.0 and quantized with Olive 0.13.0, the tools of the olive-recipes Gemma 4 E2B recipes.

Steps, scripts and the result files: john-rocky/gemma-4-e4b-on-foundry-local.

Folders

Folder	Runs on	Weights	Context
cuda-int4-8k/	NVIDIA GPU, CUDA EP	decoder int4 (RTN, block 32), embedding / vision / audio int8	8,192 tokens
cpu-int4/	CPU, CPU EP	decoder int4 (RTN, int8 compute), embedding / vision / audio int8	131,072 tokens

Each folder holds the four ONNX Runtime GenAI sessions (decoder, embedding, vision encoder, audio encoder), genai_config.json, the tokenizer and the processors.

Run it in Foundry Local

hf download mlboydaisuke/gemma-4-E4B-it-Foundry-Local --include "cuda-int4-8k/*" --local-dir gemma-4-e4b
# Foundry Local's CUDA EP loads the CUDA 12 runtime and cuDNN 9 from the daemon's PATH and does not download them
# (step 4 of the GitHub repo puts them in a folder): start the daemon with that folder on PATH.
$env:Path = "<folder with the CUDA 12 and cuDNN 9 DLLs>;$env:Path"
foundry server stop
foundry server start
foundry model register (Resolve-Path gemma-4-e4b\cuda-int4-8k) gemma-4-e4b-it-cuda-int4-8k:1 --task vision-language-chat --device gpu --execution-provider CUDAExecutionProvider
foundry model load gemma-4-e4b-it-cuda-int4-8k:1
foundry complete gemma-4-e4b-it-cuda-int4-8k:1 "What is the capital of Japan? Answer in one word." -m 48 -t 0

For the CPU folder: --device cpu --execution-provider CPUExecutionProvider. Registered local models answer over REST on the daemon's /catalogs/local/v1 path; /v1 answers HTTP 404 for them. Foundry Local's documented cache-folder route (a folder with inference_model.json in the model cache, foundry cache cd) did not list these packages with CLI 0.13.0; foundry model register works.

Agreement with the Hugging Face model

24 fixed prompts, greedy, 48 new tokens, against transformers 5.22.1 in bf16 on the same laptop. The answer keys were written before any ONNX output was measured; a form passes when it answers right wherever the reference does (20 of the 24), with at most one miss.

Package	Runner	Device	Keys	Text = HF	First token = HF
cuda-int4-8k	Foundry Local 0.13.0	RTX 5080 Laptop	20/20 PASS	13/24	—
cpu-int4	Foundry Local 0.13.0	Core Ultra 9 386H	20/20 PASS	13/24	—
cuda-int4-8k weights, 131,072 context	ONNX Runtime GenAI 0.19.0	RTX 5080 Laptop	20/20 PASS	13/24	22/24 (max abs Δ 34.4)
cpu-int4	ONNX Runtime GenAI 0.19.0	Core Ultra 9 386H	20/20 PASS	13/24	22/24 (max abs Δ 33.8)

The text differs from the reference on 11 of 24 prompts because of the decoder's int4 weights: an fp16 decoder gives 22/24 (results in the GitHub repo).

Speed

One laptop (MSI Stealth 16 AI+, RTX 5080 Laptop 16 GB, driver 591.74), 2026-11-06, Windows 11 build 26200.6899. Foundry Local: a fresh daemon, load, the 24 prompts, then a streamed chat completion with a 933-token prompt and 160 new tokens, median of 3 runs after a warm-up. The CPU run restarted the daemon and loaded the model again before its speed run.

Package	Device	Decode	First token	Daemon's GPU memory, highest
cuda-int4-8k	RTX 5080 Laptop, CUDA EP	58.8 tokens/s	246 ms	13,294 MiB
cpu-int4	Core Ultra 9 386H, CPU EP	11.6 tokens/s	20,195 ms	—

The laptop installed a Windows update at 17:31 the same day (build 26200.9711). Foundry Local, the CUDA DLLs, the GPU driver and the packages stayed the same, and the answers stayed the same word for word (results/r8_answers_vs_before.txt in the GitHub repo). Decode got slower. On a quiet PC, cuda-int4-8k decodes at 32 tokens/s, and at 40 to 42 with the daemon limited to the four P-cores (processor affinity). Its decode follows the daemon's CPU placement, and one GPU reading per run showed 21 to 43 % utilization, so the limit is on the CPU side. cpu-int4 decodes at 9.1 tokens/s with 33,072 ms to the first token. What the update changed on the CPU side is not established.

Limits

• Only text prompts were checked. The vision and audio encoders load; no image or audio prompt was run.
• On a 16 GB GPU, keep the context short. With 131,072 tokens, in a run before the Windows update, the int4 package filled the GPU over the 24 prompts and then took 15 s to the first token (43.3 tokens/s); after the update it rose only to 8,230 MiB over the 24 prompts and to 13,350 MiB in the speed run, with 249 ms to the first token. With 8,192 tokens the daemon held 13,294 MiB at most.
• Foundry Local 0.13.0's OpenVINO EP fails to load these packages on the Intel GPU, the NPU and the CPU, before and after the update (microsoft/Foundry-Local#1261), and local models do not take the WebGPU EP. Outside Foundry Local, ONNX Runtime GenAI 0.19.0 runs cpu-int4 with the OpenVINO EP 1.8.1 on the Intel GPU (7.9 tokens/s before the update).
• The NPU does not compile these graphs (dynamic shapes; with enable_causallm, 277 subgraphs).

License

Apache License 2.0, as Gemma 4.