granite-4.1-8b — LiteRT-LM

ibm-granite/granite-4.1-8b converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime: an int8 file and an int4 file, both carrying the GPU-composite graph (Conversion notes). Converted with litert-torch 0.9.4 and tested on litert-lm 0.18.0 on a Mac Studio M4 Max, where the int4 file decodes at 58.3 tok/s on the GPU, and on a Galaxy S26, where it decodes at 5.3–5.4 tok/s on the GPU with the prebuilt Android binary named below (measured 2026-10-15). On litert-lm 0.17.1 both files answered a plain run with "Paris" (one CPU process each).

IBM describes Granite-4.1-8B as an 8B-parameter long-context instruct model finetuned from Granite-4.1-8B-Base on open instruction datasets and internally collected synthetic data, with supervised finetuning and reinforcement-learning alignment for tool calling, instruction following and chat; released April 29, 2026. It is a dense decoder-only transformer (GQA, RoPE, SwiGLU, RMSNorm, shared input/output embeddings): 40 layers, hidden size 4096, 32 attention heads with 8 KV heads, a 100,352-token vocabulary with tied embeddings, a 128K context (4096 in these files); 8,791,592,960 bf16 parameters by the Hub's count, which includes a stored copy of the tied output table (8.38 B without it). IBM lists twelve languages, from English to Chinese.

File	Recipe	Size	sha256
granite-4.1-8b_int8.litertlm	int8 dynamic on linears, float activations; the embedding table as a separate section	8.95 GB (8,949,862,832 bytes)	3ed1e7d4a3429e75a77b7445cbc4ce051aba888104eb254dcede457b5dfc70d9
granite-4.1-8b_int4.litertlm	int4 blockwise-32 + OCTAV on linears, int8 embedding; the embedding table as a separate section	5.15 GB (5,151,986,096 bytes)	7de1d77fb951572d12b8b17984d6418909b841e582d1df7866c5d08b3920ea60

On GSM8K (n=100) bf16 scored 96, the int8 file 95 and the int4 file 93, both on the Mac GPU (table in Accuracy); the int4 file is 42.4 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int4 GPU	5.81	58.7 and 59.6 (two runs)	stopped by our memory rule (Performance)
int8 GPU	6.98	not run (Performance)	not run: the file is larger than the phone's RAM
int8 CPU	35.3	not run (Performance)	not run (Performance)
int4 CPU	41.8	not run (Performance)	not run (Performance)

So int4 is the faster file on both GPUs measured; on the Mac CPU int8 is faster, its prefill being twice int4's. Not measured on iOS: the int8 file's main weights section is 8,528,914,704 bytes (7.94 GiB) and the int4 file's 4,731,041,056 bytes (4.41 GiB), both over 2 GiB; the embedding table sits in a separate 419,484,080-byte section, as in the granite-4.1-3b conversion, whose card explains the iOS reason.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each): int8 8/8 on CPU and GPU, int4 8/8 on CPU and GPU, bf16 (transformers) 8/8.

All 7 decoder signatures of both files were fully delegated on the Mac GPU (each 4-op embedding signature runs one op on the CPU), and the int4 file's 7 decoder signatures on the Galaxy S26 GPU (1860 / 1661 / 1693 ops on the OpenCL delegate), where it answered "What is the capital of France? Answer in one sentence." with "The capital of France is Paris." in both runs. A prompt-length check (18 lengths from 18 to 60 tokens, a fresh engine each, both files, Mac CPU and GPU) returned the expected word every time with the engine's token count equal to transformers'; a three-turn conversation on both files and both backends carried the first turn's word into the second and rewrote the second as a question in the third (the int8 file's three GPU replies equal the bf16 model's); in a long-output check (four prompts, greedy and the runtime's default sampling, up to 512 tokens, CPU) no reply of either file repeated a sentence, and two of the eight replies per file reached the 512-token cap.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's template (no system turn), greedy, at most 1024 output tokens, a fresh process per question (litert-lm 0.18.0 Python API, Mac GPU); bf16 in transformers on MPS with the same prompts. IFEval was not run for this model. Scores are comparable only within this table, not with IBM's 92.49 (8-shot, a different prompt setup).

Configuration	GSM8K (n=100)
bf16 reference (transformers, MPS)	96
int8, litert-lm Mac GPU	95
int4, litert-lm Mac GPU	93

Against bf16, one question differs for int8 (right only in bf16) and three for int4 (all three right only in bf16); every question bf16 missed is missed by both files. The files run the GPU with the runtime's default fp16 activations: with fp32 activations declared the int8 file gave the same eight check answers and ran 15 % slower, so the files declare none. Before the GPU-composite graph rewrite the same weights scored 95 and 93, with no question changing between right and wrong.

Usage

litert-lm run ./granite-4.1-8b_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./granite-4.1-8b_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./granite-4.1-8b_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./granite-4.1-8b_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm benchmark ./granite-4.1-8b_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

All four run commands answered with the two dates (int8: "The dates extracted from the text are: 1. May 3 2. May 17"; int4: "The dates extracted from the text are: - May 3 - May 17"), with no chat marker and no thought block. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (8.40 GB for int8, 4.73 GB for int4). The bundle sets no sampler; run passes one only when you give --top-k, --top-p, --temperature or --seed. For Android, see android-gpu.md.

The chat template is IBM's chat_template.jinja byte for byte. Unlike granite-4.0-micro's it has no default system message, so a conversation without one has no system turn. No start token (the prompt begins with <|start_of_role|>; bos is eos for this tokenizer, and the engine's prefill token count equals transformers'), the only stop token <|end_of_text|> (id 100257), no thinking declared (the tokenizer has <think> tokens; the template never writes them). The runtime renders nine of ten conversation shapes exactly as transformers does. The exception: a tool result passed as a Python-API content part renders empty, and the model answered without it ("clear with a temperature of 22°C and a light breeze" for a tool that had said sunny); passed as the tool message's text it was used ("sunny with a temperature of 22°C" in both files).

Tool calling: with a get_weather tool the template writes IBM's tools system turn and both files wrote <tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call> as text, which the app has to parse (the bundle's generic model type has no tool-call parser); IBM's documents turn renders for documents passed as context (int8: "The Eiffel Tower is 330 metres tall."). Engine reuse (int8 file, Python API, Mac GPU, greedy; the shape of LiteRT-LM #3165, a second conversation on an Engine that already served one): the second conversation gave the same replies as on a fresh Engine in 3 of 3 runs; this model keeps only a KV cache, no recurrent state, so one Engine can serve one conversation after another.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two passes per file, the median of the two; GPU is WebGPU on Metal. Init is the CLI's printed init time of the first iteration; in litert-lm 0.18.0 it sums overlapping init phases (LiteRT-LM #4057) and reads about twice the engine build measured from the log's timestamps (on this batch: a 9B Mixed INT4 GPU run printed 89.2 s for a 44.5 s build, a granite-4.1-8b int4 GPU run 7.38 s for 3.66 s; the Python API built the 9B GPU engines in 43–45 s).

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	718.0	46.07	1.448	5.51
int4 GPU	719.6	58.31	1.440	7.47
int8 CPU	110.60	9.82	9.361	64.21
int4 CPU	51.50	11.68	19.976	18.19

Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, Android 16, MemTotal 11,389,756 kB = 11.66 GB): the same workload through the prebuilt litert_lm_advanced_main (sha256 adac974b…, 40,382,152 bytes) published 2026-09-18 in LiteRT-LM's public binary bucket (binaries/latest/android_arm64/litert_lm/), with --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true. Rates are the mean of iterations 2 and 3 (GPU: LITERT_CL); init warm = the benchmark process, cold = the first process after the compiled-program caches were removed; peak = the largest of the benchmark process's Peak system ram usage lines. The int4 file was benchmarked twice, in two sessions on different hours.

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init warm / cold (s)	Peak (MB)
int4 GPU, run 1	91.43	5.40	11.69	24.90 / 30.15	2,038
int4 GPU, run 2	90.43	5.30	11.80	24.30 / 25.97	2,136

The phone was warm for both runs: each benchmark started at thermal status 3 (skin 45 °C) with the GPU clock ceiling at 1300 MHz, and the ceiling fell to 342 MHz inside the runs (80 of 84 and 83 of 85 clock samples below 1300 MHz; GPU temperature up to 90.7 °C). The exported graph before the GPU-composite rewrite, run in the same two sessions in the opposite order, gave prefill 86.07 and 72.22 tok/s and decode 3.87 and 3.63 tok/s: on this phone the rewrite's decode gain is 1.4×.

The int8 file has no Galaxy S26 row: on 2026-10-14 another 8B int8 file of the same recipe class (8.64 GB) dropped this phone off USB during its GPU engine warm-up, and after such a drop we do not start 8B int8 engines on it; that is a rule of ours, not a measured limit of the phone. No CPU rows on either phone: the earlier granite-4.0-micro conversion measured a Galaxy S26 CPU peak of 1.5 × the int8 file and 2.0 × the int4 file, which puts both files above the S26's 11.66 GB and the int4 file above the Pixel 8a's 7.94 GB. Pixel 8a (Tensor G3, Mali-G715, Android 16, MemTotal 7,754,700 kB): the int4 file's one GPU try was stopped by our memory rule (a run ends at the first MemAvailable reading under 400,000 kB; here 327,604 kB, 55 s after the start, after prefill_1024 had been delegated 1860/1860 to LITERT_CL; the runtime prepares weights on the CPU side for Mali GPUs); the int8 file was not tried (8.95 GB, more than the phone's RAM). Not run on an NPU: these dynamic-range int8 and blockwise int4 graphs have no HTP path (the NPU path takes static-range or float graphs; the Pixel 8a log said the NPU accelerator could not be loaded).

Conversion notes

• Source and stack: ibm-granite/granite-4.1-8b at revision 1504002f (every file checked against the Hub's record); litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1, torch 2.13.0; packed by the litert-lm 0.18.0 CLI. granite41_work/convert_granite41_8b.sh in hf-to-litertlm is the one command; run as published on 2026-10-16 it rebuilt both files, all five sections byte-identical (only the packer's uuid and timestamp differ), as the conversion run of 2026-10-14 behind these files had.
• Lightweight export: the default export path reached 98.8 GB on a 128 GB Mac in the converter passes before a RAM guard stopped it; the script exports with litert-torch's lightweight conversion and a 1e8-byte lazy-constant threshold (with the 1 MiB default the converter left the 40 o_proj weights unquantized behind a constant-only multiply by Granite's residual multiplier 0.22). Every FULLY_CONNECTED weight is an int8 or int4 constant, no op has all-constant inputs, and Granite's multipliers stay scalar multiplies on activations (× 12 on the input embeddings, × 0.0078125 in attention, × 0.22 on each down projection, × 1/16 on the logits). Export peaks 50–56 GiB.
• Order: export (6 prefill signatures, 1024 down to 1, plus decode, over a 4096-position KV cache; the embedding table as a separate section; no start token; IBM's template embedded) with the int8 recipe or the int4 recipe → int4 block-scale scan (281 blockwise tensors, 261,881,856 scales, none zero or subnormal) → metadata edit (stop token id 100257 only, max_num_tokens 4096, no thinking or function-calling flag; the exporter's 14 string stops dropped) with every other section proven unchanged → the GPU-composite graph rewrite (hf-to-litertlm tools/gpu_graph/, --decomp exporter --mask add_bcast): each layer's two DYNAMIC_UPDATE_SLICE KV writes become one odml.cache_update and the attention products odml.runtime_bmm (280 and 548 composites per file), every weight byte and every other section unchanged, the main section 884,736 bytes larger. On the Mac GPU the rewrite took the 1024 / 256 workload from 8.20 to 6.95 s (int8) and from 6.93 to 5.78 s (int4) in the same measurement windows (the Performance table is a later re-measurement of the shipped files); on the Mac CPU the difference stayed inside the run-to-run spread.

License and changes

Distributed under the Apache License 2.0 inherited from the base model (the model repo carries the license in its README metadata and no separate LICENSE file). Changes from the original work: exported to the LiteRT-LM format and quantized to int8 and int4 as described above, with the GPU-composite graph rewrite applied to both files; chat template IBM's, stop token reduced to <|end_of_text|>, max_num_tokens 4096. This conversion was made and is published independently of IBM.