G9v3-3B — LiteRT-LM

ai9stars/G9v3-3B converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime. Tested on litert-lm 0.18.0 on a Mac; the phone rows ran the prebuilt Android binary named below. On litert-lm 0.17.1 both files answered a plain run with "The capital of France is Paris." (one CPU process each).

AI9Stars describes G9v3 as a dense 3B causal language model built for local deployment and resource-constrained use: everyday assistant tasks, coding, tool-use workflows and reasoning where a compact model is preferred; English and Chinese. It is a standard LlamaForCausalLM: 2,988,656,640 parameters, 52 layers, hidden size 2048, 16 attention heads with 2 KV heads, a 130,560-token vocabulary with a separate output head, a 131,072-token context (4096 in these files). The model can reason before it answers; in these files thinking is off by default and switched on per request (Usage).

File	Recipe	Size	sha256
G9v3-3B_int8.litertlm	int8 dynamic on linears, float activations; embedding table in its own section; GPU activations declared fp32	3.10 GB	c8c45d5bc3a5078290b38024651efda7b5a01cb1dce1d3b32284f9994d41afec
G9v3-3B_int4.litertlm	GPTQ-calibrated int4 (group 128) carried as blockwise-128 int4 on linears, int8 lm_head + embedding; GPU activations declared fp16	1.83 GB	ea2c773ec678d44ffca43c0fca21a46622fd812fb9b53706f82d966d02e3c468

On GSM8K (n=100, thinking off) bf16 scored 82, the int8 file 85 on the CPU and 84 on the GPU, the int4 file 81 on the GPU (table in Accuracy); the int4 file is 59.1 % of the int8 file's size. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int8 GPU	4.82	29.94	no row (Performance)
int4 GPU	4.84	16.41	36.76
int8 CPU	14.27	35.24	121.63
int4 CPU	14.48	35.63	130.37

The two files carry different GPU activation declarations because the accuracy measurements differ: the int8 file loses on the GPU with the runtime's default fp16 activations (GSM8K 69 against 85 on the CPU), so it declares fp32 and runs slower on the GPU; the int4 file keeps its score with fp16 (81, against 78 with fp32) and runs fastest that way. The int8 file has no Pixel 8a GPU row (Performance). Not measured on iOS: the int8 file's decoder weights section is 2,822,005,664 bytes, over 2 GiB.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each, thinking off): int8 8/8 on CPU and 7/8 on GPU, int4 7/8 on CPU and GPU; every miss is the rhyme question ("white" for "blue").

All 7 decoder signatures of both files were fully delegated on the Mac GPU and on the phone GPUs measured (the embedding section runs on the CPU). On both phones, every leg below answered "What is the capital of France? Answer in one sentence." with "Paris". The runtime prefilled exactly the tokens of the Hugging Face template, with one <s> at the start (three prompts, both files); 39 prompt lengths from 22 to 60 tokens were answered by both files; and a three-turn conversation (word → sentence → question) passed on both files with thinking off and on the int8 file with thinking on. With thinking on, the int4 file's third turn returned no answer: its trace ran to the turn's 1536-token output limit without closing.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, thinking off, greedy, at most 1024 output tokens, a fresh process per question (litert-lm 0.18.0 Python API); bf16 in transformers on MPS with the same prompts. No accuracy was measured with thinking on; IFEval-lite was not run. Scores are comparable only within this table.

Configuration	GSM8K (n=100)
bf16 reference (transformers, MPS)	82
int8, litert-lm CPU	85
int8, litert-lm Mac GPU (fp32 activations, as shipped)	84
int8, Mac GPU with fp16 activations (not shipped)	69
int4, litert-lm Mac GPU (fp16 activations, as shipped)	81
int4, Mac GPU with fp32 activations (not shipped)	78
int4, litert-lm CPU	78

Against bf16, int8 on the CPU missed 1 question bf16 got right and got 4 the other way; int8 on the GPU 4 / 6; int4 on the GPU 4 / 3; int4 on the CPU 7 / 3. The int4 weights are GPTQ-calibrated: a data-free blockwise int4 export of this checkpoint scored 58 on the same questions and is not shipped.

Usage

litert-lm run ./G9v3-3B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./G9v3-3B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./G9v3-3B_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./G9v3-3B_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./G9v3-3B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --thinking true
litert-lm benchmark ./G9v3-3B_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

The four plain run commands answered with the two dates (int8: "The dates mentioned in the text are May 3 and May 17."; int4: a two-item list), with no chat marker and no thought block. With --thinking true the reply begins with a [thought] block ("We are given: … We need to extract the dates …") before the answer. Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (2.73 GB for int8, 1.54 GB for int4). The bundle sets no sampler, so run is greedy unless you pass sampling flags, and on the 0.18.0 CLI --temperature and --top-p take effect only together with --top-k (AI9Stars recommends temperature 0.7 / top_p 0.95 with thinking off and 0.9 / 0.95 with thinking on). For Android, see android-gpu.md.

The embedded chat template is built on LiteRT-LM's models/qwen3 template, reading message content as parts; it renders the same text as AI9Stars's template through transformers for 10 of 11 conversation shapes, the difference being a request that does not set the thinking switch: the bundle treats it as thinking off, the vendor template leaves it to the model. Stops are the model's two end tokens (ids 1 and 130073), the start token is <s>. Thinking on puts the trace in the bundle's thought channel (<think> … </think>), printed by the CLI between [thought] and [/thought]: on three questions the trace closed and the answer followed in 3 of 3 runs for both files.

Tool calling is not declared: with a get_weather tool both files wrote the vendor's XML form as text, <function name="get_weather"><param name="city">Paris</param></function>, for the app to parse; the rendered tool list matched the vendor template's token count. Engine reuse (int8, Python API, CPU, greedy): a second conversation on an Engine that had served another gave the same replies as on a fresh Engine, including with the shared-prefix condition of LiteRT-LM #3165; this model keeps only a KV cache, no recurrent state.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two passes per file, the median of the two; GPU is WebGPU on Metal. Init is the first iteration's engine init. The int4 CPU row was measured on the same weights with the fp32 declaration (the CPU path does not read it); the int4 weights declared fp32 on the Mac GPU took 5.29 s for the workload against 4.84 s as shipped.

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	1429.0	62.43	0.73	4.3
int4 GPU	1506.9	61.49	0.70	4.4
int8 CPU	261.5	24.73	3.96	19.5
int4 CPU	174.6	29.73	5.93	3.7

Phones, Android 16: Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, MemTotal 10.9 GiB) and Pixel 8a (Tensor G3, Mali-G715, MemTotal 7.4 GiB), the same workload through the prebuilt litert_lm_advanced_main (sha256 adac974b…, 40,382,152 bytes) published 2026-09-18 in LiteRT-LM's public binary bucket (binaries/latest/android_arm64/litert_lm/), with --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true. Rates are the mean of iterations 2 and 3 (GPU: LITERT_CL). Init cold = the first process after the compiled-program caches were removed, warm = the benchmark process; peak = the runtime's Peak system ram usage line. Before each Galaxy S26 leg the phone rested until its clocks were at their top and the thermal status was 0 (up to 300 s).

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	113.9	12.22	9.1	8.3 / 5.4	1002
int4 GPU	402.9	18.46	2.6	7.5 / 5.0	1182
int8 CPU	108.3	9.93	9.6	3.3 / 0.3	4147
int4 CPU	63.4	13.14	16.3	3.0 / 1.0	2939

The clocks lowered during every run even after the rest: the GPU clock ceiling was lowered in 41 of 46 samples (int8) and 18 of 26 (int4), the CPU clock in 45 of 46 (int8) and 45 of 47 (int4), read every 2 s. The int4 weights declared fp32 took 25.09 s on this GPU against 16.41 s as shipped.

Pixel 8a	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	—	—	—	—	—
int4 GPU	125.1	8.96	8.3	20.6 / 6.7	1639
int8 CPU	20.85	3.53	49.4	21.8 / 0.9	3847
int4 CPU	16.3	3.79	63.1	4.4 / 2.0	2642

The int8 file has no Pixel 8a GPU row: the harness stops a process when the phone's MemAvailable falls below 400,000 kB, and it did so for the int8 file with fp32 activations (lowest 300,700 kB in the first process, 0 in the benchmark) and with fp16 activations (393,560 kB); the int4 file's GPU run stayed at 2,179,860 kB or more. The int8 CPU row is a run with no rest before it; a rested run of the same file was slower (152.59 s for the workload against 121.63 s), as the CPU clock was lowered in every sample either way (110 of 110; 134 of 137). The Mali GPU clock is not readable on this phone. Not run on the NPU.

Conversion notes

• Source and stack: ai9stars/G9v3-3B at revision d9553445 (every file checked against the Hub's record); litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1 (export); gptqmodel 7.3.5 on torch 2.14.1 (MPS) for the calibration; packed by the litert-lm 0.18.0 CLI. g9v3_work/convert_g9v3_3b.sh in hf-to-litertlm rebuilt both files, all five sections byte-identical (only the packer's uuid and timestamp differ); the GPTQ step reproduces bit for bit on the Mac.
• int4: a data-free blockwise int4 export scored 58 on GSM8K and its zero and subnormal block scales did not load on the CPU, so the int4 file is calibrated: gptqmodel (4 bits, group 128, symmetric, 256 chat samples from ultrachat_200k, alpaca and the GSM8K training split at pinned revisions, rendered with thinking off; the script rebuilds the set, which is not redistributed because of alpaca's licence), dequantized, then exported with the GPTQ scales recovered as blockwise-128 int4; 781 block scales below 2^-14 were floored to it.
• Order: export (6 prefill signatures, 1024 down to 1, plus decode, over a 4096-position KV cache; the embedding table as its own section; the bundle template embedded) with the int8 recipe or the GPTQ-recovered int4 → metadata edit (stops ids 1 and 130073, thought channel <think> / </think>, thinking declared, max_num_tokens 4096) with every other section proven unchanged → the GPU activation type declared on each file (int8 fp32, int4 fp16).

License and changes

Distributed under the Apache License 2.0 inherited from the base model (the model repo's README metadata and closing line). Changes from the original work: exported to the LiteRT-LM format and quantized to int8 (dynamic) and int4 (GPTQ-calibrated, blockwise recovery) as described above; the chat template replaced by the bundle template (thinking off by default, content as parts), stops and the thought channel set in the bundle metadata, the embedding table externalized, a GPU activation type declared per file. This conversion was made and is published independently of AI9Stars.