Tigrinya Gemma 4 E4B — GGUF Q4_K_M

A four-bit build of Ha-Te-21/tigrinya-Gemma_4_E4B for CPU inference. It runs offline on a consumer laptop with no graphics hardware and no network connection, which is what it was built for.


File	`sft_v1_q4km.gguf`
Quantisation	Q4_K_M, 5.67 effective bits per weight
Size	5,041.55 MiB (16-bit export is 14,235.64 MiB)
Memory envelope in use	5.14 GB peak resident set
Built with	`llama.cpp` at commit `0190529e`

Running it

pip install llama-cpp-python --only-binary=:all: \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

from llama_cpp import Llama

llm = Llama(model_path="sft_v1_q4km.gguf", n_ctx=4096, n_threads=8)

BOT, EOT = "<start_of_turn>", "<end_of_turn>"
prompt = f"{BOT}user\n{your_question}{EOT}\n{BOT}model\n"

out = llm(prompt, max_tokens=768, temperature=0.6, top_p=0.9, top_k=40,
          repeat_penalty=1.15, stop=[EOT, BOT, "<eos>"])
print(out["choices"][0]["text"])

The turn format is required. Without it the model runs in continuation mode and performs far worse than it should.

On a hybrid processor, set n_threads to the performance-core count, not the logical processor count: the runtime synchronises threads at each layer, so the slowest thread paces the layer. Measured 4.8 against 4.4 tokens per second on a 2P + 8E part.

Measured performance

Three machines, CPU only, no graphics offload.

	32 GB host	16 GB host	8 GB host
Processor	i5-13420H	i5-1335U	i7-7600U
Installed memory	31.7 GB	15.6 GB	7.8 GB
Model load	2.5 s	2.4 s	18.2 s
Prefill	57.5 tok/s	17.8 tok/s	12.4 tok/s
Generation	14.0 tok/s	4.4 tok/s	4.2 tok/s
Peak resident set	5.14 GB	5.14 GB	3.92 GB
Lowest free memory	11.34 GB	3.76 GB	0.21 GB
Paged to disk	no	no	yes
150-word answer	38.8 s	123.4 s	129.3 s

The 8 GB machine never holds the whole model in memory and runs it by paging. Because the runtime memory-maps the file, most of the resulting traffic is re-reading clean pages rather than writing to the pagefile, which is why throughput degrades rather than collapsing. It is usable: a short answer takes about 25 seconds.

A note on the Tigrinya tax. Tigrinya costs 3.623 tokens per word on this tokenizer against 1.283 for English. A 150-word answer is 543 tokens in Tigrinya and 192 in English, so on the fastest machine above the Tigrinya user waits 38.8 seconds where the English user waits 13.7.

Output is byte-identical across all three machines under an identical call sequence: the sampler is seeded once at load and quantised CPU inference is deterministic.

Intended use

Best on supplied text — summarisation, extraction, question answering over a document. Not an oracle; see the limitations on the parent model card.

Limitation specific to this build

The accuracy of this quantised file has not been measured. The reported 37.23% on 865 held-out multiple-choice questions was measured on the bfloat16 checkpoint, not on this four-bit build. Quantisation is lossy and its effect here is uncharacterised. Treat the accuracy of this file as unknown until it is measured.