CPU speed, RAM and quantisation loss: arm A (GGUF, llama.cpp) vs arm C (our kernel), laptop, 2026-10-14
Laptop: Intel Xeon W-11955M (8 cores / 16 threads, 24 MB L3, measured DRAM read bandwidth 36-40 GB/s), 8 threads, Windows. llama.cpp b11515 (official Windows CPU build). Arm C kernel: kernel/cgemma.c v3 (gcc -O3 -mavx2 -mfma -mf16c, NT=8, PIN=1), ternary GEMV ported from the SHADOW-50M kernel, 1-bit table readout straight off the table bits. Generation = greedy, one token at a time, 128-token context; "prefill" in our kernel is sequential (no batching), llama.cpp's is batched.
Speed and RAM (same 8 threads)
| model | format / runtime | file | peak RAM while generating | generate tok/s | prefill tok/s |
|---|---|---|---|---|---|
| A, learned table, float | GGUF f16, llama.cpp | 543 MB | 723 MB | 62 (llama-bench 61.96 +/- 1.38) | 1,105 |
| A | GGUF Q8_0, llama.cpp | 292 MB | 471 MB | 116 (115.9 +/- 0.6) | 1,192 |
| A | GGUF Q4_K_M, llama.cpp | 253 MB | 446 MB | 130 (130.1 +/- 1.3) | 1,313 |
| C, frozen table, ternary | our kernel v3 | 46 MB | 62-67 MB | 615-740 (256-token runs: 668 / 684 / 614) | ~1,450 |
| C | our kernel v2 (two-pass 2-bit readout) | 46 MB | 100 MB | 300-327 | ~1,265 |
| C | our kernel v1 (scalar attention, LUT readout) | 46 MB | 61 MB | 117 | - |
Peak RAM for A measured with psutil polling of llama-completion; for C from the process peak working set. Earlier runs of v2 on this laptop varied by +/-10% with background load; v3 figures above are from 256-token generations.
Does the GGUF quantisation of A lose quality? (llama-perplexity, export/val_300k.txt, n_ctx 512, 586 chunks)
| arm A format | perplexity | vs f16 |
|---|---|---|
| f16 | 23.857 +/- 0.187 | - |
| Q8_0 | 23.864 +/- 0.187 | +0.03% (no measurable loss) |
| Q4_K_M | 23.966 +/- 0.188 | +0.46% (small, inside the error bar) |
So Q8_0 is a free size/speed win for A (the table stays 16-bit in every GGUF type: ~250 MB of the Q4_K_M file is the embedding, which is why A never gets below 446 MB of RAM). Comparing at Q8_0 is therefore fair to A: C is still 5.3-6.4x faster and uses 7x less RAM. Note: this f16 perplexity (23.86) differs slightly from the earlier one-off 23.49 run; the three numbers above come from one identical setting and are the ones to quote.
Kernel v3 correctness (golden check vs PyTorch fp32, 64 real validation tokens)
top-1 agreement 100%, top-5 overlap 98.4%, next-token loss 4.4946 vs 4.4826. The 0.012 drift comes from the f16 KV cache (llama.cpp's default for A as well); with the f32 KV cache of v2 the loss was 4.4858. The readout is exact for the argmax: the int8 first pass bounds every row, the exact second pass runs only on rows that can still win (0.9 rows per token).
Where the time goes (v3, generate, ms/token at ~160-token context)
| section | ms | note |
|---|---|---|
| ternary GEMVs (attention + FFN) | 0.52 | 25 MB of weights per token, DRAM-bound (0.39 during prefill when more stays in L3) |
| table readout | 0.50-0.65 | 17 MB per token, DRAM-bound (0.30 with the table hot in L3) |
| attention | 0.12 | positions split across the 8 threads, flash-style merge |
| norms / int8 quantisation / RoPE | 0.12 | serial on the main thread |
| embed + output adapters | 0.03 | |
| barriers | 0.06 | 91 per token at 0.7 us |
Every token streams ~44 MB (body 25 + table 17 + seal/logits 2) through a 24 MB L3, so on this laptop the floor is 44 MB / 38 GB/s = 1.15 ms = ~860 tok/s; v3 is at 1.45-1.6 ms. Steering the cache (non-temporal streaming of the table or of the body) was measured and is slower. Faster memory or an L3 that holds the 46 MB (server CPUs) lifts the ceiling directly; the body alone already runs at 1,250 tok/s here when the table pass is skipped.