| # CPU speed, RAM and quantisation loss: arm A (GGUF, llama.cpp) vs arm C (our kernel), laptop, 2026-10-14 |
|
|
| Laptop: Intel Xeon W-11955M (8 cores / 16 threads, 24 MB L3, measured DRAM read bandwidth 36-40 GB/s), 8 threads, Windows. |
| llama.cpp b11515 (official Windows CPU build). Arm C kernel: kernel/cgemma.c v3 (gcc -O3 -mavx2 -mfma -mf16c, NT=8, PIN=1), |
| ternary GEMV ported from the SHADOW-50M kernel, 1-bit table readout straight off the table bits. Generation = greedy, one |
| token at a time, 128-token context; "prefill" in our kernel is sequential (no batching), llama.cpp's is batched. |
|
|
| ## Speed and RAM (same 8 threads) |
|
|
| | model | format / runtime | file | peak RAM while generating | generate tok/s | prefill tok/s | |
| |---|---|---|---|---|---| |
| | A, learned table, float | GGUF f16, llama.cpp | 543 MB | 723 MB | 62 (llama-bench 61.96 +/- 1.38) | 1,105 | |
| | A | GGUF Q8_0, llama.cpp | 292 MB | 471 MB | 116 (115.9 +/- 0.6) | 1,192 | |
| | A | GGUF Q4_K_M, llama.cpp | 253 MB | 446 MB | 130 (130.1 +/- 1.3) | 1,313 | |
| | **C, frozen table, ternary** | **our kernel v3** | **46 MB** | **62-67 MB** | **615-740** (256-token runs: 668 / 684 / 614) | ~1,450 | |
| | C | our kernel v2 (two-pass 2-bit readout) | 46 MB | 100 MB | 300-327 | ~1,265 | |
| | C | our kernel v1 (scalar attention, LUT readout) | 46 MB | 61 MB | 117 | - | |
| |
| Peak RAM for A measured with psutil polling of llama-completion; for C from the process peak working set. |
| Earlier runs of v2 on this laptop varied by +/-10% with background load; v3 figures above are from 256-token generations. |
| |
| ## Does the GGUF quantisation of A lose quality? (llama-perplexity, export/val_300k.txt, n_ctx 512, 586 chunks) |
| |
| | arm A format | perplexity | vs f16 | |
| |---|---|---| |
| | f16 | 23.857 +/- 0.187 | - | |
| | Q8_0 | 23.864 +/- 0.187 | +0.03% (no measurable loss) | |
| | Q4_K_M | 23.966 +/- 0.188 | +0.46% (small, inside the error bar) | |
|
|
| So Q8_0 is a free size/speed win for A (the table stays 16-bit in every GGUF type: ~250 MB of the Q4_K_M file is the |
| embedding, which is why A never gets below 446 MB of RAM). Comparing at Q8_0 is therefore fair to A: C is still 5.3-6.4x |
| faster and uses 7x less RAM. Note: this f16 perplexity (23.86) differs slightly from the earlier one-off 23.49 run; the |
| three numbers above come from one identical setting and are the ones to quote. |
|
|
| ## Kernel v3 correctness (golden check vs PyTorch fp32, 64 real validation tokens) |
|
|
| top-1 agreement 100%, top-5 overlap 98.4%, next-token loss 4.4946 vs 4.4826. The 0.012 drift comes from the f16 KV cache |
| (llama.cpp's default for A as well); with the f32 KV cache of v2 the loss was 4.4858. The readout is exact for the argmax: |
| the int8 first pass bounds every row, the exact second pass runs only on rows that can still win (0.9 rows per token). |
|
|
| ## Where the time goes (v3, generate, ms/token at ~160-token context) |
|
|
| | section | ms | note | |
| |---|---|---| |
| | ternary GEMVs (attention + FFN) | 0.52 | 25 MB of weights per token, DRAM-bound (0.39 during prefill when more stays in L3) | |
| | table readout | 0.50-0.65 | 17 MB per token, DRAM-bound (0.30 with the table hot in L3) | |
| | attention | 0.12 | positions split across the 8 threads, flash-style merge | |
| | norms / int8 quantisation / RoPE | 0.12 | serial on the main thread | |
| | embed + output adapters | 0.03 | | |
| | barriers | 0.06 | 91 per token at 0.7 us | |
|
|
| Every token streams ~44 MB (body 25 + table 17 + seal/logits 2) through a 24 MB L3, so on this laptop the floor is |
| 44 MB / 38 GB/s = 1.15 ms = ~860 tok/s; v3 is at 1.45-1.6 ms. Steering the cache (non-temporal streaming of the table or |
| of the body) was measured and is slower. Faster memory or an L3 that holds the 46 MB (server CPUs) lifts the ceiling |
| directly; the body alone already runs at 1,250 tok/s here when the table pass is skipped. |
|
|
