File size: 3,272 Bytes
b26cee8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | # HANDOFF — flint Q4_1 CPU optimization (this machine)
Updated: 2026-10-31 (UTC+1) — periodic checkpoint (uploaded every ~5 min while working).
## Read this first
This is the live handoff for continuing the Q4_1 optimization of
`HelloSun/flint_cpu_opt`. The static docs are `README.md`,
`benchmarks/q41-summary.md` (Q4_1) and `benchmarks/summary.md` (Q4_K_XL).
## Machine (the one this work targets)
- Intel Xeon Platinum 8559C, 2 sockets, 192 logical CPUs online.
- **cgroup limit: 16 CPUs** (`cpu.max = 1600000/100000`), `nproc` = 16.
- 2 TiB RAM host; memory guard: agent-started processes must stay < 80 GiB RSS.
- Flags: AVX512-F/BW/VL/DQ, AVX512-VNNI, AVX512-FP16, AMX-INT8/AMX-BF16, F16C.
- Same CPU model as the numbers in `machine.txt` (16 threads), but the host is
shared, so single runs vary ~±8 %.
- Repo base: llama.cpp `de5122d879d00d2895c36be7e0cee94105aeeabd` with
`patches/flint-cpu-opt.patch` applied in `llama.cpp/`.
## Current model/server
- Model: `flint-int4-q4_1.gguf` (24.2 GiB, every weight Q4_1; token_embd +
router F32), downloaded from `RecursiveSI/flint-int4-gguf`.
- `llama-server` running on `127.0.0.1:20000`, alias `flint-q4_1`,
`-t 16 -fa on --jinja -c 32768` (started by `build.sh`).
- pi coding agent is configured to use it (`pi-local`).
## Current Q4_1 kernel state (already in the patch)
1. `ggml_vec_dot_q4_1_q8_1` (decode): AVX512-VNNI, 2 blocks per ymm, 4
blocks/iteration, prefetch `GGML_Q41_PF` (default 192 blocks, T1 hint).
2. Tiled VNNI GEMM for Q4_1 prefill: per-32-block f32 `sd`/`sm`, `Q41`
template flag, Q8_1 activations.
3. Reported on 8559C/16t: pp256 266, pp2048 297, tg128 55.7 t/s; stock
183/189/48.3. KLD inside stock's own noise floor.
## Plan / next steps
- [x] Read existing docs and machine context.
- [ ] Build `llama-bench` + `llama-cli` (only `llama-server` was built).
- [ ] Baseline on **this** machine: interleaved A/B, `taskset -c 0-15`,
`-t 16 -fa on -b 1024 -ub 1024`, Q4_1 file.
- [ ] Sweep `GGML_Q41_PF` (e.g. 128/192/256/320) and thread counts on the
real model (not only microbench).
- [ ] Try one decode-side improvement (memory-bound path) and/or an AMX
prefill variant; keep only if it beats noise.
- [ ] A/B compare, record raw CSV/log under `benchmarks/`.
- [ ] Update `README.md` + `benchmarks/q41-summary.md` only with measured,
reproducible numbers.
- [ ] Restart `llama-server` with the final best settings.
## How to reproduce
```bash
cd /root/flint_cpu_opt
cmake --build llama.cpp/build -j16 --target llama-bench llama-cli
taskset -c 0-15 llama.cpp/build/bin/llama-bench -m flint-int4-q4_1.gguf \
-t 16 -fa on -lm mlock -b 1024 -ub 1024 -p 256,2048 -n 128
```
## Notes / gotchas
- Do **not** run 100 parallel Tor/open code probes here; memory guard kills
processes over 80 GiB / single child over 64 GiB.
- The llama-server is serving pi; stop it before a clean benchmark, restart
afterwards (`bash build.sh` reuses an already-running server).
- Host is shared: use medians of >= 3 interleaved rounds; differences under
~8 % are not a win.
- Kernel changes must keep integer dot products exact; only f32 accumulation
order may change. Check `benchmarks/graph_accuracy_test.cpp` and KLD if a
numeric path changes.
|