Model card: file comparison, results, accuracy, limitations

README.md

@@ -1,19 +1,110 @@
-# flint CPU optimization
+---
+license: apache-2.0
+pipeline_tag: text-generation
+base_model:
+- RecursiveSI/flint
+base_model_relation: quantized
+tags:
+- gguf
+- llama.cpp
+- cpu
+- avx512-vnni
+- amx
+- moe
+- flint
+---
 
-`flint-UD-Q4_K_XL.gguf` is the unmodified file from [unsloth/flint-GGUF](https://huggingface.co/unsloth/flint-GGUF).
-All changes are in `patches/flint-cpu-opt.patch` (llama.cpp commit in `patches/BASE_COMMIT`), in the style of
+# flint CPU optimization (llama.cpp patch + two GGUF files)
+
+[Recursive Flint](https://huggingface.co/RecursiveSI/flint) is a 39.8B-parameter mixture-of-experts coding model with about 2.5B
+active parameters per token (256 experts, top-6, 49 layers). This repo holds a **llama.cpp CPU patch** that speeds it up on
+Intel Xeon (AVX512-VNNI / AMX) and the two GGUF files it was tuned and measured on. **The GGUF files are unmodified**; every gain
+comes from code (`patches/flint-cpu-opt.patch`, base commit `de5122d87`). It is in the style of
 [HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT](https://huggingface.co/HelloSun/Qwen3.6-35B-A3B-GGUF_CPU_OPT).
 
-```
-./build.sh
+| file | source | quant | size |
+|---|---|---|---|
+| `flint-UD-Q4_K_XL.gguf` | [unsloth/flint-GGUF](https://huggingface.co/unsloth/flint-GGUF) | Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S) | 23.3 GiB |
+| `flint-int4-q4_1.gguf` | [RecursiveSI/flint-int4-gguf](https://huggingface.co/RecursiveSI/flint-int4-gguf) | Q4_1 for every weight (token_embd and router F32) | 24.2 GiB |
+
+## Which file to use
+
+Measured on Intel Xeon Platinum 8559C, 16 threads, with the patch (median of 3 interleaved rounds, tokens/s):
+
+| file | prefill pp256 | prefill pp2048 | decode tg128 | PPL (wikitext-2, 32x512) |
+|---|---|---|---|---|
+| Q4_K_XL, stock llama.cpp | 231 | 274 | 38.2 | 18.363 +- 0.795 |
+| **Q4_K_XL, patched** | **296** | **307** | 42.8 | 18.277 +- 0.787 |
+| Q4_1, stock llama.cpp | 183 | 189 | 48.3 | 18.260 +- 0.780 |
+| **Q4_1, patched** | 199 | 209 | **54.0** | 18.240 +- 0.779 |
+
+- **Long prompts / large contexts (prefill-heavy): Q4_K_XL**. Its K-quant weights use the tiled VNNI matmul; patched prefill is ~47 % faster than Q4_1.
+- **Chat and agent loops with a cached prefix (decode-heavy): Q4_1**. It reads fewer bytes per weight (0.625) and decodes ~26 % faster.
+- Quality is equivalent within measurement error: the PPL differences above are smaller than the +-0.8 standard error of a 32-chunk test.
+- Single runs on this shared host vary by about +-8 %; the numbers are medians, raw data in `benchmarks/`.
+
+## Gain of the patch (vs stock llama.cpp, same file)
+
+| file | pp256 | pp2048 | tg128 |
+|---|---|---|---|
+| Q4_K_XL | 1.28x | 1.12x | 1.12x |
+| Q4_1 | 1.09x | 1.10x | 1.12x |
+
+For Q4_K_XL most of this comes from the Qwen3.6 patch kernels (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, MoE tile grouping) also
+applying to flint; against the Qwen3.6 patch alone the flint-specific part is small (pp256 +12 %, decode +2 %, pp2048 equal).
+The flint-specific changes:
+
+1. **IQ4_XS x Q8_K `vec_dot` in AVX512-VNNI** (x86 had AVX2 only). Unsigned LUT (+128) with `vpdpbusd`, bias removed through `bsums`, L2 prefetch.
+2. **Q4_1 x Q8_1 `vec_dot` in AVX-512VL VNNI**. Two blocks per iteration, packed into one ymm with memory-source `vinserti128`, low/high nibbles each through one `vpdpbusd`, fp16 headers converted together. A 512-bit version using cross-lane shuffles was slower than stock and dropped.
+3. Q5_K 5th-bit merge with `vptestmb` + masked add (bit-identical to the previous version, no speed change).
+
+## Accuracy
+
+Wikitext-2 test, 32 chunks x 512, KL divergence against stock llama.cpp logits of the same file:
+
+| file | KLD patched vs stock | same top-1 | noise floor: stock vs stock with `-fa off` |
+|---|---|---|---|
+| Q4_K_XL | 0.0666 +- 0.0049 | ~92 % | 0.0762 (also `-ub 128`: 0.0700) |
+| Q4_1 | 0.0364 +- 0.0029 | 93.8 % | 0.0385 |
+
+The KLD is **inside the noise floor of stock itself**: flint is very sensitive to tiny numeric differences (changing only the
+attention path of stock gives the same KLD), so it does not indicate a defect. Supporting checks: a graph-level `mul_mat` /
+`mul_mat_id` test gives the same relative L2 error versus a float reference as stock (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q4_1 ~5.4e-3, Q8_0
+~5.4e-3). Integer dot products are exact; only float accumulation order changes. Why small differences get amplified was not
+verified (top-6-of-256 expert routing ties is a guess). The earlier version of this repo reported KLD 0.2058 for a different patch.
+
+## Where the time goes (why further kernel work is limited)
+
+Measured DRAM read bandwidth on this host is 125-175 GB/s with 16 threads (single thread 15 GB/s). Decode reads about 2 GB of
+weights per token (~1.6 GB dense Q8_0 attention/shared-expert/lm_head, ~0.5 GB for the 6 selected experts), i.e. ~90 GB/s effective:
+mostly memory-bound. Per-type effective bandwidth during decode (Q4_K_XL): Q8_0 dense ~100-170 GB/s (lm_head ~170), IQ4_XS experts ~100,
+Q5_K experts ~50. The Q5_K gap is not kernel throughput (cache-resident Q5_K and IQ4_XS run the same, 232 vs 218 GB/s); a
+prefetch-all-experts experiment did not help and was dropped; the cause was not found. Prefill is compute-bound; FlashAttention
+(24 %) and MoE matmul are the largest items. Q4_1 prefill still uses the per-row `vec_dot` path (60 % of prefill time is
+`MUL_MAT_ID q4_1`); adding it to the tiled GEMM would need per-32-block float scales in the tiled kernel and was not done.
+
+## Build and run
+
+```bash
+./build.sh        # clones llama.cpp at patches/BASE_COMMIT, applies patches/flint-cpu-opt.patch, builds with -DGGML_NATIVE=ON
 taskset -c 0-15 llama.cpp/build/bin/llama-cli -m flint-UD-Q4_K_XL.gguf -t 16 -fa on -lm mlock
 ```
 
-Measured on Xeon Platinum 8559C, 16 threads (median of 3 interleaved rounds, vs stock llama.cpp):
-pp256 231 -> 296 t/s (1.28x), pp2048 274 -> 307 (1.12x), tg128 38.2 -> 42.8 (1.12x).
-PPL 18.28 vs 18.36 stock (no degradation). KLD vs stock is 0.067 (top-1 ~92 %), which is the same as stock vs stock with a different numeric path (`-fa off` 0.076, `-ub 128` 0.070): flint is very sensitive to tiny numeric differences.
-Details, caveats and the KLD bisect: `benchmarks/summary.md`. Validated only on this CPU.
+`GGML_IQ4XS_PF`, `GGML_Q41_PF`, `GGML_K4_PF`, `GGML_K5_PF`, `GGML_Q6_PF` tune the prefetch distances (defaults are the tuned
+values). `patches/optional-profiler.patch` adds `GGML_PROF=1` per-op timing.
+
+## Limitations
+
+- Validated only on Intel Xeon Platinum 8559C (AVX512-VNNI + AMX), 16 threads, on a shared host. Other CPUs are untested; the new kernels need AVX512-VNNI (the Q4_1 one also AVX512VL/F16C).
+- The Q4_K_XL numbers were measured before the Q4_1 kernel was added; the K-quant code path is unchanged except for removing unused constants, and was not re-benchmarked.
+- PPL and KLD use a short test (32 x 512 tokens); no downstream task evaluation was run.
+
+## Files
+
+- `patches/flint-cpu-opt.patch` (base `patches/BASE_COMMIT`), `patches/optional-profiler.patch`, `build.sh`, `machine.txt`
+- `benchmarks/summary.md` (Q4_K_XL), `benchmarks/q41-summary.md` (Q4_1), raw A/B CSVs, KLD logs, op profiles, KLD bisect, `graph_accuracy_test.cpp`
+
+## Credits and license
 
-## Also: RecursiveSI/flint-int4-gguf (Q4_1)
-The same patch adds an AVX512-VNNI Q4_1 kernel: on `flint-int4-q4_1.gguf` pp256 +9 %, pp2048 +10 %, tg128 +12 % versus stock
-(54 t/s decode, faster than the Q4_K_XL file's 43 t/s, but prefill is 209 vs 307 t/s). See `benchmarks/q41-summary.md`.
+Model: Recursive Flint (Apache-2.0), GGUF quantizations by Unsloth and RecursiveSI, unmodified. Patch builds on llama.cpp (MIT)
+and reuses ideas from the Qwen3.6 CPU-OPT patch. This repo is Apache-2.0 for the model files; the patch follows llama.cpp's license.