flint_cpu_opt / benchmarks /q41-tiled-kld.log
HelloSun's picture
Add Q4_1 tiled VNNI GEMM for prefill: pp256 +34%, pp2048 +42% vs vec_dot patch
616131c verified
Raw History Blame Contribute Delete
807 Bytes
== base logits (this patch, Q4_1 prefill via vec_dot; GGML_CPU_TILED_MM=0)
1.38.076.158 I Final estimate: PPL = 18.2396 +/- 0.77914
== tiled Q4_1 vs vec_dot base (GGML_CPU_TILED_MM=1, --kl-divergence)
chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
Mean PPL(Q) : 18.272596 ± 0.781391
Mean PPL(base) : 17.736553 ± 0.734102
Cor(ln(PPL(Q)), ln(PPL(base))): 98.98%
Mean ln(PPL(Q)/PPL(base)) : 0.029775 ± 0.006166
Mean KLD: 0.037245 ± 0.002587
Same top p: 94.154 ± 0.260 %
wikitext-2 test, 32 chunks x 512, Xeon Platinum 8559C, 16 threads.
For reference, the existing vec_dot patch is 0.036404 +- 0.002873 vs stock, and the stock -fa off noise floor is 0.038469 +- 0.002729.