Q8_0 vs Q6_K on RTX 3060: decode, prefill and perplexity on Lean 4 source

#1
by TitanMamo10 - opened

Measured both quants on a single RTX 3060 12 GB (llama.cpp CUDA build, -ngl 99, 3 reps).
Posting in case it saves someone a download.

quant size pp512 (t/s) tg128 (t/s)
Q8_0 4.37 GiB 3654 ± 244 68.8 ± 0.1
Q6_K 3.37 GiB 2724 ± 724 80.6 ± 0.2

Decode is +17% for Q6_K, not the ~+29% that pure memory-bandwidth scaling predicts from
4.69 GB vs 3.62 GB. Q8_0 dequantization is largely free on tensor cores, while K-quants pay a
real per-weight dequantization cost — so the usual "size ratio = speed ratio" rule
overestimates K-quant gains on this card.

Prefill is -25% for Q6_K. On a 512-token prompt that is 140 ms vs 188 ms, so it only
matters when prompts get long.

Perplexity on an identical corpus (95 KB of Lean 4 source, 7 chunks, n_ctx 4096):

  • Q8_0: 2.4012 ± 0.0428
  • Q6_K: 2.4165 ± 0.0431

Difference +0.64%, i.e. inside one standard error. No detectable quality loss at this sample
size — though this only rules out gross degradation, it says nothing about downstream proving
ability.

Prover sanity check (Q8_0 only): 14 problems sampled from MiniF2F-test, 4 samples each,
temperature 0.7, every proof verified by Lean 4.25.0 — 9 of 14 solved (pass@4 = 64%).
Small sample and not comparable to the model card's pass@32 figure; included only to confirm
the quant is functional as a prover and not just as a language model.

Practical conclusion for a 4B prover on a 12 GB card: decode dominates the wall clock, so
Q6_K is the faster choice end to end (~15% on a 2000-token proof), while Q8_0 wins on prefill
and is the safer pick if quality is the only consideration. Both fit four 16k context slots.

Sign up or log in to comment