Quick question: any plans for Q4_K_M / other quants?
Hi! Really impressive work with this franken-merge. Grafting the 20 MTP layers from the unsloth Qwen3.6-35B-A3B-MTP model onto Bartowski's GGUF builds is a clever approach, and the ~20–30% inference speed gain on AMD hardware is exactly what I've been looking for. Thanks for sharing the model and the conversion script!
I noticed you currently have the IQ4_XS variant up (Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_XS.bartowski.mtp.gguf). I was wondering how you've found IQ4_XS comparing to Q4_K_M in terms of quality and stability? I know IQ4_XS saves a bit of VRAM and can be slightly faster on AMD, but I've also heard Q4_K_M tends to be more consistent across different workloads.
Would you consider releasing a Q4_K_M (or even Q5_K_M) version as well? Lower-bit options would be incredibly helpful for anyone working with tighter VRAM targets, and even a couple of key variants would make a big difference for broader accessibility.
If you're open to it, I'd be happy to help test or benchmark any new quants. Either way, thanks again for pushing this forward — really appreciate the effort!
Keep up the great work!
Thank you, nice to see that I'm not the only one interested.
Yep, hopefully I'll upload more versions on the weekend.
Due to space limitations, I am first doing extensive testing (at least a few hours of mostly ai-centered coding work on real-life coding tasks; so far it's been mostly Python and C) of the built models trying to find an optimal set of parameters for each quantization (based on the tasks tailored to my local problem domain, admittedly), and then either skip a specific quantization entirely, or schedule it for upload.
So far I was not too happy by Q4_* line (may be just was unlucky with a specific task; trying to kill two birds with one stone, so after a problem from my backlog is complete, I don't yet have time to return and retry it -- rather switch to a new problem in hope that it will still be somewhat representative of the same domain), but had some luck with Kwaipilot_KAT-Coder-V2.5-Dev-IQ4_NL and Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_S (so far tested with a shorter context window size, but it seems to be doing Ok (albeit slower) with larger (offloaded) contexts); so I'm planning to upload these first, and then if I am not restricted yet I am totally happy to upload some conversions as per your request.
Hi thread13,
Thanks for the update! Your testing approach makes total sense, quality first is definitely the right call.
I wanted to follow up on my offer to help benchmark. I ran some tests on my end, but unfortunately, I'm seeing a significant speed drop with the MTP models compared to the original.
Hardware/Setup: Intel Arc Pro B60 (24G VRAM), llama.cpp with Gigatoken and SYCL.
Results: The original model consistently hits 60–70 t/s, but with the MTP variants, I'm stuck around 30–40 t/s. I've tried tweaking various settings, but I can't seem to close that gap or reach the baseline speed.
Is this performance drop expected with MTP on this hardware, or do you think there might be a configuration detail I'm missing?
What hardware do you use?
I read that MTP works better at low temperatures (0.6–0.7) because the tokens are supposedly more predictable at those levels. (Just something I heard.) Can you confirm that?
Let me know if you need any further testing!
Hi MisterFynn,
I don't have any direct experience with an Intel Arc Pro B60, but I'd say that 60-70 t/sec would be an exceptionally good result, given its average performance (I am looking at https://www.techpowerup.com/gpu-specs/arc-pro-b60.c4350), while 30-40 t/s would be indeed below expectations; very roughly I'd expect ~45 t/s without MTP and ~60 t/s with it -- but again, I have no direct experience with this card, so I'm glad that it performs better for you.
Now for an MTP/non-MTP test I would avoid my franken-merges to avoid extra complication and would just go with e.g. an Unsloth Qwen3.6 35B MTP quantizations.
For a benchmark, I would probably also avoid high quantization levels and take something that is guaranteed to fit into VRAM including the cache -- on Linux, you can use something like nvtop to watch VRAM utilization, and check if there are any weird memory use patterns (such as memory usage getting very close to the ceiling and then suddenly dropping).
Next, there are probably much better sources than that, but for a very quick start -- some time ago I created a section in this repository's README, which also has a short listing fragment of a llama.cpp server log -- to quote:
draft acceptance = 0.95482 ( 317 accepted / 332 generated), mean len = 4.82
draft acceptance = 0.33333 ( 12 accepted / 36 generated), mean len = 2.33
this example shows that in two subsequent tasks, --spec-draft-n-max 4 was once an overshot (the second line), and once a bit too conservative (the first line). So my first question would be -- what would these numbers tell in your case.
For example, I wonder if an Intel Arc Pro B60 would be better responsive to shorter spec-draft-n-max values of 1 or 2.
Now if these numbers won't be telling you something -- then I shall add that in my local experience, the most drastic effect on token inference speed comes from under-fitting or overfitting of the KV cache into VRAM, and only then comes anything else.
So while the draft model is really normally very small compared to the main one -- still, if somehow the first version (without loading the draft model) comes at the watermark of your 24 Gb, then if loading draft layers can cross that watermark, then you would likely see a sharp decrease in the inference speed.
One way to test all this is to use a really low quantization level and compare the generation speed only, forgetting for a moment about inference quality.
If that works, then you would know your enemy, so to speak, and will likely be able to use a better quantization with other sacrifices -- such as the context window length or KV cache quantization.
In fact, the examples that I added to the README file here shall all fit into 24Gb vram, unless I made some terrible arithmetic mistakes somewhere -- and that seems to be supported by practice, as when I cross this 24 Gb limit, I see the inference rate going down sharply.
I read that MTP works better at low temperatures (0.6–0.7)
Well yes, that returns us to the first point -- I wonder what are the acceptance levels reported by your server.
Thing is, I am very narrowly targeting one specific case and one case only -- code generation, which might be the easiest case of them all: that is, one's average programming language has circa 30-40 reserved words, which we are using there all the time; so for an average coding block there's a ridiculously high chance that the next word would be simply a reserved one -- that is, if I would have decided to work as a draft model, I could just toss at random a few language keywords based on their frequency in preceding text and would still will have a decent chance to win (not a too high chance admittedly, but try to do the same with prose, and you'll see the challenge here).
As for temperature settings -- it might very well be so, and I may very well be using a too high value for temperature in testing.
I must say that I shamelessly stole it from Qwen 3.6 official examples and Unsloth recommendations (which were basically the same), and my logic was that any model trained on Qwen3.5+ family (like Kwai models) must be following the same semantics of temperature values, unless they have a really, really good reason not to (in which case I would expect to see them mentioning that reason on the model page).