Surprisingly good at creative writing!
I haven't dug into it too deeply, but, this model seems to perform surprisingly well at creative writing tasks - it's made some interesting and unexpected (in a good way) decisions I haven't seen in the actions characters take, vs even Gemma-4-31B.
I'm using the UD-Q4_K_XL quantization. I was less impressed with the creative writing when using the UD-Q2_K_XL quant, but I was able to squeeze surprisingly solid performance (PP: 453 t/s, TG:15.3) out of a system with only 12 GB of VRAM, and only moderate optimizing! (4070 Ti 12GB, 64 GB DDR5,14700k)
Despite this being described as a model aimed at agentic coding and long-horizon tasks, I do encourage folks to try it out for other sorts of things too!
Fully agree, very creative model
I haven't dug into it too deeply, but, this model seems to perform surprisingly well at creative writing tasks - it's made some interesting and unexpected (in a good way) decisions I haven't seen in the actions characters take, vs even Gemma-4-31B.
I'm using the UD-Q4_K_XL quantization. I was less impressed with the creative writing when using the UD-Q2_K_XL quant, but I was able to squeeze surprisingly solid performance (PP: 453 t/s, TG:15.3) out of a system with only 12 GB of VRAM, and only moderate optimizing! (4070 Ti 12GB, 64 GB DDR5,14700k)
Despite this being described as a model aimed at agentic coding and long-horizon tasks, I do encourage folks to try it out for other sorts of things too!
How can you fit UD-Q4_K_XL in 12VRAM+64RAM
How can you fit UD-Q4_K_XL in 12VRAM+64RAM
Oh, sorry, I wasn't clear on that - I've got two host systems - I ran it on the 12 GB VRAM/64 GB (DDR5) System, and i've also run it on a 48 GB VRAM (2x3090), 64 GB System (DDR4) - with the Q2 on the smaller VRAM system, and the Q4 on the larger. I was just saying that the Q2 system was "meh" in creative writing when comparing them.
For some metrics on the beefier system, from my llama-bench testing. (Worth noting, one of these 3090s only has a 4x PCI-E link, and I don't have NVLINK:
(ubatch is 1024, batch is 4096, split-mode is LAYER)
| Context | PP Speed | TG Speed | Tensor-Split | n-cpu-moe |
|---|---|---|---|---|
| 8192 | 367.0 | 24.8 | 11,5 | 21 |
| 32768 | 342.3 | 23.1 | 13,6 | 22 |
| 65536 | 310.0 | 21.5 | 11,5 | 23 |
| 262144 | 176.0 | 14.4 | 11,4 | 31 |
The splits seem weird,but that has to do - I believe - with the KV Cache (at least for this model) all ending up on GPU 0. Without these balacing values, at least with sm = layer, I get a lot of the model on one card's ram and a lot of it still in system ram, as the other card has very little loaded on it.