So I hopped on the HF page and noticed a few things that stood out.
The model is a collaboration between different people. This model isn't a one man show like most of his models iirc.
There are benchmarks. And some of them have large improvements and none of them show regression. Too good to be true. But benchmarks are not the full story. But what this showed is that whatever is going on, didn't hurt the model.
There is a focus on agentic performance, but I think adding fable in the name might be a bit of a stretch since we don't have the true thought traces for fable.
--
Previously, I'm using IQ4XS by unsloth with preserve thinking. I need 262K context on my Hermes agent so I do have to run Q4 KV on my 3090 (I got two more GPUs on the way)
Upon loading DavidAU's model and using my unsloth settings, a few things immediately stood out to me.
The model is not broken.
The model reasoning is much more efficient.
The reasoning of this model is different than the stock model. I often see it make a structured plan in the reasoning block the stock model doesn't do. There seems to be more nuance in the reasoning as well.
--
The immediate vibe is obvious. This feels better than stock model. But vibes aint shit. So I put it through my personal crons through my Hermes agent.
I have a long context setup where I need the agent to log into my work scheduling software, locate overtime bonus pay shifts (with very specific filters) and notify me. The website is quite complex with various traps and JS elements.
I've ran it 10 times. With full context clear each time. The unsloth stock model fails 2/10 times. This model did not fail. No failed tool calls either at KV Q4.
This model is going to be interesting at unquanted KV and Q8 weights. Y'all gotta try it. DavidAU cooked for once on this model.
FYI: use the vision mmprog from unsloth. it's half the size and I had zero issues with it.
I've been using this model as well. So far it's been better than the stock model.
The OP does point out that the unsloth mmprog is smaller and still works well. You probably should switch yours out. I got some free VRAM back when I swapped.
David, I saw in the other thread that the source of this is posted elsewhere, but that model's MTP head layer is broken. Can you upload the unquantized version of this so that people can try it with autoround? llama.cpp is noticably worse than vllm on dual graphics card setups
