oooooh wow, this is your best work yet. Everything is amazing; I only wish I had more VRAM to truly feel the glory. Thanks a lot

It is very good, first local model that answered most of the simple bench questions correctly
I only tested the first 5 so far, and it got 5/5 right on first attempt. I have never seen any local model answer anything past the second question.

Running the q4_k_m , - NON MTP - (for some reason I cant ever get it to work, and if I do its Worst and 3x slower lol rather than faster) on llama.cpp windows 11.

5800x3d + rtx4090 + 32gb ddr4 = 40-50tps which is very usable, great work man

simple-bench-git

Thank You!

The base comparison is the part that holds up. arc/c 0.647 to 0.711 on ARC-C's 1172 test items is about 4.8 standard errors. That is a real move, not harness noise.

The 700 line is where I would not put weight. mxfp8 0.711 and mxfp4 0.701 is 1.0pp, roughly 12 questions out of 1172. A single ARC-C score at p near 0.71 carries a standard error of about 1.32pp, so the threshold the model is named after sits inside the gap between your own two quants. Both quants also answered the same 1172 items, so the honest test there is paired, McNemar on the discordant ones, not two numbers compared by eye. Do you have per-item outputs to run it?

The other thing I cannot untangle from the table: it compares fine tune plus merge plus heretic against plain base, so no stage gets credit for anything. The heretic step is the strange one. Refusals go 99/100 to 4/100 at KL 0.0469 and all seven benchmarks go up. If the harness scores by loglikelihood over the fixed options, refusal direction should be close to invisible, because the model never gets to emit a refusal in the first place. That points the gain at the fine tune stages rather than the ablation.

Is there a bench of base plus heretic only, no fine tunes? That one arm separates them, and it is cheap.

I did read the card before asking, and the bench block is why I asked.

Every comparison row in the Nightmedia table is labelled [base, non heretic]. Qwen3.6-27B, 35B-A3B, Qwen3.5-27B, all non heretic. So the table has fine tune plus merge plus heretic on one side and nothing but plain bases on the other. The arm I am asking about is not on the card.

By cheap I meant compute, not quality. No training, no new stage. Base plus heretic is a checkpoint that already exists in your pipeline, and your own card says testing and benching was done at each stage. I am not asking you to publish every stage. One row.

And the 2 to 4 point cost you just named is the reason to publish it, because it argues your way, not mine. ARC-C goes 0.647 to 0.711, so plus 6.4 points end to end. If heretic costs you 2 to 4 on the way through, the fine tune stages are carrying 8.4 to 10.4 and paying a tax on top. Right now the table hands that credit to a stack, and the stage that did the work is invisible inside it.

Your framing of heretic tuned versus heretic base is the same subtraction from the other side, and it works just as well. Either one needs the base plus heretic number to exist.

Still curious about per item outputs on the two quants, since mxfp8 and mxfp4 answered the same 1172 ARC-C items and a paired test is free once you have them.

Which is closer to hand, base plus heretic, or the per item dumps?