ManniX PRO

Need to finish a fine-tune on the RTX6000s, when one will be free I will re-run with the new version all of them.
Tested Q4_K_M on the 3090 where the concurrency stressed the system much sooner and was scoring 85%.
Now it scores 100% so the F16 quant will for sure score perfectly as before.

Sorted out and it was not a quant problem, quite unexpected instead.
There was a bug in the prefill code which was invisible in normal usage.
It was triggered by the high number of requests in parallel, some sessions were simply getting stuck.

A4B is the fastest model in prefill, really fast, and the Q4_K_M quant enough faster than the F16 to trigger it.
Plus on top some issues with the test harness.

So I had a review of the API, the Langgraf extension and the test harness with a consultants council using ollama models and implemented a lot of fixes and improvements overall.
Next release will carry them.

There's always dequant-requant hop and llama.cpp has already native low-bit kernels, opencoti-llamafile has more.

The matter is that shouldn't impact quality on stress.
I'm re-testing everything as those were done with an older version of opencoti-llamafile and I have updated the test-harness.
I've added some specific fixes for Gemma-4 since then and fixed the quants (new template from Google and a bug in the GGUF converter, but they should not have an impact on this harness).

I've also updated the summary in the README.md, it was completely wrong.
Added some information on cache re-use as well.

It's a performance gate more than correctness; the resulting wall time depends on the gen tps of the backend on that quant.
The operator needs to characterize and find the exact sweet spot for his workload; does it really needs a lot of agents in parallel? Is it a requirement that they complete their job quickly? or maybe they will heavily depend on async calls and stay most of the time idle and/or quit early so it's better that they pickup more items as possible from the queue?

The best way to calibrate would be to adapt the test harness and replicate synthetically the real agents workload but it may be just an over-achievement that doesn't bring much value.
Once validated that the model can use tools properly and what's the floor_tps, I think it can be transferred with high confidence on almost anything else.

Testing with an RTX6000, which is the best Pro GPU you can get, it's pretty evident that you need to use small models or MoE if you really want run an agentic fleet. The right floor_tps is generally between 15 and 25 tps.
Using dense models like Gemma-4 31B and Qwen 27B, needs a more careful tuning than faster ones; it's a floor value anyway, higher is the aggregate tps and easier it gets.
Things gets probably more interesting with datacenter class GPUs or running SLMs on CPU, but I didn't have time to test it yet.

About the correctness gate, on top of the basic tools usage check, there's also the quantization impact on quality that comes into play.
Some models just drops packages always, like OmniMerge v4, others are perfect like Qwen 35b A3B.
I have tested only Gemma-4 A4B quantized vs F16 and indeed under massive stress, at floor_tps 5 and 1, the Q4 quant starts dropping packages while the F16 doesn't.
This is another useful metric and more than a model limitation is probably an engine limitation, likely inherited from llama.cpp.
Gemma-4 is very peculiar and running any model quantized means that in many inference steps it has to be lifted to F16 and quantized back down. Running it directly at F16 is way much easier for the engine and the GPU, the conversions are always lossy.

In any case, the test harness gives you a critical overview of your chosen setup. It's a must-run.

The package_courier test harness does it; it's the only thing an operator has to find, the right floor_tps for that model & quantization.
Looping the test harness at different floor_tps, find the sweet spot.

Fair question — I ran the cells to answer it directly, including the overload run you suggested.

Same fixed workload (10 packages × 55 steps, fixed seed, same model/GPU), both gates 100%: the pre-P7 gate finished with 9 workers admitted, P7 with 8 — essentially equal concurrency — yet P7 delivered the identical workload in 1919 s vs 2187 s (−12% wall) with p50 delivery 764 s vs 1350 s (−43%). If the gain were stricter throttling, equal-or-fewer workers would mean a longer wall; it's shorter because admitted workers spent far less time degraded (time-under-floor −52%, deep sub-floor −83%). Less thrash, not less work.

Overload cell (16 packages, up to 16 workers, ~2× past this GPU's knee): neither gate drops tasks — both score 100% (legacy: wall 3029 s, p50 769 s, workers →10, 68% of samples under the 15 tok/s floor; P7: wall 2999 s, p50 737 s, workers →11, 71% under floor). At full saturation an admission policy can't manufacture throughput — the GPU is the ceiling and both gates pin it; the enforced-admission 429 backstop keeps sessions alive either way, so work gets slow, not lost. I didn't rig a hard deadline to force drops — degradation is graceful by design on both sides. P7's value is the mid-load regime: honoring a tps floor while scheduling headroom exists (that's the −43% p50), plus the idle-gap estimate that treats tool-call idle as free capacity in /capacity projections (projected_idle_estimate: true) — the opposite of counting it as busy.

Raw work-reports and logs available if you want to dig.

https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/bench/courier-admission/

I use HE+ mostly to verify if the quant quality, if it doesn't collapse the quant is good, or to evaluate the loss at 2/3-bit.
Added MPE cause it's another canary and it has a bit more discriminative signal (depends on the mode capabilities with other coding languages than Python).
I don't use LCB because it takes hours to run; with 20-27 quants per model it's a NO GO, while usually HE+/MPE are taking 3-7 minutes each.

I cannot say:

• there's not much overlap at ISO disk size, I should compare Qwen3.5-9B F16 vs v7-coder Q6_K: not really much use for it, nobody would use it at F16
• I didn't run the evaluations for Qwen3.5-9B other than Q6_K and there's no matching quant even at Q8_0 with v7-coder
• they serve different domains; Qwen3.5-9B is an excellent Coder but the thinking is awful. Different tasks. Qwen3.5-9B can be used for simple tasks, completion or as coding assistant, not for agentic coding or any too complex thinking. It has a surprisingly good knowledge of Science but I think the high GPQA score comes more from a training bias than real reasoning capabilities. It scores very good on HE/HE+/MPE, but the LiveCodeBench score is revealing: only 58% vs v7-coder 95-98% and low AIME. Still, it's a 9B model and in that class you can hardly do more.

Yes indeed, I have my canonical OmniMergeKit evaluations.
If you check the v7-coder GGUF repo you can see the scores of Gemma-4 A4B which it's a 26B.
The balanced 184e version was awful compared to that and even miserably beaten in more than a few by the (excellent) Qwen3.5-9B. The speed advantage of A3B wasn't worth at all the 3x times VRAM usage of 27B vs 9B.