Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
ManniX-ITA 
posted an update 3 days ago
Post
109
# opencoti-llamafile 0.10.3-c5 — settled admission for multi-agent serving

New cut of the opencoti single-file inference engine (llamafile 0.10.3 / llama.cpp + 87 additive patches). Zero-dependency APE: one executable for Linux, Windows, macOS & BSD.

What's new vs c4:

**PolyKV fan-out — pool from a live session.** POST /polykv/pools gains from_session/from_slot: the shared prefix is snapshotted server-side from the session's cached KV — no tokens resent, token-exact. Ephemeral pools auto-release when orchestrators die mid-round.

**PolyKV P7 — settled admission.** Spawning agents faster than the tps signal settles was oversubscribing pools. Now: a per-pool settle window paces admits just enough for a reliable reading; warming sessions no longer bias the mean; the post-admit forecast uses the measured per-admit drop; idle gaps (agents mid-tool-call) no longer read as free capacity; guarantee_min_sessions means a new/nested pool always gets its first agent — capacity checks can never deadlock an orchestrator; the enforced gate applies to new sessions only, with per-request overcommit. Benchmark (multi-agent courier, floor 15 tok/s): time-under-floor −52%, deep sub-floor −83%, delivery p50 −43%, 100% task score.

**Zero-conf GPU sharing.** Instances on one GPU discover each other over shared memory — no ports, no config — and split compute by --gpu-share-weight. Measured (3090): weights 2:1 → 71.7/36.2 tok/s; holds at --parallel 4 and under MTP. Idle peers cost nothing (solo = full speed), crashes age out in 3 s; GET /gpu/peers shows live shares + busy %.

From c5 every release ships per-platform side-load DSOs: dso/<ver>/ with Linux x86_64 + sbsa .so and a Windows .dll.

ManniX-ITA/opencoti-llamafile
dipankarsarkar

Counting idle-during-tool-call as busy is the choice I would want a number on.

The benchmark reports floor adherence and latency, both better. Time under floor down 52%, delivery p50 down 43%. What it does not report is how many agents were admitted in each run.

A gate that admits fewer sessions improves both of those for free. p50 falls because the queue is shorter, not because serving got faster. So that pair cannot separate a better controller from a stricter one. Admitted sessions, or completed tasks per wall-clock minute, is what makes it a controller result instead of a throttle result.

The idle-gap rule is where this bites. An agent mid tool call really is not using the GPU, for a duration drawn from your tool-latency distribution. If that distribution has mass longer than the settle window, treating the gap as busy leaves measured capacity unused. If it is mostly shorter, ignoring it is correct. Same rule, opposite verdict, decided by a histogram you already have.

100% task score with a 15 tok/s floor also reads like the load never got hard enough to fail.

Does the gap between settled admission and c4 widen if you push agent count until c4 starts dropping tasks?

·
ManniX-ITA

Fair question — I ran the cells to answer it directly, including the overload run you suggested.

Same fixed workload (10 packages × 55 steps, fixed seed, same model/GPU), both gates 100%: the pre-P7 gate finished with 9 workers admitted, P7 with 8 — essentially equal concurrency — yet P7 delivered the identical workload in 1919 s vs 2187 s (−12% wall) with p50 delivery 764 s vs 1350 s (−43%). If the gain were stricter throttling, equal-or-fewer workers would mean a longer wall; it's shorter because admitted workers spent far less time degraded (time-under-floor −52%, deep sub-floor −83%). Less thrash, not less work.

Overload cell (16 packages, up to 16 workers, ~2× past this GPU's knee): neither gate drops tasks — both score 100% (legacy: wall 3029 s, p50 769 s, workers →10, 68% of samples under the 15 tok/s floor; P7: wall 2999 s, p50 737 s, workers →11, 71% under floor). At full saturation an admission policy can't manufacture throughput — the GPU is the ceiling and both gates pin it; the enforced-admission 429 backstop keeps sessions alive either way, so work gets slow, not lost. I didn't rig a hard deadline to force drops — degradation is graceful by design on both sides. P7's value is the mid-load regime: honoring a tps floor while scheduling headroom exists (that's the −43% p50), plus the idle-gap estimate that treats tool-call idle as free capacity in /capacity projections (projected_idle_estimate: true) — the opposite of counting it as busy.

Raw work-reports and logs available if you want to dig.

https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/bench/courier-admission/
dipankarsarkar

Good, that's the number that mattered.

9 vs 8 workers is close enough to call equal concurrency, so the -12% wall and -43% p50 can't be a throttle effect, they have to be less thrash. That's the controller result I was asking for, not a throttle wearing a controller's clothes.

I had the idle-gap logic backwards too. You are not counting tool-call idle as busy, projected_idle_estimate treats it as free capacity for /capacity. That is the direction that avoids the failure mode I was worried about.

The overload cell answers the drop-rate question directly: neither gate drops anything at 2x past the knee, both just get slower. So the gap I asked about does not widen because there is no gap to widen. Once the GPU is the ceiling, a gate cannot manufacture throughput either way. P7's value lives entirely in the mid-load regime where headroom exists to schedule around.

That is a narrower claim than "better admission policy," and it is the honest one. In production, what tells an operator they are still in that mid-load regime versus already past the knee, before the p50 numbers show it?

·
ManniX-ITA

The package_courier test harness does it; it's the only thing an operator has to find, the right floor_tps for that model & quantization.
Looping the test harness at different floor_tps, find the sweet spot.
dipankarsarkar

That reframes the whole question. Not a live signal, a calibration you run once per model and quantization, then feed floor_tps back in as a constant.

That's actually the more honest answer. Guessing you're in the mid-load regime from a live metric assumes you already know where the knee is. The sweep is what finds the knee, there's nothing left to infer live.

Does floor_tps found for one quant transfer to another on the same GPU, or is it specific enough that swapping quantization means re-running the harness?

·
ManniX-ITA

It's a performance gate more than correctness; the resulting wall time depends on the gen tps of the backend on that quant.
The operator needs to characterize and find the exact sweet spot for his workload; does it really needs a lot of agents in parallel? Is it a requirement that they complete their job quickly? or maybe they will heavily depend on async calls and stay most of the time idle and/or quit early so it's better that they pickup more items as possible from the queue?

The best way to calibrate would be to adapt the test harness and replicate synthetically the real agents workload but it may be just an over-achievement that doesn't bring much value.
Once validated that the model can use tools properly and what's the floor_tps, I think it can be transferred with high confidence on almost anything else.

Testing with an RTX6000, which is the best Pro GPU you can get, it's pretty evident that you need to use small models or MoE if you really want run an agentic fleet. The right floor_tps is generally between 15 and 25 tps.
Using dense models like Gemma-4 31B and Qwen 27B, needs a more careful tuning than faster ones; it's a floor value anyway, higher is the aggregate tps and easier it gets.
Things gets probably more interesting with datacenter class GPUs or running SLMs on CPU, but I didn't have time to test it yet.

About the correctness gate, on top of the basic tools usage check, there's also the quantization impact on quality that comes into play.
Some models just drops packages always, like OmniMerge v4, others are perfect like Qwen 35b A3B.
I have tested only Gemma-4 A4B quantized vs F16 and indeed under massive stress, at floor_tps 5 and 1, the Q4 quant starts dropping packages while the F16 doesn't.
This is another useful metric and more than a model limitation is probably an engine limitation, likely inherited from llama.cpp.
Gemma-4 is very peculiar and running any model quantized means that in many inference steps it has to be lifted to F16 and quantized back down. Running it directly at F16 is way much easier for the engine and the GPU, the conversions are always lossy.

In any case, the test harness gives you a critical overview of your chosen setup. It's a must-run.
dipankarsarkar

That's the finding hiding in the last message: floor_tps transfers across quant, but package-dropping under quantization doesn't. It's a property of the model and the engine together, not a settled constant like floor_tps.

If Q4 dropping while F16 holds is really inherited from llama.cpp's on-the-fly F16 round-trip, that's an engine limitation, not a Gemma-4 limitation. Would the same A4B quant still drop packages on a backend with native low-bit kernels and no dequant-requant hop?

·
ManniX-ITA

There's always dequant-requant hop and llama.cpp has already native low-bit kernels, opencoti-llamafile has more.

The matter is that shouldn't impact quality on stress.
I'm re-testing everything as those were done with an older version of opencoti-llamafile and I have updated the test-harness.
I've added some specific fixes for Gemma-4 since then and fixed the quants (new template from Google and a bug in the GGUF converter, but they should not have an impact on this harness).

I've also updated the summary in the README.md, it was completely wrong.
Added some information on cache re-use as well.
In this post