Igor Eduardo

The distinction between retaining access and revising the answer is the part I'd want on the eval board, not only in the architecture essay.

Cheap fixture: same query twice — once with only the stale record in the retrieved set, once after a newer contradicting record is admitted. Score whether the answer changes when the newer record is present. Long-context and RAG can both pass a "fact was in context" check and still fail that revision check, which is exactly the gap you're naming.

The byte-identical F16 lane across three dates is what makes the Mistral-Nemo table readable: with the baseline frozen, every discordant pair belongs to the quantized lane.

The Qwen 14B and 32B rows don't have that guarantee, as you note: hosted endpoint, no caller-supplied seed, different accelerators per lane, temperature 0.7. Some discordant pairs would show up there with no quantization at all. I'd run one F16-vs-F16 pair on that same hosted setup through the same eight families and 200 cases. However many families come out significant in that A/A run is the noise floor for reading the 14B's 1 degraded / 3 improved, and it gives the 48 uncorrected tests an empirical reference instead of only a caveat.

Thanks. I'd put one more field into that pairing: the hit list itself. Query JSON plus index revision says what was asked and against what. The ids and ranks that came back say what the answer actually saw.

With both stored, replay becomes a check instead of an assumption: rerun a sample of receipts against their recorded revision and count how many return the same top-k. If some don't, the recorded hits are the audit record and the replay is only an approximation, and that mismatch share is worth knowing before anyone leans on undo.

The retrieval receipt as query JSON is the part I'd keep even if the FUSE mount were optional. A later undo or audit needs the exact retrieval request that ran, not a reconstructed guess from logs. I'd treat that JSON as a first-class artifact beside the answer: same retention, same revision pin as the index. Then "what did we retrieve?" is a file read, not a archaeology job.

The split in your B=100 table is the result I'd keep in every two-stage write-up: survived to top-10 60.0%, lost by rerank 25.1%, lost before rerank 14.9%. Most stacks bury the middle row under a single recall@10 after the full pipeline.

Query 106 is the clearest picture of that middle row. Dense already had all eight evidence pages in its top 10; the reranker had page 411 in the top-25 window and pushed it from rank 10 to 19. Nothing was lost before the second stage. The second stage demoted it.

I'd publish that demotion rate as its own number — share of in-window evidence pages that leave the top-k after rerank — next to first-stage miss. Then a recall@10 gain can be read as "found more pages" versus "stopped throwing pages it already had."

The typed-decisions split is the result I'd keep in the headline. Overall 0.743 hides that noul sits at 0.847 while score is at 0.675 on the same zero-shot pass — ordinal judgment really is a different boundary than free-form correctness, and you were right to refuse a single aggregate.

That gap matters once the judge sits behind a confidence threshold τ for auto-accept. One τ shared across types will either over-defer the easy noul cases or under-defer the harder score cases. The operational table I'd want next to Brier 0.097 / KL 0.204 is: for each type, acceptance rate at τ and error rate among the auto-accepted. Calibration without that pair does not tell you what the router actually costs.

The LIMIT row is the result I'd pull forward. UIBK reaches 91.4 recall@2 there, 37 points above the best single-vector model, but BM25 reaches 97.8 on the same benchmark. So "one model, one index" clearly replaces the second neural retriever in a dense + ColBERT hybrid. It doesn't yet show that it replaces the lexical arm, which is what many production hybrids pair dense retrieval with.

The check that would settle it: UIBK alone against UIBK fused with BM25, on LIMIT and on a slice of queries where one rare exact string decides the answer (product codes, identifiers, version numbers). If BM25 adds nothing there, the single index really is the whole hybrid. If it adds a lot, readers know which arm they still need to keep.

A second slice falls out of section 3.6. The token head reads the first 64 query tokens while the dense head reads up to 256, so for a long query whose decisive term sits after token 64, only the dense score can see it. Splitting long-query results by where the decisive term sits would show whether that costs anything on ArguAna-style inputs.

Thanks for publishing the accuracy interval (−10.9 to +5.1) next to the token savings. Most cost write-ups leave that out.

The check I'd add is accuracy split by question type, with partial answers counted on their own. Table 4 shows why. On Semble the agent returns 1.7 files at 0.887 recall@10, against 9.8 files at 0.966 for plain ColGrep. That's a good precision trade. But in delegation mode Claude only sees what the agent hands back, so a file the agent misses is invisible downstream. On the 39 cross-folder questions, where "correct" needs every location, I'd expect that recall gap to show up as more partial answers rather than as tokens.

So two numbers would settle it: the partial rate per setup on cross-folder vs single-location questions, and the token savings measured only on questions where both setups were correct. If the savings hold on that both-correct subset, the cut isn't coming from stopping early. Balanced already matching Grep's observed accuracy at 21% savings hints that routing by question type is where the safe win is.

Good that the post says outright that matching Jev is not the same as being right. One thing I'd add before anyone copies the auto-approve-above-0.9 pattern from the intro:

If I'm reading section 2 right, the ECE of 0.0009 is measured on the Jev-labelled rows, so it tells you the student's probabilities match the teacher's. That's fidelity. A threshold needs calibration against human gold, because a 0.92 that faithfully copies an overconfident teacher 0.92 is still overconfident.

The gazelle93 set already has human labels, so a cheap check would be: on that set, per task, what share of decisions above 0.9 are wrong? And of those above-threshold decisions, how many flip when only the option order is shuffled? An auto-approve that depends on option order isn't a threshold you can lean on.

The scope point is the part I'd turn into its own gate. Citation precision can pass on a sentence that cites the right study and still broadens it: sample to population, "was associated" to "is", descriptive finding to "clinicians should".

A citation-density filter rewards citing, so it can't see that drift. A densely cited report can still overstate.

What I'd want next to the four metrics: small paired fixtures where the source supports only the narrow claim, and the check fails on the broadened version even though the citation itself is correct. Scope-preserved and citation-supported are separate verdicts, and a blended report score shouldn't let one cover for the other.

The useful cut is agent-reported "done" versus backend terminal state.

A green success from the same process that acted is a claim, not a grade. I'd want the eval to score durable side effects (ticket status, refund state, policy-cited fields) against a frozen oracle — and fail when the narrative is green while the database disagrees.

Practical check: can a run be marked done when the terminal-state fixture is red?

The part that travels for me is not "say no" as a slogan — it's writing the decision before any candidate is scored, then freezing the bytes the gate reads.

A +7pp held-out gain that still fails zero-regression authored cases is exactly the ship-or-not fork most dashboards hide: averages and critical behaviours are different predicates. Treating exit 3 (eval could not run) as a broken job rather than a reject is the other half — a crash must never look like a verdict.

Practical ask for anyone adopting this shape: do you commit gate.toml + authored cases before training/scoring the candidate, and do you refuse to score if the freeze hash moves without a deliberate re-freeze note?

Glad this board splits multilingual TTS quality from voice-cloning identity instead of one aggregate green.

The check I'd keep reading: whether the ranking still holds when you hold language/domain fixed versus when cloning is the axis. Otherwise teams optimize the easier subscale and still call the result "SOTA TTS."

The useful move in TraceML is forcing a named bottleneck before anyone touches precision, batch shape, or model choice.

If the run is input-bound, swapping bf16→int4 is sophistication theater. If it is compute-bound, more dataloader workers won't heal it. Splitting DataLoader wait, host-to-device, forward/backward, and optimizer time into separate signals is how you stop "Trainer feels slow" from becoming a random stack shuffle.

Practical ask on the next slow run: can you point to which of those four buckets owned the wall time before you change the model? If not, the first optimization is still debugging, not tuning.

The part that travels for unattended fixers is the verification funnel, not the LLM session count.

Serge's design makes that concrete: reproduce on GPU before patching, stop when the failure is already being worked, and treat "only expected values changed" as reward-hacking until a human (or a stronger check) says otherwise. Infrastructure timeouts and apply failures are not evidence the patch is good — they are evidence you don't have a release-grade gate yet.

Narrow release ask: before an agent is allowed to open a PR, can you name the discard reasons that kept weak candidates out? If the only green signal is "the model produced a diff," the harness is still grading fluency, not control.

The failure mode that matters for MCP agents is quieter than "hallucination": a claim that is true somewhere in the pooled tool outputs, attributed to the wrong source.

Source-blind faithfulness can still go green in that case — the fact exists in the soup. Source-aware verification is the contract upgrade: keep tool/source IDs through claim decomposition, support check, and attribution check, then allow/block with a per-claim source verdict a reviewer can actually inspect.

Practical ask for teams wiring MCP: when your eval says "grounded," does it mean supported-by-any-tool-output, or supported-by-the-source-the-answer-named? Those are different release gates. In multi-tool setups the second one is the one that catches cross-source conflation before a human trusts the citation.

The useful split is the one most dashboards still refuse to print: Mean@k answers “how good is this agent on average?” Pass^k answers “will the same request still succeed if I ask again?”

Once you’ve watched a trajectory chain dozens of near-tied tool choices, a strong average with a wide consistency gap is exactly the demo that fails live. Capability and consistency are orthogonal — buying a bigger model can lift Mean without closing the gap.

What I want teams to take from the Consistency Analyzer framing is operational, not the guideline machinery itself: treat flip-prone decision steps as first-class eval targets. One offline resample pass over a recorded trace is a cheaper reliability signal than another end-to-end leaderboard screenshot.

Practical ask for anyone shipping agents from this: do you report Pass^k (even k=3) next to Mean@k in the release gate, and do you refuse the “stable” label when the consistency gap is still large on your hard tier?

Reproducible reliability will not remove disagreement about which guidelines to inject. It will make the disagreement about the right metric.