Captures real wall-clock + real NVML energy + Mellea grounding
pass-rate from the warmed lablab demo Space, run on 2026-05-09.
All four addresses pass with all five Stones firing on the three
single_address queries; Carleton Manor falls through to a
neighborhood-intent route because the geocoder can't pin the
literal NYCHA development name (documented as such, not failure).
Headline numbers:
median wall-clock 123.4 s
median per-query energy 1.51 Wh / 5435 J
median tokens / query 4434
calls measured on real GPU 48 / 48 (100%)
Mellea grounding pass rate 4 / 4 on every query
Adds:
scripts/probe_benchmarks.py — collector that consumes
/api/agent/stream, accumulates the full SSE trace, and
emits a per-query JSON record (paragraph, mellea, emissions,
by-Stone fired counts, errored vs by-design skips, geocode).
Reproducer for any row in BENCHMARKS.md.
docs/BENCHMARKS.md — per-query result tables + aggregates +
where-the-energy-goes bre…
