Captures real wall-clock + real NVML energy + Mellea grounding
pass-rate from the warmed lablab demo Space, run on 2026-05-09.
All four addresses pass with all five Stones firing on the three
single_address queries; Carleton Manor falls through to a
neighborhood-intent route because the geocoder can't pin the
literal NYCHA development name (documented as such, not failure).
Headline numbers:
median wall-clock 123.4 s
median per-query energy 1.51 Wh / 5435 J
median tokens / query 4434
calls measured on real GPU 48 / 48 (100%)
Mellea grounding pass rate 4 / 4 on every query
Adds:
scripts/probe_benchmarks.py — collector that consumes
/api/agent/stream, accumulates the full SSE trace, and
emits a per-query JSON record (paragraph, mellea, emissions,
by-Stone fired counts, errored vs by-design skips, geocode).
Reproducer for any row in BENCHMARKS.md.
docs/BENCHMARKS.md — per-query result tables + aggregates +
where-the-energy-goes bre…
fix(specialists): never fall through to broken local terratorch
The Sat May 9 trace surfaced two regressions on the lablab UI Space:
prithvi_nyc_pluvial: RuntimeError: operator torchvision::nms does
not exist
terramind.lulc: deps unavailable on this deployment:
terratorch (RuntimeError), peft
terramind.buildings: deps unavailable on this deployment:
terratorch (RuntimeError), peft
Root cause in both: the specialist tries the configured remote
inference service, the call fails (RemoteUnreachable, ok=False, or
generic exception), and the code silently falls through to the local
terratorch path. Local on the cpu-basic UI Space has a torch /
torchvision binary mismatch (torchvision::nms not registered), so
either we crash with that RuntimeError or — for terramind_nyc, where
_DEPS_OK is False at import time — we surface a misleading "deps
unavailable" message that hides the real (remote) failure.
Fix: when remote was attempted, never fall through to local. Return
a …
fix(redeploy): use cached huggingface-cli login by default
HF_TOKEN was a hard requirement; HfApi() falls back to the
~/.cache/huggingface/token written by 'huggingface-cli login' so the
env var is unnecessary when the user is already authed.
- update_hf_env.sh: HfApi(token=os.environ.get('HF_TOKEN')) — None
routes to cached login.
- redeploy.sh: replace the HF_TOKEN guard with a real auth probe via
HfApi().whoami(); exits early with a clear hint if neither path is
authed.
- runbook: drop the HF_TOKEN= prefix from the quick-redeploy snippet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cornerstone (DEP stormwater + Sandy 2012) bottleneck on HF Space CPU
is GDB cold-load, not the join itself: dep_extreme_2080 alone is 30s
to read on first call, and worker memory pressure evicts the
lru_cache between queries. Bake DEP scenarios + Sandy extent to
DEFLATE uint8 GeoTIFFs at 10 ft/px (data/baked/, 7 MB total) and
sample with rasterio. ~500x faster cold-load, ~5x faster per-query,
full parity on 4 canonical addresses across 3 DEP scenarios + Sandy.
- app/flood_layers/dep_stormwater.py: add join_raster() with
thread-local DatasetReader cache; legacy GDB join() retained as
fallback when baked rasters absent (local dev) and as the
polygon-overlap path for coverage_for_polygon (neighborhood mode)
- app/flood_layers/sandy_inundation.py: same pattern, inside_raster()
- app/fsm.py: step_dep / step_sandy use the raster path
- scripts/bake_cornerstone_rasters.py: one-shot bake (~50s)
- experiments/22_cornerstone_optim/: bench harness + RESULTS.md
UI honesty: stoneRegistry.ts…
update_hf_env.sh: sets the 6 HF Space variables (RIPRAP_LLM_PRIMARY,
RIPRAP_LLM_BASE_URL, RIPRAP_LLM_API_KEY, RIPRAP_ML_BACKEND,
RIPRAP_ML_BASE_URL, RIPRAP_ML_API_KEY) via huggingface_hub Python API
(add_space_variable / restart_space -- huggingface-cli space does not
exist in v0.36.x), then polls /api/backend up to 120 s for HTTP 200.
redeploy.sh: thin orchestrator -- generates a fresh token, calls
deploy_droplet.sh, calls update_hf_env.sh, calls probe_addresses.py,
prints a pass/fail summary. Exit 0 only when all three steps pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Squashed from 5438cc8..6904684. The slides/asce/deck.pptx is tracked
via git-lfs (added *.pptx to .gitattributes), so this commit carries
only the LFS pointer — no binary blob in history.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Insurance for the case where the public-base Dockerfile diverges
from the bootstrap droplet's ABI in a way that breaks specialists.
Commits the running container as an image, saves to a compressed
tarball (zstd if available, gzip fallback), and scp's it
off-droplet to a backup dir.
Not run by default — tarball is ~15 GB compressed and the scp takes
the better part of an hour. Use only if the Dockerfile build can't
reproduce the working env (e.g. AMD changes the public ROCm release
in a way that drops MI300X kernels we depend on). For 99% of
redeploys, scripts/deploy_droplet.sh + the Dockerfile are the right
path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three artifacts that turn the bootstrap-droplet's hand-built
container into a reproducible bring-up on any AMD ROCm GPU node:
- services/riprap-models/Dockerfile — extends the public
rocm/pytorch:rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.9.1
image (same minor torch version as the bootstrap droplet's
bespoke +git build) with our pinned terratorch / granite-tsfm /
transformers / peft / sentence-transformers / gliner stack. Bakes
in the MI300X tuning env (HIP_FORCE_DEV_KERNARG=1 etc) so a fresh
container doesn't need the "remember to set these" incantation.
- services/riprap-models/requirements-full.txt — exact pip pins
captured from the running terramind container on 2026-05-05.
Curated: only the leaves the Dockerfile installs on top of the
ROCm PyTorch base; transitive deps resolve from these. Excludes
torch / torchvision / torchaudio / amd-* (provided by the base).
- scripts/deploy_droplet.sh — idempotent one-shot bring-up. Takes
IP + bearer token, verifie…
Harden remote-ML routing + add address test suite
Wires up the regressions surfaced by a real PS 188 / Lower East Side
run on the AMD MI300X stack. Five fixes + one new probe:
1. Lift the `RIPRAP_HEAVY_SPECIALISTS` gate when remote ML is
reachable. The heavy specialists' GPU work (TerraMind LULC +
Buildings, Prithvi-live, register joins) runs on the droplet, so
the local wall-clock cost drops from ~60 s to ~5 s — gate the
default to ON so the public demo never silently disables them.
2. TTM specialists never crash with the
`ModuleNotFoundError("Could not import module 'PreTrainedModel'")`
that surfaced on local-fallback paths. Force-import
`PreTrainedModel` + `TinyTimeMixerForPrediction` before
`get_model()` so the transformers lazy-registry resolves the
class strings under FSM worker threads.
3. When remote ML is configured but returns non-ok, surface the
remote error rather than fall through to a brittle local model
load (`ttm_forecast`, `ttm_battery_surge`, `prithvi_live`). Local
fallback is r…
The fixtures Riprap ships baked all start their lives in scripts/:
fetch_nyc_dem — pull USGS 3DEP tiles, mosaic to NYC bbox
compute_hydrology_indices — TWI + HAND from the DEM via whitebox
fetch_ida_hwms — STN API → ida_2021_hwms_ny.geojson
run_prithvi_flood — Prithvi-EO 2.0 segmentation runner
run_prithvi_ida — Ida-specific pre/post diff variant
build_{nycha,schools,mta_entrances}_register
— full FSM pass over each register's
rows, JSON cached to data/registers/
audit — full evidence audit per query for the
printable report
dry_run — end-to-end CLI smoke without uvicorn
These run offline, on a fat workstation; only their outputs land in
the runtime image.
