Cornerstone (DEP stormwater + Sandy 2012) bottleneck on HF Space CPU
is GDB cold-load, not the join itself: dep_extreme_2080 alone is 30s
to read on first call, and worker memory pressure evicts the
lru_cache between queries. Bake DEP scenarios + Sandy extent to
DEFLATE uint8 GeoTIFFs at 10 ft/px (data/baked/, 7 MB total) and
sample with rasterio. ~500x faster cold-load, ~5x faster per-query,
full parity on 4 canonical addresses across 3 DEP scenarios + Sandy.
- app/flood_layers/dep_stormwater.py: add join_raster() with
thread-local DatasetReader cache; legacy GDB join() retained as
fallback when baked rasters absent (local dev) and as the
polygon-overlap path for coverage_for_polygon (neighborhood mode)
- app/flood_layers/sandy_inundation.py: same pattern, inside_raster()
- app/fsm.py: step_dep / step_sandy use the raster path
- scripts/bake_cornerstone_rasters.py: one-shot bake (~50s)
- experiments/22_cornerstone_optim/: bench harness + RESULTS.md
UI honesty: stoneRegistry.tsβ¦
Upload data/registers/mta_entrances.json with huggingface_hub
Two more user-facing modes on top of the address endpoint:
/compare β side-by-side briefings for two queries, useful
when picking between candidate parcels or
comparing two neighborhoods.
/register/{class} β pre-computed exposure for an entire asset
register: NYC DOE schools, NYCHA developments,
MTA subway entrances. The JSON ships baked
(built offline by register_builder), so the
UI is just paginated rendering β every row
shows the same paragraph + tier the address
mode would produce.
Asset loaders normalise each register's source GeoJSON into the
common (name, lat, lon, props) shape register_builder expects.
USGS 3DEP 1/3 arcsec DEM (resampled to 30 m for runtime), with two
hydrology indices computed offline by whitebox-workflows:
TWI β topographic wetness, ln(a/tan Ξ²), high where flow accumulates
HAND β height above nearest drainage, low where overbank flooding
pools first
These three rasters give the briefing terrain context that the
empirical/forward-modelled flood layers can't: even a parcel outside
every published zone can be in a topographic sink that any heavy rain
will fill.
.gitattributes routes *.tif through LFS.
Prithvi-EO 2.0 baked Ida 2021 satellite segmentationβ
NASA/IBM Prithvi-EO 2.0 600M flood-segmentation head run offline over
HLS scenes pre/post Ida 2021, then diffed and shipped as polygons.
This is the 'satellite saw water' layer β independent of the Sandy
extent and the USGS HWM points, comes from a foundation model that
was fine-tuned on labelled flood imagery.
Pre-computed (the model itself isn't in the runtime image): the
specialist just answers 'is this point inside any polygon, what's
the nearest, how confident'. The full live-segmentation pipeline
lands in a later phase as an experiment.
Field-surveyed peak water elevations from the USGS Short-Term Network
flood-response deployment after Ida (Sep 1, 2021). Where Sandy is
the storm-tide reference for coastal NYC, Ida is the cloudburst-pluvial
reference for inland and basement flooding β different physics, same
'what actually happened' grounding role.
Specialist returns the nearest mark by haversine distance plus depth
in feet; the reconciler treats it as point evidence, not extent.
DEP stormwater specialist + 3 forward-modelled rainfall scenariosβ
NYC DEP Stormwater Flood Maps as Esri FileGDBs (current, moderate
2050, extreme 2080-with-SLR). These are forward-modelled β what
DEP's hydrology team thinks 1%/0.2% AEP rain events will pool, given
present and projected sea levels β and they pair with Sandy's
empirical extent to give the briefing both 'what happened' and
'what is modelled to happen'.
.gitattributes routes the eight Esri internal binary types through
LFS.
Sandy 2012 inundation specialist + .gitattributes LFS for *.geojsonβ
Empirical extent of Hurricane Sandy storm-tide inundation, published
by NYC Open Data from FEMA MOTF surge modelling. This is the
historical-observation layer β what actually went under in October
2012 β and serves as the empirical floor when the forward-modelled
DEP scenarios disagree with lived experience.
The 91 MB GeoJSON ships pre-baked; *.geojson now routes through LFS.
