Validates the existing literature-calibrated UHICorrector against Landsat
8/9 Collection 2 Level 2 surface temperatures at 15 Dar zone centroids
(2020-2024, 707 cloud-masked samples across 13 zones). Finding: the
synthetic model systematically overpredicts UHI at valley/coastal Dar —
Jangwani LST hot-season anomaly −5.5°C vs synthetic model +3.8°C implied
delta; Msasani LST −5.9°C vs synthetic +1.8°C. Surface-temperature rank
order (Tandale +3.6°C hottest; Jangwani and Msasani coolest) contradicts
the "settlement_type=informal → high UHI" heuristic.
This commit ships:
- src/downscaling/uhi_lst_model.py: new UHILSTCorrector with the same
correct_temperature() interface. Direct per-zone LST lookup (zone
hot-season anomaly × Voogt & Oke 2003 midpoint LST_TO_AIR_SCALE=0.5)
with XGBoost fallback for zones outside LST coverage (DAR-KIN, DAR-VIN).
- src/downscaling/__init__.py: factory + get_zone_uhi_range() helper.
UHI_MODEL env var picks synthetic or lst at instance time; default
…
Cut XGBoost/LSTM heat predictor from pipeline and demo
Parallels the MI MOS cut. HeatWavePredictor (XGBoost classifier +
ensembled LSTM) was trained on 2 years of pure-synthetic AR(1) noise
using CITY_CLIMATE seasonal curves (heat_forecast.py:331, lstm auto-
retrain at :530). It wrote trigger_probability_7d / prediction_confidence
/ xgb_probability / lstm_probability into the predictions table, but
nothing downstream used those numbers for a decision -- GraphCast drives
trigger firing, burn analysis drives pricing. Classic orphan layer.
Changes:
- Drop HeatWavePredictor import + self._predictor in pipeline.py
- Drop the predictor.predict() call inside _step_predict; the step now
only does GraphCast/observed trigger logic and burn-analysis pricing
- predictions row: NULL for trigger_probability_7d, prediction_confidence,
xgb_probability, lstm_probability. model_tier and ensemble_method
retained (they describe the active forecast source, not the dropped
ensemble). annual_cost_per_worker / payout_factor / learned_frequency
keep com…
Refresh Neon connection at the start of each pipeline run
Same bug family as the stale-conn fix from earlier in the session, but
for the longer-lived _db_conn that's set up once in lifespan and reused
across many pipeline runs. _refresh_conn only ran during __init__, so
the first DB write of a run that fired hours after the last one would
hit a Neon-autosuspended socket.
Concrete failure mode (observed on run fcb5cf78):
- _db_conn created fresh in lifespan, first run writes land fine
- Space idle for 15 min, Neon autosuspends the conn
- Next trigger: start_pipeline_run throws on conn.execute
- _db_write catches it and sets self.db=None for the rest of the run
- Pipeline reports status=ok because the step logic doesn't check DB
outcomes, but Neon has zero new rows for the run (not even the
pipeline_runs row) and the HF status page proudly shows "done"
Fix: before each run, call _db_conn._refresh_conn() which does a
SELECT 1 and reconnects from the pool if the socket's dead. If the
pool itself is also toast, recreate _db_conn via init_db(…
Prewarm GraphCast at FastAPI startup to avoid cold-start fallback
Mirrors the MI Chronos prewarm pattern. load_model() was called lazily
on the first _step_predict, which on a freshly-rebuilt HF Space took
~30-120s (downloading the 1.4GB checkpoint from GCS plus normalization
stats). That puts the pipeline's first weekly trigger at risk of the
same silent-fallback path that bit MI before PR #5.
Fix: spawn a daemon thread in the lifespan context that calls
load_model() at startup. JAX compile + GCS download happen in the
background while FastAPI comes up; by the time someone hits
/api/pipeline/trigger the model is resident. Failures log and move on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fix pipeline failures: api.py import, NASA POWER resilience, CPU torch
- api.py: Replace NeuralActuarialPricer import with BurnAnalysisPricer
- pipeline.py: Relax ingest completeness check — NASA POWER lags ~7 days,
treat any zone with readings as usable instead of requiring >50% quality
- Dockerfile: Restore CPU torch (sentence-transformers/RAG depends on it)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Harden serverless DB checks, CORS, cursor lifecycle, and healing JSON parse
Replace the process.env.DATABASE_URL! bang assertion across every
Vercel serverless function with an explicit check that returns 500,
removing a silent crash path. Make src/api.py CORS origins configurable
via ALLOWED_ORIGINS. Fix a cursor leak in PgConnection by closing the
previously-returned cursor before opening the next one, and add a
bulk insert_daily_readings_batch() helper using executemany. Catch
JSONDecodeError on Claude healing responses and return an empty
assessment list rather than crashing mid-heal. Update CLAUDE.md to
the new extreme-heat-insurance.jeff-levine.com custom domain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: async CRUD wrappers around sync psycopg2 blocked the event
loop, causing the pipeline to hang on HF Spaces.
Fix: everything is sync now.
- PgConnection uses SimpleConnectionPool with _refresh_conn() for Neon cold starts
- All 21 CRUD functions are plain def (no async)
- Pipeline run() is plain def (no async)
- Only httpx ingestion is async, wrapped in asyncio.run() for that one call
- init_db() returns PgConnection or None
- Lifespan seeds zones synchronously at startup
Tested locally with real Neon DB: 14s, 15 zones, DB writes succeed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip DB writes in pipeline — psycopg2 blocks async event loop
DB init now runs in background task — doesn't block app startup.
Fixed AttributeError in downscale step (reading.date is a string, not datetime).
Pipeline runs in 2.6s locally with 15 zones, 15 triggers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New architecture (target: ~$0.02/run instead of $3+):
- Heal: rule-based for all readings, Claude only for flagged anomalies (0-1 calls)
- Explain: template-based alerts (free, instant)
- Review: ONE Claude call at the end to validate pricing and write
weekly recommendation in English + Swahili
HF Space status page: vertical pipeline tracker with step dots,
run button, and last-run result. Auto-refreshes every 5s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PgConnection now has reconnect() which drops the stale pool and creates
a fresh one with a SELECT 1 health check. Called before each pipeline
run so Neon cold-start is handled automatically.
Pool reduced to 1-5 connections, 15s command timeout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The pipeline tracker belongs on the HF Space status page (where you
monitor runs), not on the Vercel frontend (where donors see the demo).
HF Space root page now shows:
- Live progress bar with step name while pipeline runs (auto-refreshes 5s)
- Last run result (status, zones, triggers, duration) when idle
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default to rule-based healing (free) instead of Claude ($3/run)
Claude healing is expensive for weekly runs. Rule-based fallback does
the same validation (range checks, gap filling) without API costs.
Claude healing remains available when explicitly enabled.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Parallelize Claude healing (5 concurrent), add pipeline step tracker
- Healing step now runs up to 5 zones concurrently via asyncio.gather +
Semaphore, reducing healing from ~8min sequential to ~2min
- /api/pipeline/status now returns current_step and current_step_index
while pipeline is running, so frontend/monitoring can show progress
- Pipeline reports step progress via callback to api.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add regionConfig for geographic portability, REBUILD.md fork guide
Extract hardcoded geography strings from 5 frontend components into a
single regionConfig.ts. Someone forking for a new city edits one file
instead of hunting through Dashboard, Sidebar, Zones, Notifications,
and Pipeline. UI is identical after this refactor.
REBUILD.md documents the full fork-and-adapt process: which files to
change, what's portable, and a Claude Code prompt template.
Also: remove dead model_info from API response, remove unused import
and redundant baseline computation from eval tests (/simplify fixes).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Push red accent (#e63946) through tabs, buttons, inputs, sidebar, tour tooltips
to visually differentiate from Weather AI 2's gold identity
- Rewrite all 12 tour steps for first-time clarity: specific neighborhoods,
worker types, intuitive UHI explanation, no model architecture jargon
- Tour now ends on How It Works page instead of looping back to Home
- Add collapsible "About this model" card on Coverage Recommendation page
- Add model_info block to /api/coverage-recommendation response
- New eval suite (tests/eval_neural_pricer.py): 8 tests covering model loading,
price variation, rank preservation, correction bounds, settlement ordering,
GPD parameters, GLM fallback, and climate sensitivity
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pipeline runs on HF Spaces (not locally), writes to Neon:
- POST /api/pipeline/trigger — kicks off pipeline in background
- GET /api/pipeline/status — check if running + last result
- Weekly APScheduler cron (Tuesdays 00:30 UTC)
- Pipeline uses Claude healing/explanation when ANTHROPIC_API_KEY is set
- Pipeline writes to Neon when DATABASE_URL is set
- DATABASE_URL added as HF Space secret
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix UHI_RANGES inconsistency: neural_actuarial.py now imports from
uhi_model.py instead of defining its own (mixed was 1.5 vs 1.0)
- Fix healer.py crash: replaced flood-era attributes (flood_susceptibility,
drainage_quality, rainy_seasons) with heat-era equivalents
- Fix N+1 linear scan in /api/calibrate: build indices_by_id dict upfront
- Remove redundant WBGT recomputation in build_training_samples
- Remove unused UHICorrector() instantiation in build_training_samples
- Remove dead imports: math, asdict, HEAT_THRESHOLDS, PAYOUT_PER_EVENT_USD,
DailyReading from pipeline.py
- Fix train_neural_pricer.py bug: .items() → removed dead variable
- Update eval test for 30 zones (was hardcoded to 20)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Redesign Program Designer as Coverage Recommendation page
The neural model RECOMMENDS the budget — the page no longer asks
"what's your budget?" It leads with: "$X needed this week to protect
Y workers across Z zones at elevated risk."
New /api/coverage-recommendation endpoint. Zone cards ranked by urgency.
Cost decomposition bar (worker payouts vs admin vs reserves). Neural
model insight per zone. Tour updated to tell the neural model story.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add neural actuarial pricing engine trained on real Dar es Salaam data
Three-headed temporal neural model (LSTM encoder + Neural EVT hazard +
WHO/ILO vulnerability + CANN pricing) trained on 20 years of NASA POWER
daily climate data for 5 Dar es Salaam zones.
Key results:
- Jangwani (informal, +4.5C UHI) priced highest at $5,305/worker/yr
- Kinondoni (formal, +1C UHI) priced lowest at $2,776/worker/yr
- Basis risk learned: informal 0.53 > mixed 0.43 > formal 0.30
- Neural correction (delta_NN) is modest: -0.9% to -4.0% vs GLM baseline
- Rank correlation 0.914 on held-out validation data
Architecture: ~230K params, 910 KB saved model. CANN skip connection
means delta_NN=0 reproduces the existing ActuarialPricer formula exactly.
Falls back to GLM if model not found.
Based on: Chen et al. (Management Science 2024), EQRN (Pasche & Engelke
2024), CANN (Richman & Wuthrich), ISO 7243 occupational heat stress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Retrain models on real NASA POWER data, wire pipeline to Neon, enable Claude
Models:
- LSTM retrained on 727 days of real NASA POWER data (20 zones, val AUROC 0.92)
- XGBoost retrained on same data (val AUROC 0.86)
- UHI model kept on literature-calibrated synthetic (no ground truth available)
Database:
- Pipeline writes to Neon after each step (daily_readings, healed_readings,
heat_indices, predictions, trigger_events, pipeline_runs)
- All DB writes wrapped in try/except — pipeline never crashes on DB errors
- run_pipeline.py updated: loads .env, uses HeatRiskPipeline, async DB init
Claude:
- Healing and explanation enabled by default (use_claude_healer=True)
- Activates when ANTHROPIC_API_KEY is set (HF Spaces secrets)
- Graceful fallback to rule-based when key is missing
Also: dotenv loading in api.py, zones seeded to Neon on startup,
training script at scripts/train_on_nasa_power.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace React SPA with lightweight API status page on HF Space
The Vercel frontend is the real UI — the HF Space only needs to serve
the API. Removes the Node.js frontend build stage from the Dockerfile
(saves ~2 min build time, smaller image) and replaces the SPA static
file serving with a simple HTML status page showing zone count, active
alerts, model tiers, and API endpoint docs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes the AI/ML sophistication gap with Weather AI 2:
- ERA5 real climate data via Google ARCO archive (free, no API key)
- PyTorch LSTM heat predictor ensembled with XGBoost (4-tier degradation)
- FAISS+BM25 hybrid RAG for alert generation (44-doc index)
- Neon PostgreSQL schema (11 heat-specific tables) with asyncpg + InMemory fallback
- 19-test eval suite (UHI accuracy, predictor AUROC, RAG precision, healing recall)
- Fix broken flood-era pipeline/store/schema imports
- De-jargonify all frontend text for donor/program officer audience
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6-step ML pipeline for parametric heat insurance in East Africa.
Three XGBoost models (UHI correction, heat wave prediction, actuarial pricing),
Claude AI healing agent, interactive program designer with budget allocation.
20 urban zones across Nairobi, Dar es Salaam, Kampala, Kigali.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
