Adds a zone_thresholds Neon table populated by the pipeline and consumed
by the zones serverless endpoint, so the Dashboard can show per-zone
alert/payout thresholds when zone-specific mode is active instead of
hardcoded city-wide defaults.
- src/database/schema.py: new zone_thresholds DDL (zone_id PK + alert/
payout thresholds + uhi_model + threshold_mode + computed_at).
- src/pricing/zone_thresholds.py: new persist_to_neon() helper that
UPSERTs current thresholds; no-op when THRESHOLD_MODE=global.
- src/pipeline.py: calls persist_to_neon at the start of _step_predict
so every pipeline run pushes current thresholds to Neon (dependent on
the active UHI_MODEL and THRESHOLD_MODE env vars).
- frontend/api/zones.ts: LEFT JOINs zone_thresholds and exposes
alert_threshold_c + payout_threshold_c + threshold_uhi_model +
threshold_mode. Risk-level bucketing now uses per-zone thresholds
when available, falling back to 35/38°C city-wide defaults.
- frontend/src/lib/api.ts: adds thr…
predictions ON CONFLICT: overwrite xgb/lstm/confidence/ensemble too
After cutting HeatWavePredictor, pipeline started writing NULL for
trigger_probability_7d on conflict. But the ON CONFLICT clause only
listed 5 columns for update -- xgb_probability, lstm_probability,
prediction_confidence, and ensemble_method were left out, so every
existing predictions row kept the last XGBoost-era values forever
(e.g. DAR-JAN on 04-21 was still carrying xgb_probability=0.0403
from a pre-cut run).
Expanded UPDATE SET to overwrite all of those plus predicted_at=NOW()
so next run fully heals the stale fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
heat_indices + healed_readings ON CONFLICT: overwrite all derived fields
Run 54cb7520 (first post-fix pipeline run) surfaced a sibling of the
daily_readings bug: every heat_indices row was frozen in time. The
pipeline wrote fresh rows with real NASA data but ON CONFLICT only
updated heat_risk_score + risk_level -- neither of which the pipeline
writes (both passed as None at pipeline.py:483-484). So the INSERT
was a silent no-op for the computed fields.
Result: the live dashboard was serving grid_temp_c=30.0 hardcoded-
fallback values left over from the pre-NASA-fix runs, despite having
accurate NASA data in daily_readings.
Fix: expand UPDATE SET to overwrite all derived fields on conflict
(grid_temp_c, uhi_delta_c, corrected_temp_c, wbgt_c, heat_index_c,
heat_risk_score, risk_level, consecutive_hot_days, computed_at=NOW()).
Unlike daily_readings (source data -- use COALESCE to preserve any
good reading), heat_indices is fully derived, so the latest
computation always wins.
Same pattern applied to healed_readings, which had a narrower but
identical bug: O…
NASA POWER ingest: heal NULL fields, request precip, respect 2-day lag
Diagnostic on run 67b30124 showed 81% of humidity/temp_mean/temp_min/wind
were NULL and 100% of precip_mm was NULL across 315 rows since 2026-04-01.
Three distinct bugs, all in the ingest path:
1. PRECTOTCORR was in the parser but not requested from NASA POWER, so
every precip row came back empty. Added it to NASA_POWER_PARAMS.
2. ON CONFLICT (zone_id, date) in insert_daily_reading + batch variant
only updated temp_max_c and data_quality. If the first insert for a
(zone, date) had NULL humidity (NASA not yet published when the zone
was first ingested) the NULL stuck forever -- subsequent runs
returning full data couldn't overwrite it. Confirmed live: NASA
returned full data for 04-15..04-17 but Neon still showed only
temp_max. Switched to per-field COALESCE(EXCLUDED.x, old.x) so any
later run can fill in fields that were previously null. data_quality
now GREATEST of old+new.
3. ingest_all_zones fetched through today, bypassing nasa_power's
_default_date…
_refresh_conn now retries up to 3 times after SELECT 1 fails. The
previous single-retry path could hand back a second dead connection
because SimpleConnectionPool initializes minconn=2 fresh conns at
startup which typically go stale together after Neon's ~5-min idle
timeout — if the first cached conn was dead, the second often was too.
Matches the Weather AI 2 fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coverage polish + DPI-style worker pilot with Disbursements drill-down
Coverage page:
- Cost section now renders Annual program cost, Cost per worker, and
3-year total in matching 44px Source Serif blocks (previously the
Annual cost was a 56px custom headline and the other two were default
MetricCards, creating an awkward size asymmetry).
- "Where it lands" table rebuilt: dropped the "Total / worker" column
(redundant with fast cash + insurance per worker), added a "Total to
neighborhood" column showing aggregate $ flowing to each triggered
neighborhood this week (per_worker × enrolled_workers), added a
"Workers" column, and renamed the zone header to "Neighborhood".
Reads as: "Kinondoni → $138,600 this week to 7,700 workers".
DPI-style worker pilot (the climate-risk equivalent of Market Intel's
Lakshmi/Kumar/Meena and Weather AI's 2k DPI farmer registry):
- New `workers` table in Neon (src/database/schema.py) with
worker_id / name / name_swahili / nida_id / phone / zone_id /
occupation / age / years_outdoor / household_size / mobile_…
Harden serverless DB checks, CORS, cursor lifecycle, and healing JSON parse
Replace the process.env.DATABASE_URL! bang assertion across every
Vercel serverless function with an explicit check that returns 500,
removing a silent crash path. Make src/api.py CORS origins configurable
via ALLOWED_ORIGINS. Fix a cursor leak in PgConnection by closing the
previously-returned cursor before opening the next one, and add a
bulk insert_daily_readings_batch() helper using executemany. Catch
JSONDecodeError on Claude healing responses and return an empty
assessment list rather than crashing mid-heal. Update CLAUDE.md to
the new extreme-heat-insurance.jeff-levine.com custom domain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix $0.00 cost: pass total_cost_usd to finish_pipeline_run
Pipeline now persists neural model outputs (trigger_prob, payout_factor,
annual_cost_per_worker, learned_frequency) to Neon predictions table.
Vercel coverage-recommendation reads these for real pricing data.
Added columns to predictions: annual_cost_per_worker, payout_factor,
learned_frequency.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rewrite pipeline + DB to fully sync psycopg2 (matches Weather AI 2)
Root cause: async CRUD wrappers around sync psycopg2 blocked the event
loop, causing the pipeline to hang on HF Spaces.
Fix: everything is sync now.
- PgConnection uses SimpleConnectionPool with _refresh_conn() for Neon cold starts
- All 21 CRUD functions are plain def (no async)
- Pipeline run() is plain def (no async)
- Only httpx ingestion is async, wrapped in asyncio.run() for that one call
- init_db() returns PgConnection or None
- Lifespan seeds zones synchronously at startup
Tested locally with real Neon DB: 14s, 15 zones, DB writes succeed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch from asyncpg to psycopg2 for Neon DB (same as Weather AI 2)
asyncpg's connection pool breaks with Neon auto-suspend. psycopg2
with fresh connections per query handles Neon cold-start reliably.
Same pattern that works in Weather AI 2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix _pool references in CRUD functions for poolless connection manager
All 21 'if db._pool:' guards replaced with 'if db.is_postgres and not db._memory:'
to match the new per-query connection approach.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace asyncpg pool with fresh connections per query
Neon auto-suspends after inactivity, killing pooled connections.
Fresh connections per query let Neon wake automatically on connect.
For a weekly pipeline writing ~100 rows, a pool is unnecessary overhead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PgConnection now has reconnect() which drops the stale pool and creates
a fresh one with a SELECT 1 health check. Called before each pipeline
run so Neon cold-start is handled automatically.
Pool reduced to 1-5 connections, 15s command timeout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix compute_heat_risk argument order bug, wire basis risk, remove dead import
- compute_heat_risk was called as (zone, temps, humidity) but signature is
(temps, humidity, dates, zone) — the predict step silently failed every run
- Wire up assess_all_zones to populate _basis_risk (was imported but never called)
- Remove unused PIPELINE_STEPS import
- Fix _to_date return type annotation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Retrain models on real NASA POWER data, wire pipeline to Neon, enable Claude
Models:
- LSTM retrained on 727 days of real NASA POWER data (20 zones, val AUROC 0.92)
- XGBoost retrained on same data (val AUROC 0.86)
- UHI model kept on literature-calibrated synthetic (no ground truth available)
Database:
- Pipeline writes to Neon after each step (daily_readings, healed_readings,
heat_indices, predictions, trigger_events, pipeline_runs)
- All DB writes wrapped in try/except — pipeline never crashes on DB errors
- run_pipeline.py updated: loads .env, uses HeatRiskPipeline, async DB init
Claude:
- Healing and explanation enabled by default (use_claude_healer=True)
- Activates when ANTHROPIC_API_KEY is set (HF Spaces secrets)
- Graceful fallback to rule-based when key is missing
Also: dotenv loading in api.py, zones seeded to Neon on startup,
training script at scripts/train_on_nasa_power.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Closes the AI/ML sophistication gap with Weather AI 2:
- ERA5 real climate data via Google ARCO archive (free, no API key)
- PyTorch LSTM heat predictor ensembled with XGBoost (4-tier degradation)
- FAISS+BM25 hybrid RAG for alert generation (44-doc index)
- Neon PostgreSQL schema (11 heat-specific tables) with asyncpg + InMemory fallback
- 19-test eval suite (UHI accuracy, predictor AUROC, RAG precision, healing recall)
- Fix broken flood-era pipeline/store/schema imports
- De-jargonify all frontend text for donor/program officer audience
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
6-step ML pipeline for parametric heat insurance in East Africa.
Three XGBoost models (UHI correction, heat wave prediction, actuarial pricing),
Claude AI healing agent, interactive program designer with budget allocation.
20 urban zones across Nairobi, Dar es Salaam, Kampala, Kigali.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
