Spaces:
Sleeping
Sleeping
Anna Zhou commited on
Commit ·
318732d
1
Parent(s): 51b3964
Fix all-case summaries for Hugging Face Space
Browse files- AGENTS.md +290 -0
- README.md +11 -8
- app/app.py +21 -9
- case_study_pipeline/onboard_case_study.py +1 -1
- case_study_pipeline/select_best_run.py +2 -2
- data/extracted/case_summaries.json +9 -0
- data/extracted/precomputed_embeddings.json +0 -0
- docs/extraction_schema.md +79 -22
- ingest/create_summary_chunks.py +27 -0
- ingest/embed_and_index.py +21 -3
- ingest/ingest_pipeline_outputs.py +4 -4
- rag/pipeline.py +6 -6
- rag/prompt_builder.py +53 -13
AGENTS.md
ADDED
|
@@ -0,0 +1,290 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# RFR Co-op: LLM-Assisted Managed Retreat Pipeline
|
| 2 |
+
|
| 3 |
+
> This file is read automatically by Codex. It provides full project context so every session starts with shared understanding.
|
| 4 |
+
|
| 5 |
+
## What this project is
|
| 6 |
+
|
| 7 |
+
This is a 4-month co-op research project embedded in the **Retreat From Risk (RFR)** project — a NFRF-funded international research initiative (Canada, USA, Indonesia) studying **managed retreat (MR)** as a climate adaptation strategy for flood-prone communities.
|
| 8 |
+
|
| 9 |
+
The co-op goal is to build an **LLM-assisted framework** for extracting and synthesizing insights from MR case study literature, and to deliver a working **RAG pipeline** that helps decision-makers (municipal planners, First Nations leaders) query those insights through a guided questionnaire and persona-aware interface.
|
| 10 |
+
|
| 11 |
+
**Supervisors:** Dr. Costa (systems engineering, UW) and Dr. Doberstein (geography, UW)
|
| 12 |
+
**PhD collaborator:** Ana (has an existing human-coded extraction framework — this project compares LLM vs human extraction)
|
| 13 |
+
**GitHub:** https://github.com/AnnasBeef
|
| 14 |
+
**Linear:** https://linear.app/managed-retreat (team: Managed Retreat, issues prefixed MAN-)
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## Working in Cowork (read this before running the pipeline here)
|
| 19 |
+
|
| 20 |
+
The Cowork sandbox is **ephemeral** (installed packages don't persist between sessions) and its **egress proxy blocks the LLM APIs**. To avoid re-diagnosing this every time:
|
| 21 |
+
|
| 22 |
+
- **Setup once per session:** `bash cowork_setup.sh` — installs only the minimal deps (`requirements-cowork.txt`), not the heavy RAG stack (`torch`, `sentence-transformers`) in `requirements.txt`.
|
| 23 |
+
- **The live API run does NOT work in Cowork.** `api.anthropic.com` returns a proxy `401`; `api.openai.com` is unreachable. Run the full 4-step pipeline on your **Mac**, not here.
|
| 24 |
+
- **If attempting outbound HTTPS from Python here,** the proxy uses a self-signed cert — prefix with `SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt`.
|
| 25 |
+
- **Works fine offline in Cowork:** PDF rendering (`pdf_utils.text_to_pdf`), source-text extraction (`pdfplumber`), manual Ver1/Ver2 drafting.
|
| 26 |
+
|
| 27 |
+
### Finding & downloading source papers for a case study
|
| 28 |
+
|
| 29 |
+
1. First try `python3 -m case_study_pipeline.fetch_papers --case-id CSxx` —
|
| 30 |
+
pulls open-access PDFs automatically (Unpaywall, Semantic Scholar, Sci-Hub).
|
| 31 |
+
Also blocked by the Cowork egress proxy; run on the Mac.
|
| 32 |
+
2. For anything still paywalled, use UWaterloo's institutional library
|
| 33 |
+
access (EZproxy) via the Codex-in-Chrome browser extension — **this
|
| 34 |
+
requires a human to complete WatIAM login + Duo 2FA, it cannot be
|
| 35 |
+
scripted**. Full step-by-step procedure, including the one-time Chrome
|
| 36 |
+
download-folder setting, is in
|
| 37 |
+
`case_study_pipeline/README.md` under "Fetching paywalled papers via
|
| 38 |
+
UWaterloo EZproxy (Chrome-assisted)". Read that section before attempting
|
| 39 |
+
this — it documents the exact URL pattern (`ezproxy.py`) and the
|
| 40 |
+
inbox-sorting + Drive-upload script (`sort_inbox.py`).
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## Core research questions
|
| 45 |
+
|
| 46 |
+
1. Can LLMs reliably extract key information from MR case studies compared to human coders?
|
| 47 |
+
2. How does question structure (the guided questionnaire) affect LLM output quality?
|
| 48 |
+
3. What does a useful decision-support tool look like for a community leader considering managed retreat?
|
| 49 |
+
|
| 50 |
+
---
|
| 51 |
+
|
| 52 |
+
## What we're building
|
| 53 |
+
|
| 54 |
+
### Pipeline architecture
|
| 55 |
+
|
| 56 |
+
```
|
| 57 |
+
MR case studies (PDFs, spreadsheets)
|
| 58 |
+
↓
|
| 59 |
+
Case-study source catalog (Ajibade et al. spreadsheet + local case_meta.json)
|
| 60 |
+
↓
|
| 61 |
+
LLM extraction/audit pipeline (Ver1 → Check1 → Ver2 → Check2)
|
| 62 |
+
↓
|
| 63 |
+
Section-level JSON chunks in data/extracted/
|
| 64 |
+
↓
|
| 65 |
+
Precomputed embedding store (data/extracted/precomputed_embeddings.json)
|
| 66 |
+
↓
|
| 67 |
+
NumPy cosine retrieval (semantic search)
|
| 68 |
+
↓
|
| 69 |
+
Persona layer (decision-maker framing)
|
| 70 |
+
↓
|
| 71 |
+
Guided questionnaire (shapes the query)
|
| 72 |
+
↓
|
| 73 |
+
LLM response (grounded, source-cited)
|
| 74 |
+
↓
|
| 75 |
+
Web interface (hosted on AWS / HuggingFace Spaces)
|
| 76 |
+
```
|
| 77 |
+
|
| 78 |
+
### Month 1 deliverables (by ~June 6)
|
| 79 |
+
- [ ] v0 RAG pipeline running locally (MAN-16)
|
| 80 |
+
- [ ] Decision-maker persona defined (MAN-11)
|
| 81 |
+
- [ ] Guided questionnaire v1 (MAN-12)
|
| 82 |
+
- [ ] Persona + questionnaire integrated into RAG (MAN-13)
|
| 83 |
+
- [ ] Cloud hosting decision made (MAN-17)
|
| 84 |
+
|
| 85 |
+
---
|
| 86 |
+
|
| 87 |
+
## Tech stack
|
| 88 |
+
|
| 89 |
+
| Layer | Tool | Notes |
|
| 90 |
+
|---|---|---|
|
| 91 |
+
| Language | Python 3.11+ | |
|
| 92 |
+
| Retrieval store | `data/extracted/precomputed_embeddings.json` | JSON file with chunk text, metadata, embeddings, and a content hash |
|
| 93 |
+
| Embeddings | sentence-transformers | Free, local (`all-MiniLM-L6-v2`) |
|
| 94 |
+
| Retrieval | NumPy cosine similarity | Runtime replacement for ChromaDB; see `rag/retriever.py` |
|
| 95 |
+
| LLM (local fallback) | Ollama | `rag/pipeline.py` currently calls Ollama model `gemma4` for non-Claude/OpenAI mode |
|
| 96 |
+
| Web interface | Streamlit | `app/app.py` |
|
| 97 |
+
| Hosting | HuggingFace Spaces (dev) → AWS EC2 (prod) | |
|
| 98 |
+
| Project tracking | Linear (MAN- issues) | |
|
| 99 |
+
| Version control | GitHub (AnnasBeef) | Branch names: `MAN-XX-short-description` |
|
| 100 |
+
|
| 101 |
+
---
|
| 102 |
+
|
| 103 |
+
## Repo structure
|
| 104 |
+
|
| 105 |
+
```
|
| 106 |
+
rfr-rag/
|
| 107 |
+
├── AGENTS.md ← you are here
|
| 108 |
+
├── data/
|
| 109 |
+
│ ├── raw/ # original PDFs and spreadsheets
|
| 110 |
+
│ └── extracted/ # Ver2-derived JSON chunks + precomputed embeddings
|
| 111 |
+
├── ingest/
|
| 112 |
+
│ ├── ingest_pipeline_outputs.py
|
| 113 |
+
│ ├── create_summary_chunks.py
|
| 114 |
+
│ └── embed_and_index.py
|
| 115 |
+
├── rag/
|
| 116 |
+
│ ├── retriever.py
|
| 117 |
+
│ ├── prompt_builder.py
|
| 118 |
+
│ └── pipeline.py
|
| 119 |
+
├── persona/
|
| 120 |
+
│ └── decision_maker.json # persona config
|
| 121 |
+
├── questionnaire/
|
| 122 |
+
│ └── questions.json # guided questionnaire structure
|
| 123 |
+
├── app/
|
| 124 |
+
│ └── app.py # Streamlit/Gradio interface
|
| 125 |
+
├── case_study_pipeline/ # Ver1/Check1/Ver2/Check2 extraction workflow
|
| 126 |
+
├── docs/
|
| 127 |
+
│ └── extraction_schema.md
|
| 128 |
+
├── tests/
|
| 129 |
+
└── requirements.txt
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
---
|
| 133 |
+
|
| 134 |
+
## Current extraction/indexing format
|
| 135 |
+
|
| 136 |
+
The original plan used a compact structured schema. The current RAG index uses
|
| 137 |
+
long-form Ver2 memo sections instead:
|
| 138 |
+
|
| 139 |
+
```json
|
| 140 |
+
{
|
| 141 |
+
"case_id": "CS24",
|
| 142 |
+
"chunk_index": 0,
|
| 143 |
+
"section": "overview",
|
| 144 |
+
"location": "Queens, New York",
|
| 145 |
+
"country": "United States",
|
| 146 |
+
"source": "Pipeline Ver2 — Queens",
|
| 147 |
+
"text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
|
| 148 |
+
}
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
+
The active index is built by:
|
| 152 |
+
|
| 153 |
+
1. `python3 -m case_study_pipeline.select_best_run --case-id CSxx` if multiple runs exist.
|
| 154 |
+
2. `python3 ingest/ingest_pipeline_outputs.py --case-id CSxx` to convert canonical `CSxx_Ver2.md` into `data/extracted/CSxx.json`.
|
| 155 |
+
3. `python3 ingest/create_summary_chunks.py` to refresh one overview chunk per case in `data/extracted/case_summaries.json`.
|
| 156 |
+
4. `python3 ingest/embed_and_index.py` to refresh `data/extracted/precomputed_embeddings.json`.
|
| 157 |
+
|
| 158 |
+
`embed_and_index.py` checks chunk IDs, text, metadata, and a content hash. If a
|
| 159 |
+
Ver2 memo or JSON chunk changes, rebuild the embedding file before testing the app.
|
| 160 |
+
|
| 161 |
+
---
|
| 162 |
+
|
| 163 |
+
## Case studies and source spreadsheet
|
| 164 |
+
|
| 165 |
+
The expanded case-study catalog is based on the Ajibade et al. spreadsheet:
|
| 166 |
+
`https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`
|
| 167 |
+
|
| 168 |
+
As of July 14, 2026, the local RAG knowledge base has canonical Ver2 outputs,
|
| 169 |
+
extracted JSON, and embeddings for 34 cases:
|
| 170 |
+
|
| 171 |
+
`CS1`-`CS31`, `CS33`, `CS34`, and `CS35`.
|
| 172 |
+
|
| 173 |
+
`CS32` exists in the spreadsheet/local metadata as Nono District / Jiru Gamachu
|
| 174 |
+
Resettlement Site, Ethiopia, but does not currently have a canonical Ver2 output
|
| 175 |
+
or extracted JSON in this repo.
|
| 176 |
+
|
| 177 |
+
Current CS1-CS20 local names from `case_meta.json` and canonical Ver2 outputs:
|
| 178 |
+
|
| 179 |
+
| ID | Current local name | Location | Country |
|
| 180 |
+
|----|--------------------|----------|---------|
|
| 181 |
+
| CS1 | Isle de Jean Charles Resettlement | Isle de Jean Charles, Terrebonne Parish, Louisiana | USA |
|
| 182 |
+
| CS2 | Shaanxi Ecological Resettlement | Shaanxi Province | China |
|
| 183 |
+
| CS3 | Oakwood Beach Buyout Program | Oakwood Beach, Staten Island, New York | USA |
|
| 184 |
+
| CS4 | Metro Manila Resettlement (Oplan LIKAS) | Metro Manila | Philippines |
|
| 185 |
+
| CS5 | Matata Managed Retreat | Matata, Bay of Plenty | New Zealand |
|
| 186 |
+
| CS6 | Pointe-Gatineau Home Buyout Program | Pointe-Gatineau, Quebec | Canada |
|
| 187 |
+
| CS7 | Hambantota Post-Tsunami Relocation | Hambantota | Sri Lanka |
|
| 188 |
+
| CS8 | Cedar Rapids Buyout Program | Cedar Rapids, Iowa | USA |
|
| 189 |
+
| CS9 | Grand Forks Floodplain Buyout | Grand Forks, North Dakota | USA |
|
| 190 |
+
| CS10 | San Juan Managed Retreat | San Juan, Puerto Rico | USA |
|
| 191 |
+
| CS11 | Terrebonne Parish Resettlement | Terrebonne Parish, Louisiana | USA |
|
| 192 |
+
| CS12 | Dhye Community Relocation | Dhye, Mustang District | Nepal |
|
| 193 |
+
| CS13 | Soldiers Grove Floodplain Relocation | Soldiers Grove, Wisconsin | USA |
|
| 194 |
+
| CS14 | Simbach am Inn Household Relocation | Simbach am Inn, Bavaria | Germany |
|
| 195 |
+
| CS15 | Tegua Island Planned Relocation | Tegua, Torba Province | Vanuatu |
|
| 196 |
+
| CS16 | Vunidogola Village Relocation | Vunidogola, Koralau Island | Fiji |
|
| 197 |
+
| CS17 | San Martin Planned Relocation | San Martin Region | Peru |
|
| 198 |
+
| CS18 | High River Buyout Program | High River, Alberta | Canada |
|
| 199 |
+
| CS19 | Nusa Hope Village Relocation | Nusa Hope, New Georgia | Solomon Islands |
|
| 200 |
+
| CS20 | East Riding Managed Retreat | East Riding of Yorkshire | UK |
|
| 201 |
+
|
| 202 |
+
---
|
| 203 |
+
|
| 204 |
+
## Decision-maker persona (v0)
|
| 205 |
+
|
| 206 |
+
The RAG pipeline frames responses for this user:
|
| 207 |
+
|
| 208 |
+
- **Role:** Municipal planner or First Nations community leader
|
| 209 |
+
- **Knowledge level:** Non-technical; familiar with community planning, not with ML or hydrology
|
| 210 |
+
- **Key concerns:** Equity, community buy-in, funding availability, cultural impacts, legal barriers
|
| 211 |
+
- **Preferred tone:** Plain language, practical, grounded in real examples
|
| 212 |
+
- **What "useful" looks like:** "Here's what worked in a similar community, here's what to watch out for, here are the key decisions you need to make"
|
| 213 |
+
|
| 214 |
+
---
|
| 215 |
+
|
| 216 |
+
## Linear issues (current)
|
| 217 |
+
|
| 218 |
+
| ID | Title | Due | Priority |
|
| 219 |
+
|----|-------|-----|----------|
|
| 220 |
+
| MAN-5 | Review human-coded extraction framework with Ana | May 16 | Urgent |
|
| 221 |
+
| MAN-6 | Read key MR case studies and synthesis papers | May 23 | High |
|
| 222 |
+
| MAN-7 | Build structured extraction schema | May 23 | High |
|
| 223 |
+
| MAN-8 | Apply framework to Isle de Jean Charles | May 23 | High |
|
| 224 |
+
| MAN-9 | Write Week 2 reflection memo | May 23 | Medium |
|
| 225 |
+
| MAN-10 | Build basic RAG pipeline | May 16 | High |
|
| 226 |
+
| MAN-11 | Define decision-maker persona | May 16 | High |
|
| 227 |
+
| MAN-12 | Design guided questionnaire | May 29 | High |
|
| 228 |
+
| MAN-13 | Integrate persona + questionnaire into RAG | May 29 | Medium |
|
| 229 |
+
| MAN-14 | Set up GitHub repo (AnnasBeef) | May 12 | Urgent |
|
| 230 |
+
| MAN-15 | Connect GitHub to Linear and Codex | May 12 | Urgent |
|
| 231 |
+
| MAN-16 | Build v0 RAG pipeline locally today | May 11 | Urgent |
|
| 232 |
+
| MAN-17 | Research cloud hosting strategy | May 16 | High |
|
| 233 |
+
|
| 234 |
+
Branch naming convention: `MAN-XX-short-description`
|
| 235 |
+
e.g. `MAN-16-rag-v0-local`
|
| 236 |
+
|
| 237 |
+
---
|
| 238 |
+
|
| 239 |
+
## Key files in this repo to read first
|
| 240 |
+
|
| 241 |
+
- `data/extracted/` — Ver2-derived JSON case study chunks and embedding index
|
| 242 |
+
- `case_study_pipeline/README.md` — end-to-end extraction/audit workflow
|
| 243 |
+
- `ingest/ingest_pipeline_outputs.py` — converts canonical Ver2 memos into JSON chunks
|
| 244 |
+
- `ingest/create_summary_chunks.py` — builds overview chunks for summary queries
|
| 245 |
+
- `ingest/embed_and_index.py` — builds `precomputed_embeddings.json`
|
| 246 |
+
- `rag/pipeline.py` — main RAG query loop
|
| 247 |
+
- `persona/decision_maker.json` — persona config used in prompt construction
|
| 248 |
+
- `questionnaire/questions.json` — guided questionnaire structure
|
| 249 |
+
|
| 250 |
+
---
|
| 251 |
+
|
| 252 |
+
## Quick start (local dev)
|
| 253 |
+
|
| 254 |
+
```bash
|
| 255 |
+
# 1. Install dependencies
|
| 256 |
+
pip install -r requirements.txt
|
| 257 |
+
|
| 258 |
+
# 2. Optional: start local LLM fallback
|
| 259 |
+
ollama pull gemma4
|
| 260 |
+
ollama run gemma4
|
| 261 |
+
|
| 262 |
+
# 3. Convert canonical Ver2 memos to JSON chunks when case outputs change
|
| 263 |
+
python ingest/ingest_pipeline_outputs.py
|
| 264 |
+
python ingest/create_summary_chunks.py
|
| 265 |
+
|
| 266 |
+
# 4. Build or refresh the precomputed embedding index
|
| 267 |
+
python ingest/embed_and_index.py
|
| 268 |
+
|
| 269 |
+
# 5. Run the RAG pipeline
|
| 270 |
+
python rag/pipeline.py
|
| 271 |
+
|
| 272 |
+
# 6. Launch the web app
|
| 273 |
+
streamlit run app/app.py
|
| 274 |
+
```
|
| 275 |
+
|
| 276 |
+
---
|
| 277 |
+
|
| 278 |
+
## Key concepts to know
|
| 279 |
+
|
| 280 |
+
**Managed retreat (MR):** The purposeful relocation of people, property, and infrastructure out of areas vulnerable to recurrent climatic hazards. Ranges from voluntary property buyouts to forced community resettlement.
|
| 281 |
+
|
| 282 |
+
**RAG (Retrieval-Augmented Generation):** An LLM pattern where relevant documents are retrieved from a vector store and included in the prompt, so the model's response is grounded in specific source material rather than relying on training data alone.
|
| 283 |
+
|
| 284 |
+
**Decision Support Framework (DSF):** The broader research output of the RFR project — a framework to help communities evaluate MR as a viable adaptation strategy. This pipeline is a technical tool to support the DSF.
|
| 285 |
+
|
| 286 |
+
**OCAP principles:** Ownership, Control, Access, Possession — Indigenous data governance principles that must be respected when working with First Nations case study data.
|
| 287 |
+
|
| 288 |
+
---
|
| 289 |
+
|
| 290 |
+
*Last updated: May 11, 2026 — generated from Codex.ai project context*
|
README.md
CHANGED
|
@@ -27,7 +27,9 @@ pip install -r requirements.txt
|
|
| 27 |
export ANTHROPIC_API_KEY=your_key_here
|
| 28 |
export OPENAI_API_KEY=your_key_here
|
| 29 |
|
| 30 |
-
# 2.
|
|
|
|
|
|
|
| 31 |
python ingest/embed_and_index.py
|
| 32 |
|
| 33 |
# 3. Run CLI pipeline
|
|
@@ -40,13 +42,13 @@ streamlit run app/app.py
|
|
| 40 |
## Ingesting documents
|
| 41 |
|
| 42 |
```bash
|
| 43 |
-
#
|
| 44 |
-
python ingest/
|
| 45 |
|
| 46 |
-
#
|
| 47 |
-
python ingest/
|
| 48 |
|
| 49 |
-
#
|
| 50 |
python ingest/embed_and_index.py
|
| 51 |
```
|
| 52 |
|
|
@@ -55,8 +57,9 @@ python ingest/embed_and_index.py
|
|
| 55 |
```
|
| 56 |
rfr-rag/
|
| 57 |
├── data/raw/ # Original PDFs and spreadsheets (not committed)
|
| 58 |
-
├── data/extracted/ # JSON
|
| 59 |
-
├── ingest/ #
|
|
|
|
| 60 |
├── rag/ # Retriever, prompt builder, pipeline
|
| 61 |
├── persona/ # Decision-maker persona config
|
| 62 |
├── questionnaire/ # Guided questionnaire structure
|
|
|
|
| 27 |
export ANTHROPIC_API_KEY=your_key_here
|
| 28 |
export OPENAI_API_KEY=your_key_here
|
| 29 |
|
| 30 |
+
# 2. Convert canonical Ver2 memos to JSON chunks, then build embeddings
|
| 31 |
+
python ingest/ingest_pipeline_outputs.py
|
| 32 |
+
python ingest/create_summary_chunks.py
|
| 33 |
python ingest/embed_and_index.py
|
| 34 |
|
| 35 |
# 3. Run CLI pipeline
|
|
|
|
| 42 |
## Ingesting documents
|
| 43 |
|
| 44 |
```bash
|
| 45 |
+
# Convert canonical case-study memos into section chunks
|
| 46 |
+
python ingest/ingest_pipeline_outputs.py
|
| 47 |
|
| 48 |
+
# Refresh overview-summary chunks for broad comparison queries
|
| 49 |
+
python ingest/create_summary_chunks.py
|
| 50 |
|
| 51 |
+
# Build the precomputed embedding store used by rag/retriever.py
|
| 52 |
python ingest/embed_and_index.py
|
| 53 |
```
|
| 54 |
|
|
|
|
| 57 |
```
|
| 58 |
rfr-rag/
|
| 59 |
├── data/raw/ # Original PDFs and spreadsheets (not committed)
|
| 60 |
+
├── data/extracted/ # Ver2-derived JSON chunks + embedding store
|
| 61 |
+
├── ingest/ # Ver2 ingestion, summary chunks, embedding script
|
| 62 |
+
├── case_study_pipeline/# Source extraction/audit workflow
|
| 63 |
├── rag/ # Retriever, prompt builder, pipeline
|
| 64 |
├── persona/ # Decision-maker persona config
|
| 65 |
├── questionnaire/ # Guided questionnaire structure
|
app/app.py
CHANGED
|
@@ -10,14 +10,26 @@ sys.path.insert(0, str(Path(__file__).parent))
|
|
| 10 |
from sheets_logger import log_query
|
| 11 |
|
| 12 |
|
| 13 |
-
def
|
| 14 |
"""Ensure precomputed_embeddings.json has texts+metadatas (needed by numpy retriever)."""
|
| 15 |
precomputed = Path("data/extracted/precomputed_embeddings.json")
|
| 16 |
needs_rebuild = True
|
| 17 |
if precomputed.exists():
|
| 18 |
import json as _json
|
| 19 |
pre = _json.loads(precomputed.read_text())
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
print(f"Embeddings ready ({len(pre['ids'])} chunks).")
|
| 22 |
needs_rebuild = False
|
| 23 |
if needs_rebuild:
|
|
@@ -31,8 +43,8 @@ DATA_RAW = Path("data/raw")
|
|
| 31 |
|
| 32 |
|
| 33 |
@st.cache_data
|
| 34 |
-
def
|
| 35 |
-
"""Load case metadata for all case studies
|
| 36 |
meta = {}
|
| 37 |
if not DATA_RAW.exists():
|
| 38 |
return meta
|
|
@@ -68,8 +80,8 @@ def _clean_excerpt(text: str, max_chars: int = 800) -> str:
|
|
| 68 |
return text
|
| 69 |
|
| 70 |
|
| 71 |
-
# ── Build
|
| 72 |
-
|
| 73 |
|
| 74 |
# ── Page config ───────────────────────────────────────────────────────────────
|
| 75 |
st.set_page_config(page_title="AI-Assisted Knowledge Platform for Managed Retreat", layout="wide")
|
|
@@ -207,7 +219,7 @@ with main_col:
|
|
| 207 |
st.warning("Please enter a question.")
|
| 208 |
elif re.search(r"\b(what|which|list|show|how many).{0,30}case stud", query, re.I):
|
| 209 |
# Meta question — list all case studies directly from extracted JSONs
|
| 210 |
-
cs_meta =
|
| 211 |
extracted_dir = Path("data/extracted")
|
| 212 |
all_cases = []
|
| 213 |
for f in sorted(extracted_dir.glob("CS*.json"), key=lambda p: int(re.search(r'\d+', p.stem).group())):
|
|
@@ -264,7 +276,7 @@ with main_col:
|
|
| 264 |
st.markdown(f"**{q_text}** {val}")
|
| 265 |
|
| 266 |
# ── Case studies used ─────────────────────────────────────────
|
| 267 |
-
cs_meta =
|
| 268 |
|
| 269 |
seen_ids: set[str] = set()
|
| 270 |
retrieved_cs = []
|
|
@@ -275,7 +287,7 @@ with main_col:
|
|
| 275 |
retrieved_cs.append(c)
|
| 276 |
|
| 277 |
mentioned_ids = {
|
| 278 |
-
f"CS{n}" for n in re.findall(r'\bCS(
|
| 279 |
}
|
| 280 |
|
| 281 |
if retrieved_cs:
|
|
|
|
| 10 |
from sheets_logger import log_query
|
| 11 |
|
| 12 |
|
| 13 |
+
def _ensure_embedding_index_built():
|
| 14 |
"""Ensure precomputed_embeddings.json has texts+metadatas (needed by numpy retriever)."""
|
| 15 |
precomputed = Path("data/extracted/precomputed_embeddings.json")
|
| 16 |
needs_rebuild = True
|
| 17 |
if precomputed.exists():
|
| 18 |
import json as _json
|
| 19 |
pre = _json.loads(precomputed.read_text())
|
| 20 |
+
from ingest.embed_and_index import _content_hash, load_all_chunks
|
| 21 |
+
|
| 22 |
+
chunks = load_all_chunks()
|
| 23 |
+
texts = [c.get("text", _json.dumps(c)) for c in chunks]
|
| 24 |
+
ids = [f"{c.get('case_id', 'unknown')}_{c.get('chunk_index', i)}" for i, c in enumerate(chunks)]
|
| 25 |
+
metadatas = [{k: str(v) for k, v in c.items() if k != "text"} for c in chunks]
|
| 26 |
+
content_hash = _content_hash(ids, texts, metadatas)
|
| 27 |
+
if (
|
| 28 |
+
pre.get("ids") == ids
|
| 29 |
+
and pre.get("texts") == texts
|
| 30 |
+
and pre.get("metadatas") == metadatas
|
| 31 |
+
and pre.get("content_hash") == content_hash
|
| 32 |
+
):
|
| 33 |
print(f"Embeddings ready ({len(pre['ids'])} chunks).")
|
| 34 |
needs_rebuild = False
|
| 35 |
if needs_rebuild:
|
|
|
|
| 43 |
|
| 44 |
|
| 45 |
@st.cache_data
|
| 46 |
+
def load_case_meta() -> dict[str, dict]:
|
| 47 |
+
"""Load case metadata for all case studies with local case_meta.json files."""
|
| 48 |
meta = {}
|
| 49 |
if not DATA_RAW.exists():
|
| 50 |
return meta
|
|
|
|
| 80 |
return text
|
| 81 |
|
| 82 |
|
| 83 |
+
# ── Build embedding index before any Streamlit calls ──────────────────────────
|
| 84 |
+
_ensure_embedding_index_built()
|
| 85 |
|
| 86 |
# ── Page config ───────────────────────────────────────────────────────────────
|
| 87 |
st.set_page_config(page_title="AI-Assisted Knowledge Platform for Managed Retreat", layout="wide")
|
|
|
|
| 219 |
st.warning("Please enter a question.")
|
| 220 |
elif re.search(r"\b(what|which|list|show|how many).{0,30}case stud", query, re.I):
|
| 221 |
# Meta question — list all case studies directly from extracted JSONs
|
| 222 |
+
cs_meta = load_case_meta()
|
| 223 |
extracted_dir = Path("data/extracted")
|
| 224 |
all_cases = []
|
| 225 |
for f in sorted(extracted_dir.glob("CS*.json"), key=lambda p: int(re.search(r'\d+', p.stem).group())):
|
|
|
|
| 276 |
st.markdown(f"**{q_text}** {val}")
|
| 277 |
|
| 278 |
# ── Case studies used ─────────────────────────────────────────
|
| 279 |
+
cs_meta = load_case_meta()
|
| 280 |
|
| 281 |
seen_ids: set[str] = set()
|
| 282 |
retrieved_cs = []
|
|
|
|
| 287 |
retrieved_cs.append(c)
|
| 288 |
|
| 289 |
mentioned_ids = {
|
| 290 |
+
f"CS{n}" for n in re.findall(r'\bCS(\d+)\b', answer)
|
| 291 |
}
|
| 292 |
|
| 293 |
if retrieved_cs:
|
case_study_pipeline/onboard_case_study.py
CHANGED
|
@@ -326,7 +326,7 @@ def main() -> None:
|
|
| 326 |
copied = copy_pdfs_from_drive_folder(service, source_id, folder_id, local_dest)
|
| 327 |
print(f"✓ Copied {len(copied)} PDF(s) into the Drive folder and into {local_dest}/")
|
| 328 |
|
| 329 |
-
# 3. Write case_meta.json so the ingestion bridge can tag
|
| 330 |
import json as _json
|
| 331 |
_meta_path = REPO_ROOT / "data" / "raw" / case_id / "case_meta.json"
|
| 332 |
_meta_path.write_text(_json.dumps({
|
|
|
|
| 326 |
copied = copy_pdfs_from_drive_folder(service, source_id, folder_id, local_dest)
|
| 327 |
print(f"✓ Copied {len(copied)} PDF(s) into the Drive folder and into {local_dest}/")
|
| 328 |
|
| 329 |
+
# 3. Write case_meta.json so the ingestion bridge can tag RAG chunks
|
| 330 |
import json as _json
|
| 331 |
_meta_path = REPO_ROOT / "data" / "raw" / case_id / "case_meta.json"
|
| 332 |
_meta_path.write_text(_json.dumps({
|
case_study_pipeline/select_best_run.py
CHANGED
|
@@ -6,7 +6,7 @@ reads the Check 2 (or Check 1) grade from each, selects the winner, copies its
|
|
| 6 |
Ver2.md to the canonical location (pipeline_output/CSX_Ver2.md), and writes
|
| 7 |
pipeline_output/selected.json.
|
| 8 |
|
| 9 |
-
The canonical Ver2.md is what ingest_pipeline_outputs.py reads into
|
| 10 |
|
| 11 |
Usage:
|
| 12 |
python -m case_study_pipeline.select_best_run --case-id CS1
|
|
@@ -167,7 +167,7 @@ def main(argv: list[str] | None = None) -> None:
|
|
| 167 |
for case_id in case_ids:
|
| 168 |
select_for_case(case_id, force=args.force)
|
| 169 |
|
| 170 |
-
print("\nDone. Run ingest/ingest_pipeline_outputs.py to
|
| 171 |
|
| 172 |
|
| 173 |
if __name__ == "__main__":
|
|
|
|
| 6 |
Ver2.md to the canonical location (pipeline_output/CSX_Ver2.md), and writes
|
| 7 |
pipeline_output/selected.json.
|
| 8 |
|
| 9 |
+
The canonical Ver2.md is what ingest_pipeline_outputs.py reads into JSON chunks.
|
| 10 |
|
| 11 |
Usage:
|
| 12 |
python -m case_study_pipeline.select_best_run --case-id CS1
|
|
|
|
| 167 |
for case_id in case_ids:
|
| 168 |
select_for_case(case_id, force=args.force)
|
| 169 |
|
| 170 |
+
print("\nDone. Run ingest/ingest_pipeline_outputs.py to convert selected Ver2 files into JSON chunks.")
|
| 171 |
|
| 172 |
|
| 173 |
if __name__ == "__main__":
|
data/extracted/case_summaries.json
CHANGED
|
@@ -89,6 +89,15 @@
|
|
| 89 |
"source": "Pipeline Ver2 — San Juan Managed Retreat",
|
| 90 |
"text": "Case study: San Juan Managed Retreat (San Juan, Puerto Rico, USA)\nSection: Case Study Overview (Summary)\n\n## Case Name and Location\n**El Caño Martín Peña (CMP) — Eight Communities Aledañas al Caño Martín Peña Special Planning District**\nSan Juan, Puerto Rico — a U.S. territory with a distinct colonial governance context (classified as Global North by project codebook definition, though Puerto Rico's territorial and colonial status introduces important caveats)\n\n## Hazard(s)\nPrimary hazard: **riverine/tidal flooding**, driven by the progressive narrowing and blockage of a 3.75-mile tidal channel (El Caño Martín Peña) connecting the San José Lagoon to the San Juan Bay Estuary. The channel was severely obstructed by decades of informal landfill (debris, trash, wetland conversion), leading to chronic flooding that historically affected approximately 70% of the area [Source 1]. Secondary hazard: **hurricane events**, including the 1928 San Felipe and 1932 San Ciprián hurricanes (which originally accelerated migration into the area) and Hurricanes Irma and María (September 2017), after which 1,200 homes lost their roofs and 100 homes were destroyed within the eight communities [Source 1].\n\n## Type of Retreat\nA **community-driven, participatory managed retreat** embedded within a broader ecosystem restoration and community redevelopment initiative. The retreat component requires approximately 1,000 families who live along the canal right-of-way to relocate to allow for channel widening and dredging. Households are offered a choice between relocating within the eight communities (within the CLT footprint) or accepting a housing relocation voucher to move elsewhere. It is important to note that relocation is not purely voluntary in the conventional sense: households along the right-of-way *must* move for dredging to proceed; what is choice-based is the *destination and form* of their new housing, not whether relocation occurs at all [Source 1, Source 2]. The retreat is explicitly framed as inseparable from environmental restoration, flood mitigation, and long-term community resilience. It is not a standalone buyout program.\n\n## Scale\n- **Population**: Approximately 10,997 current residents; the population had previously been estimated at approximately 15,693–18,103. Sources identify a range of possible contributing factors to this decline, including governmental discrimination and stigmatization of the communities, though causation is not established [Source 1].\n- **Housing units**: 6,946 total units in the eight communities; approximately 1,000 families required to relocate for the dredging right-of-way; approximately 600 families had already moved into safe housing as of 2017 [Source 1, Source 2].\n- **Land/tenure**: The Community Land Trust (*Fideicomiso de la Tierra del Caño Martín Peña*) holds collective tenure on behalf of approximately 2,000 families (around 16.51% of the area's households), who share collective clear title [Source 1]. Source 2 states that \"the community collectively owns 200 acres (80 ha) of land\" [Source 2]; this figure is cited from Source 2 (Yarina et al. 2020) and is not independently confirmed in Source 1.\n- **Geographic scope**: Eight neighborhoods — Barrio Obrero Oeste, Barrio Obrero, San Ciprián, Parada 27, Buena Vista Santurce, Las Monjas, Buena Vista Hato Rey, and Israel–Bitumul — forming the Martín Peña Special Planning District [Source 1].\n\n## Key Actors\n- **G8 Coalition** (*Grupo de las Ocho Comunidades Aledañas al Caño Martín Peña*): Community coalition of the eight neighborhoods; central community governance body. Formed in 2004 [Source 1].\n- **Proyecto ENLACE del Caño Martín Peña**: Public corporation established by Puerto Rico Law 489 (2004) to manage environmental restoration and community development initiatives; serves as institutional intermediary between community and government [Source 1, Source 2].\n- **Fideicomiso de la Tierra del Caño Martín Peña (CLT)**: Community Land Trust; a private, non-profit organization created to manage the land, secure collective tenure, and prevent residents from being priced out [Source 2]. Provides housing opportunities for relocating families and protects residents from speculative displacement [Source 1, Source 2].\n- **Government of Puerto Rico / Puerto Rico Department of Housing**: State-level authority administering CDBG-DR and CDBG-MIT funds; responsible for Program R3 (Repair, Reconstruction, or Relocation) [Source 3].\n- **U.S. Army Corps of Engineers**: Dredging project partner; government of San Juan partnered with the Corps in 2001 to plan canal restoration; $163 million assigned to the Corps for dredging in 2022 [Source 1].\n- **U.S. Department of Housing and Urban Development (HUD)**: Federal funder; provided $46 million in CDBG-MIT funds for relocation [Source 1].\n- **U.S. Environmental Protection Agency (EPA)**: Urban Waters Federal Partnership participant [Source 1].\n- **Residents / community members**: Primary decision-makers and participants throughout all phases; the community-driven process ensured fair treatment and sought to minimize displacement-related risks as a main goal [Source 1].\n\n## Timeline\n- **1928, 1932**: San Felipe and San Ciprián hurricanes accelerate rural-to-urban migration; people settle along the tidal channel wetlands [Source 1].\n- **1930s–1950s**: Informal settlement of wetlands; rapid environmental degradation; mangrove swamps progressively filled in [Source 1].\n- **1940s–1960s**: Operation Bootstrap industrialization; urban renewal programs displace thousands, some settling near the Caño [Source 1].\n- **2001**: City of San Juan partners with U.S. Army Corps of Engineers on canal dredging proposal; residents fear displacement and begin to organize [Source 1].\n- **2002**: Residents begin organized coalition meetings; San Juan designates the eight communities as a Special Planning District [Source 1].\n- **2004**: Eight communities form the G8 Coalition; Puerto Rico Law 489 enacted, creating ENLACE as a public corporation; CLT established; Caño Martín Peña Comprehensive Infrastructure Master Plan (CIMP) developed [Source 1, Source 2].\n- **2016**: UN World Habitat Award received by the CLT/G8/ENLACE model [Source 1].\n- **2017**: Hurricanes Irma and María devastate Puerto Rico; 1,200 homes lost roofs and 100 homes destroyed in CMP; ENLACE identified opportunity to accelerate project but encountered CDBG-DR action plan constraints that, by not allowing new infrastructure or construction/rehabilitation of housing within the floodplain, promoted displacement rather than community cohesion [Source 1, Source 2]; approximately 600 families had already relocated by this point [Source 2].\n- **2022**: $163 million assigned to the U.S. Army Corps of Engineers for dredging; $46 million in CDBG-MIT provided by HUD for relocation [Source 1].\n- **2023**: Dredging work begins; updated CIMP issued (OLIN, 2023) [Source 1].\n- **Current stage**: **Ongoing** — dredging and relocation underway; CLT continues to operate; future land use planning for vacated corridor areas in progress.\n\n## Key Outcomes\n- **Achieved**: Collective tenure regularized for approximately 2,000 low-income families through the CLT; flooding incidence reduced through initial infrastructure works (prior to full dredging, flooding historically affected ~70% of the area); approximately 600 families successfully relocated with choice as of 2017; community governance and social cohesion preserved; international recognition (2016 UN World Habitat Award); legal institutionalization of community governance through Law 489 [Sources 1, 2].\n- **In progress**: Full-scale dredging and canal restoration; relocation of remaining families from right-of-way; ongoing CLT governance [Source 1].\n- **Unresolved/contested**: Tensions between CDBG-DR action plan restrictions (which, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than on-site community cohesion) and ENLACE's goals [Source 2]; population decline from approximately 15,693–18,103 to approximately 10,997, linked by community leaders to governmental discrimination and stigmatization, though causal attribution is uncertain [Source 1].\n\n---\n\n## Narrative Summary\n\nEl Caño Martín Peña is a 3.75-mile tidal channel running through the heart of San Juan, Puerto Rico, surrounded by eight densely populated informal neighborhoods that developed from the 1930s onward as rural migrants — many displaced by the 1928 and 1932 hurricanes and by government urban renewal programs — settled in the wetlands. Over decades, the canal became severely degraded: mangrove swamps approximately 250 feet wide were progressively filled in, narrowing the channel dramatically, and communities were chronically flooded (historically affecting about 70% of the area), lacked basic sewage and water infrastructure, and faced persistent threats of top-down displacement given the high real estate value of their land. With median household incomes of $17,817 and over 61% of residents living below the poverty line, these communities were among Puerto Rico's most economically marginalized [Source 1].\n\nIn 2001, when the City of San Juan and the U.S. Army Corps of Engineers announced plans to dredge the canal, residents feared forced removal. Rather than accepting displacement, the eight communities organized. By 2004, they had formed the G8 Coalition and successfully lobbied the Puerto Rican government to pass Law 489, creating Proyecto ENLACE — a public corporation tasked with developing and implementing a comprehensive community-driven plan for the canal and surrounding neighborhoods. Simultaneously, the communities established the Fideicomiso de la Tierra (Community Land Trust), a private non-profit organization that secured collective tenure for approximately 2,000 families and protected residents from speculative displacement [Sources 1, 2]. The CLT and ENLACE together produced a Comprehensive Development and Land Use Plan, developed through a sustained participatory process involving over 700 community engagement activities, that integrated dredging, flood mitigation, housing stabilization, social services, and managed retreat into a single cohesive vision [Source 2].\n\nThe managed retreat component requires approximately 1,000 families to relocate to allow for canal widening and dredging. This is choice-based within a constrained setting: households along the right-of-way are required to move, but they select housing within the CLT footprint (preserving community membership and social networks) or accept a relocation voucher [Source 1]. As relocated resident Vicky Castro described: \"They give you the option to choose, they don't choose for you, you make your own decisions\" [Source 2]. The relocation process is explained clearly and repeatedly at community meetings; ENLACE provides mental health services and comprehensive support throughout [Source 2]. One constraint prompting relocation is that the Government of Puerto Rico does not allow elevation of homes in the floodplain [Source 1]. By 2017, approximately 600 families had already relocated [Source 2]. Funded by $46 million in CDBG-MIT from HUD and $163 million assigned to the Army Corps for dredging, construction began in 2023. The case has received international recognition, including the 2016 UN World Habitat Award, as an example of participatory, equity-driven managed retreat that achieves resilience goals without reproducing the top-down displacement historically imposed on Puerto Rico's informal communities. However, post-Hurricane María complications — including CDBG-DR action plan restrictions that, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than community cohesion — have created tensions between federal recovery frameworks and ENLACE's vision [Source 2].\n\n---"
|
| 91 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
{
|
| 93 |
"case_id": "CS12",
|
| 94 |
"chunk_index": 999,
|
|
|
|
| 89 |
"source": "Pipeline Ver2 — San Juan Managed Retreat",
|
| 90 |
"text": "Case study: San Juan Managed Retreat (San Juan, Puerto Rico, USA)\nSection: Case Study Overview (Summary)\n\n## Case Name and Location\n**El Caño Martín Peña (CMP) — Eight Communities Aledañas al Caño Martín Peña Special Planning District**\nSan Juan, Puerto Rico — a U.S. territory with a distinct colonial governance context (classified as Global North by project codebook definition, though Puerto Rico's territorial and colonial status introduces important caveats)\n\n## Hazard(s)\nPrimary hazard: **riverine/tidal flooding**, driven by the progressive narrowing and blockage of a 3.75-mile tidal channel (El Caño Martín Peña) connecting the San José Lagoon to the San Juan Bay Estuary. The channel was severely obstructed by decades of informal landfill (debris, trash, wetland conversion), leading to chronic flooding that historically affected approximately 70% of the area [Source 1]. Secondary hazard: **hurricane events**, including the 1928 San Felipe and 1932 San Ciprián hurricanes (which originally accelerated migration into the area) and Hurricanes Irma and María (September 2017), after which 1,200 homes lost their roofs and 100 homes were destroyed within the eight communities [Source 1].\n\n## Type of Retreat\nA **community-driven, participatory managed retreat** embedded within a broader ecosystem restoration and community redevelopment initiative. The retreat component requires approximately 1,000 families who live along the canal right-of-way to relocate to allow for channel widening and dredging. Households are offered a choice between relocating within the eight communities (within the CLT footprint) or accepting a housing relocation voucher to move elsewhere. It is important to note that relocation is not purely voluntary in the conventional sense: households along the right-of-way *must* move for dredging to proceed; what is choice-based is the *destination and form* of their new housing, not whether relocation occurs at all [Source 1, Source 2]. The retreat is explicitly framed as inseparable from environmental restoration, flood mitigation, and long-term community resilience. It is not a standalone buyout program.\n\n## Scale\n- **Population**: Approximately 10,997 current residents; the population had previously been estimated at approximately 15,693–18,103. Sources identify a range of possible contributing factors to this decline, including governmental discrimination and stigmatization of the communities, though causation is not established [Source 1].\n- **Housing units**: 6,946 total units in the eight communities; approximately 1,000 families required to relocate for the dredging right-of-way; approximately 600 families had already moved into safe housing as of 2017 [Source 1, Source 2].\n- **Land/tenure**: The Community Land Trust (*Fideicomiso de la Tierra del Caño Martín Peña*) holds collective tenure on behalf of approximately 2,000 families (around 16.51% of the area's households), who share collective clear title [Source 1]. Source 2 states that \"the community collectively owns 200 acres (80 ha) of land\" [Source 2]; this figure is cited from Source 2 (Yarina et al. 2020) and is not independently confirmed in Source 1.\n- **Geographic scope**: Eight neighborhoods — Barrio Obrero Oeste, Barrio Obrero, San Ciprián, Parada 27, Buena Vista Santurce, Las Monjas, Buena Vista Hato Rey, and Israel–Bitumul — forming the Martín Peña Special Planning District [Source 1].\n\n## Key Actors\n- **G8 Coalition** (*Grupo de las Ocho Comunidades Aledañas al Caño Martín Peña*): Community coalition of the eight neighborhoods; central community governance body. Formed in 2004 [Source 1].\n- **Proyecto ENLACE del Caño Martín Peña**: Public corporation established by Puerto Rico Law 489 (2004) to manage environmental restoration and community development initiatives; serves as institutional intermediary between community and government [Source 1, Source 2].\n- **Fideicomiso de la Tierra del Caño Martín Peña (CLT)**: Community Land Trust; a private, non-profit organization created to manage the land, secure collective tenure, and prevent residents from being priced out [Source 2]. Provides housing opportunities for relocating families and protects residents from speculative displacement [Source 1, Source 2].\n- **Government of Puerto Rico / Puerto Rico Department of Housing**: State-level authority administering CDBG-DR and CDBG-MIT funds; responsible for Program R3 (Repair, Reconstruction, or Relocation) [Source 3].\n- **U.S. Army Corps of Engineers**: Dredging project partner; government of San Juan partnered with the Corps in 2001 to plan canal restoration; $163 million assigned to the Corps for dredging in 2022 [Source 1].\n- **U.S. Department of Housing and Urban Development (HUD)**: Federal funder; provided $46 million in CDBG-MIT funds for relocation [Source 1].\n- **U.S. Environmental Protection Agency (EPA)**: Urban Waters Federal Partnership participant [Source 1].\n- **Residents / community members**: Primary decision-makers and participants throughout all phases; the community-driven process ensured fair treatment and sought to minimize displacement-related risks as a main goal [Source 1].\n\n## Timeline\n- **1928, 1932**: San Felipe and San Ciprián hurricanes accelerate rural-to-urban migration; people settle along the tidal channel wetlands [Source 1].\n- **1930s–1950s**: Informal settlement of wetlands; rapid environmental degradation; mangrove swamps progressively filled in [Source 1].\n- **1940s–1960s**: Operation Bootstrap industrialization; urban renewal programs displace thousands, some settling near the Caño [Source 1].\n- **2001**: City of San Juan partners with U.S. Army Corps of Engineers on canal dredging proposal; residents fear displacement and begin to organize [Source 1].\n- **2002**: Residents begin organized coalition meetings; San Juan designates the eight communities as a Special Planning District [Source 1].\n- **2004**: Eight communities form the G8 Coalition; Puerto Rico Law 489 enacted, creating ENLACE as a public corporation; CLT established; Caño Martín Peña Comprehensive Infrastructure Master Plan (CIMP) developed [Source 1, Source 2].\n- **2016**: UN World Habitat Award received by the CLT/G8/ENLACE model [Source 1].\n- **2017**: Hurricanes Irma and María devastate Puerto Rico; 1,200 homes lost roofs and 100 homes destroyed in CMP; ENLACE identified opportunity to accelerate project but encountered CDBG-DR action plan constraints that, by not allowing new infrastructure or construction/rehabilitation of housing within the floodplain, promoted displacement rather than community cohesion [Source 1, Source 2]; approximately 600 families had already relocated by this point [Source 2].\n- **2022**: $163 million assigned to the U.S. Army Corps of Engineers for dredging; $46 million in CDBG-MIT provided by HUD for relocation [Source 1].\n- **2023**: Dredging work begins; updated CIMP issued (OLIN, 2023) [Source 1].\n- **Current stage**: **Ongoing** — dredging and relocation underway; CLT continues to operate; future land use planning for vacated corridor areas in progress.\n\n## Key Outcomes\n- **Achieved**: Collective tenure regularized for approximately 2,000 low-income families through the CLT; flooding incidence reduced through initial infrastructure works (prior to full dredging, flooding historically affected ~70% of the area); approximately 600 families successfully relocated with choice as of 2017; community governance and social cohesion preserved; international recognition (2016 UN World Habitat Award); legal institutionalization of community governance through Law 489 [Sources 1, 2].\n- **In progress**: Full-scale dredging and canal restoration; relocation of remaining families from right-of-way; ongoing CLT governance [Source 1].\n- **Unresolved/contested**: Tensions between CDBG-DR action plan restrictions (which, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than on-site community cohesion) and ENLACE's goals [Source 2]; population decline from approximately 15,693–18,103 to approximately 10,997, linked by community leaders to governmental discrimination and stigmatization, though causal attribution is uncertain [Source 1].\n\n---\n\n## Narrative Summary\n\nEl Caño Martín Peña is a 3.75-mile tidal channel running through the heart of San Juan, Puerto Rico, surrounded by eight densely populated informal neighborhoods that developed from the 1930s onward as rural migrants — many displaced by the 1928 and 1932 hurricanes and by government urban renewal programs — settled in the wetlands. Over decades, the canal became severely degraded: mangrove swamps approximately 250 feet wide were progressively filled in, narrowing the channel dramatically, and communities were chronically flooded (historically affecting about 70% of the area), lacked basic sewage and water infrastructure, and faced persistent threats of top-down displacement given the high real estate value of their land. With median household incomes of $17,817 and over 61% of residents living below the poverty line, these communities were among Puerto Rico's most economically marginalized [Source 1].\n\nIn 2001, when the City of San Juan and the U.S. Army Corps of Engineers announced plans to dredge the canal, residents feared forced removal. Rather than accepting displacement, the eight communities organized. By 2004, they had formed the G8 Coalition and successfully lobbied the Puerto Rican government to pass Law 489, creating Proyecto ENLACE — a public corporation tasked with developing and implementing a comprehensive community-driven plan for the canal and surrounding neighborhoods. Simultaneously, the communities established the Fideicomiso de la Tierra (Community Land Trust), a private non-profit organization that secured collective tenure for approximately 2,000 families and protected residents from speculative displacement [Sources 1, 2]. The CLT and ENLACE together produced a Comprehensive Development and Land Use Plan, developed through a sustained participatory process involving over 700 community engagement activities, that integrated dredging, flood mitigation, housing stabilization, social services, and managed retreat into a single cohesive vision [Source 2].\n\nThe managed retreat component requires approximately 1,000 families to relocate to allow for canal widening and dredging. This is choice-based within a constrained setting: households along the right-of-way are required to move, but they select housing within the CLT footprint (preserving community membership and social networks) or accept a relocation voucher [Source 1]. As relocated resident Vicky Castro described: \"They give you the option to choose, they don't choose for you, you make your own decisions\" [Source 2]. The relocation process is explained clearly and repeatedly at community meetings; ENLACE provides mental health services and comprehensive support throughout [Source 2]. One constraint prompting relocation is that the Government of Puerto Rico does not allow elevation of homes in the floodplain [Source 1]. By 2017, approximately 600 families had already relocated [Source 2]. Funded by $46 million in CDBG-MIT from HUD and $163 million assigned to the Army Corps for dredging, construction began in 2023. The case has received international recognition, including the 2016 UN World Habitat Award, as an example of participatory, equity-driven managed retreat that achieves resilience goals without reproducing the top-down displacement historically imposed on Puerto Rico's informal communities. However, post-Hurricane María complications — including CDBG-DR action plan restrictions that, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than community cohesion — have created tensions between federal recovery frameworks and ENLACE's vision [Source 2].\n\n---"
|
| 91 |
},
|
| 92 |
+
{
|
| 93 |
+
"case_id": "CS11",
|
| 94 |
+
"chunk_index": 999,
|
| 95 |
+
"section": "overview_summary",
|
| 96 |
+
"location": "Terrebonne Parish, Louisiana",
|
| 97 |
+
"country": "USA",
|
| 98 |
+
"source": "Pipeline Ver2 — Terrebonne Parish Resettlement",
|
| 99 |
+
"text": "Case study: Terrebonne Parish Resettlement (Terrebonne Parish, Louisiana, USA)\nSection: Case Study Overview (Summary)\n\n## Case Identification\n\n**Case study title:** LA SAFE Terrebonne Parish Adaptation Strategy and the Buyout/Optional Home Owner Assistance for Permanent Resident Households program (with reference to the related, but administratively separate, Isle de Jean Charles resettlement), Terrebonne Parish, Louisiana, United States.\n\n**Global North or South:** North. Terrebonne Parish is located in the United States, a Global North country, consistent with the codebook's definition of \"North\" as broadly describing countries in Europe, North America, and the Oceania nations of Australia and New Zealand [2].\n\n---\n\n## General Information\n\n### Is there any evidence of a proactive or reactive trigger in the MR decision process? (Reactive/Proactive)\n\n**Answer:** Both Reactive and Proactive elements are supported, with the overall LA SAFE program best characterized as Proactive (anticipatory), layered on top of a long history of Reactive, incremental household-level adaptation.\n\n**Narrative:** The LA SAFE (Louisiana's Strategic Adaptations for Future Environments) planning process that produced the Terrebonne Parish Adaptation Strategy (2019) was directly enabled by funding from the U.S. Department of Housing and Urban Development's National Disaster Resilience Competition (NDRC) and the Rockefeller Foundation, which became available in response to Hurricane Isaac (2012) [1]. In this sense, the program's origin is reactive — it was a \"Qualifying Disaster\" (Hurricane Isaac) that created the funding opportunity [1]. However, once funded, the LA SAFE process was explicitly framed as anticipatory: it sought to plan for \"future environments\" and used 50-year flood-risk projections (CPRA's 2017 Coastal Master Plan modeling, Medium Environmental Scenario, 2067) to define high-, moderate-, and low-risk zones and to develop strategies before the projected risk materializes [1]. The Terrebonne Adaptation Strategy explicitly states that LA SAFE \"complements\" reactive structural efforts (the Morganza to the Gulf levee system) with \"community-driven, smaller-scale adaptation strategies\" oriented toward the future [1].\n\nAt the household level, however, Nelson et al. [4] document a long pattern of reactive adaptation: residents in lower Terrebonne Parish communities (Chauvin, Dulac, Isle de Jean Charles, Montegut, Theriot) have historically responded to specific flood events by elevating their homes after the fact (e.g., \"Resident 28\" raised their home after Hurricane Juan in 1985, and again experienced flooding after Hurricane Lili in 2002, eventually relocating through the Isle de Jean Charles resettlement program) [4]. The Isle de Jean Charles resettlement itself, funded through a $48.3 million CDBG-DR award in 2016, was also tied to a \"Qualifying Disaster\" / NDRC funding cycle, and followed decades of land loss and repeated storm damage (including Tropical Storm Lili in 2002, documented by Burley et al. [6]) — making it reactive in its funding trigger but framed by the State as a proactive, anticipatory resettlement given the island's documented loss of more than 98% of its land area over six decades [4].\n\n**Category rationale:** Both categories are selected because the evidence shows a mixed picture: program funding triggers are reactive (tied to disaster declarations and post-disaster federal competitions), while the planning content and stated intent of the LA SAFE Adaptation Strategy and the Isle de Jean Charles resettlement are explicitly proactive/anticipatory, using future risk projections (2067 horizon) rather than only addressing damage already incurred [1], [4].\n\n**Uncertainty/Alternative interpretations:** An alternative interpretation could weight the funding origin more heavily and code the case primarily Reactive, since post-Isaac NDRC/CDBG-DR funding was the proximate trigger that made the LA SAFE process possible [1]. However, while post-Isaac funding was a critical trigger, the LA SAFE program encompasses both proactive planning and responses to past disaster impacts: it is built on the explicit forward-looking framing and the 50-year (2067) risk-modeling basis for zoning described above, and states a goal of acting \"before major impacts occur\" for most of the parish — only approximately 7-12 households are currently being relocated [1], [3], [5], none of which have yet experienced the catastrophic land loss seen at Isle de Jean Charles [4]. The working interpretation therefore favors Proactive as the dominant trigger type for the decision process itself, with Reactive noted as the funding-origin trigger, rather than treating the case as driven solely by reactive, post-disaster motivations.\n\n---\n\n### Date, Number of households affected, Number of population affected, Total funding\n\n**Answer:** LA SAFE planning process: 2016-2019 (Adaptation Strategy published April 2019). Buyout/Optional Home Owner Assistance for Permanent Resident Households: approximately 7-12 households, reflecting inconsistencies in source reporting — the LA SAFE Adaptation Strategy and Summary documents state approximately seven households [1], [3], while Birch et al. report approximately 12 for what appears to be the same or a closely related program [5]. Isle de Jean Charles resettlement (related but distinct program): funded 2016, ongoing through the 2020s. Total funding: the only total funding figure documented in the reviewed sources is the $48.3 million CDBG-DR award, which is specifically attributed to the Isle de Jean Charles resettlement [4]; no total funding figure distinct to the Terrebonne Buyout/Optional Home Owner Assistance program was found.\n\n**Narrative:** The LA SAFE program for Terrebonne Parish ran its community engagement process across five rounds of meetings between roughly 2016 and 2018, engaging 371 residents and community leaders, with the final Adaptation Strategy document published in April 2019 [1]. The program was funded through HUD's National Disaster Resilience Competition and the Rockefeller Fou"
|
| 100 |
+
},
|
| 101 |
{
|
| 102 |
"case_id": "CS12",
|
| 103 |
"chunk_index": 999,
|
data/extracted/precomputed_embeddings.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
docs/extraction_schema.md
CHANGED
|
@@ -1,22 +1,79 @@
|
|
| 1 |
-
# Extraction
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Current Extraction and Indexing Format
|
| 2 |
+
|
| 3 |
+
The original project sketch used a compact structured extraction schema for a
|
| 4 |
+
small pilot set of managed-retreat cases. The current RAG knowledge base uses
|
| 5 |
+
long-form case-study memo sections generated by the LLM extraction/audit
|
| 6 |
+
pipeline.
|
| 7 |
+
|
| 8 |
+
## Source Lineage
|
| 9 |
+
|
| 10 |
+
Case-study candidates come from the Ajibade et al. spreadsheet:
|
| 11 |
+
|
| 12 |
+
`https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`
|
| 13 |
+
|
| 14 |
+
Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and
|
| 15 |
+
pipeline outputs. A case becomes queryable only after it has:
|
| 16 |
+
|
| 17 |
+
1. A canonical `data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md`.
|
| 18 |
+
2. An extracted `data/extracted/CSxx.json`.
|
| 19 |
+
3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`.
|
| 20 |
+
|
| 21 |
+
## Extracted Chunk Shape
|
| 22 |
+
|
| 23 |
+
Each `data/extracted/CSxx.json` file is a list of section chunks:
|
| 24 |
+
|
| 25 |
+
```json
|
| 26 |
+
{
|
| 27 |
+
"case_id": "CS24",
|
| 28 |
+
"chunk_index": 0,
|
| 29 |
+
"section": "overview",
|
| 30 |
+
"location": "Queens, New York",
|
| 31 |
+
"country": "United States",
|
| 32 |
+
"source": "Pipeline Ver2 — Queens",
|
| 33 |
+
"text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
|
| 34 |
+
}
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
Field meanings:
|
| 38 |
+
|
| 39 |
+
| Field | Meaning |
|
| 40 |
+
|-------|---------|
|
| 41 |
+
| `case_id` | Case identifier such as `CS24`. |
|
| 42 |
+
| `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. |
|
| 43 |
+
| `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. |
|
| 44 |
+
| `location` | Location copied from `case_meta.json` when available. |
|
| 45 |
+
| `country` | Country copied from `case_meta.json` when available. |
|
| 46 |
+
| `source` | Human-readable source label for retrieved evidence. |
|
| 47 |
+
| `text` | Self-contained chunk text sent to the retriever and LLM prompt. |
|
| 48 |
+
|
| 49 |
+
`data/extracted/case_summaries.json` contains one `overview_summary` chunk per
|
| 50 |
+
case when a usable overview section can be extracted. These chunks support
|
| 51 |
+
overview and comparison queries.
|
| 52 |
+
|
| 53 |
+
## Refresh Commands
|
| 54 |
+
|
| 55 |
+
Run these after adding or changing canonical Ver2 outputs:
|
| 56 |
+
|
| 57 |
+
```bash
|
| 58 |
+
python3 ingest/ingest_pipeline_outputs.py
|
| 59 |
+
python3 ingest/create_summary_chunks.py
|
| 60 |
+
python3 ingest/embed_and_index.py
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
`embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It
|
| 64 |
+
checks chunk IDs, text, metadata, and a content hash so changed Ver2 content
|
| 65 |
+
triggers a rebuild.
|
| 66 |
+
|
| 67 |
+
## Current State Checked July 14, 2026
|
| 68 |
+
|
| 69 |
+
- Canonical Ver2 cases: 34.
|
| 70 |
+
- Extracted case JSON files: 34.
|
| 71 |
+
- Embedded cases: 34.
|
| 72 |
+
- Embedded chunks: 231.
|
| 73 |
+
- Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`.
|
| 74 |
+
- `CS32` is present in the spreadsheet/local metadata but does not currently
|
| 75 |
+
have a canonical Ver2 output or extracted JSON.
|
| 76 |
+
- `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the
|
| 77 |
+
opening `Case Identification` / `General Information` sections as a fallback
|
| 78 |
+
because its Ver2 document does not have a literal `Case Study Overview`
|
| 79 |
+
heading.
|
ingest/create_summary_chunks.py
CHANGED
|
@@ -34,6 +34,9 @@ _PREAMBLE_RE = re.compile(
|
|
| 34 |
re.IGNORECASE | re.DOTALL,
|
| 35 |
)
|
| 36 |
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
def _find_case_folder(case_id: str) -> Path | None:
|
| 39 |
for p in DATA_RAW.iterdir():
|
|
@@ -61,6 +64,28 @@ def _all_case_ids() -> list[str]:
|
|
| 61 |
return sorted(set(ids), key=lambda x: int(x[2:]))
|
| 62 |
|
| 63 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
def build_summary_chunks() -> list[dict]:
|
| 65 |
chunks = []
|
| 66 |
for case_id in _all_case_ids():
|
|
@@ -79,6 +104,8 @@ def build_summary_chunks() -> list[dict]:
|
|
| 79 |
# Fallback: grab preamble before Q&A questions
|
| 80 |
m2 = _PREAMBLE_RE.match(text)
|
| 81 |
overview_text = m2.group(1).strip() if m2 else ""
|
|
|
|
|
|
|
| 82 |
|
| 83 |
if not overview_text:
|
| 84 |
print(f" [{case_id}] No CASE STUDY OVERVIEW section found — skipping.")
|
|
|
|
| 34 |
re.IGNORECASE | re.DOTALL,
|
| 35 |
)
|
| 36 |
|
| 37 |
+
_H2_RE = re.compile(r"^##\s+(.+)$", re.MULTILINE)
|
| 38 |
+
_FALLBACK_SECTION_LIMIT = 6000
|
| 39 |
+
|
| 40 |
|
| 41 |
def _find_case_folder(case_id: str) -> Path | None:
|
| 42 |
for p in DATA_RAW.iterdir():
|
|
|
|
| 64 |
return sorted(set(ids), key=lambda x: int(x[2:]))
|
| 65 |
|
| 66 |
|
| 67 |
+
def _fallback_opening_sections(text: str) -> str:
|
| 68 |
+
"""Use opening memo sections when a Ver2 lacks a Case Study Overview heading."""
|
| 69 |
+
matches = list(_H2_RE.finditer(text))
|
| 70 |
+
if not matches:
|
| 71 |
+
return ""
|
| 72 |
+
|
| 73 |
+
pieces = []
|
| 74 |
+
for i, match in enumerate(matches):
|
| 75 |
+
heading = match.group(1).strip()
|
| 76 |
+
if not re.search(r"case identification|general information", heading, re.I):
|
| 77 |
+
continue
|
| 78 |
+
start = match.end()
|
| 79 |
+
end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
|
| 80 |
+
body = text[start:end].strip()
|
| 81 |
+
if body:
|
| 82 |
+
pieces.append(f"## {heading}\n\n{body}")
|
| 83 |
+
if len("\n\n".join(pieces)) >= _FALLBACK_SECTION_LIMIT:
|
| 84 |
+
break
|
| 85 |
+
|
| 86 |
+
return "\n\n".join(pieces)[:_FALLBACK_SECTION_LIMIT].strip()
|
| 87 |
+
|
| 88 |
+
|
| 89 |
def build_summary_chunks() -> list[dict]:
|
| 90 |
chunks = []
|
| 91 |
for case_id in _all_case_ids():
|
|
|
|
| 104 |
# Fallback: grab preamble before Q&A questions
|
| 105 |
m2 = _PREAMBLE_RE.match(text)
|
| 106 |
overview_text = m2.group(1).strip() if m2 else ""
|
| 107 |
+
if len(overview_text) < 100:
|
| 108 |
+
overview_text = _fallback_opening_sections(text)
|
| 109 |
|
| 110 |
if not overview_text:
|
| 111 |
print(f" [{case_id}] No CASE STUDY OVERVIEW section found — skipping.")
|
ingest/embed_and_index.py
CHANGED
|
@@ -2,12 +2,14 @@
|
|
| 2 |
embed_and_index.py — Precompute and save embeddings for all case study chunks.
|
| 3 |
|
| 4 |
Saves data/extracted/precomputed_embeddings.json:
|
| 5 |
-
{ "ids": [...], "embeddings": [...], "texts": [...], "metadatas": [...]
|
|
|
|
| 6 |
|
| 7 |
Retrieval uses numpy cosine similarity in rag/retriever.py (no ChromaDB —
|
| 8 |
it is broken on Python 3.13 / HuggingFace Spaces).
|
| 9 |
"""
|
| 10 |
import json
|
|
|
|
| 11 |
import sys
|
| 12 |
from pathlib import Path
|
| 13 |
from sentence_transformers import SentenceTransformer
|
|
@@ -30,6 +32,15 @@ def load_all_chunks() -> list[dict]:
|
|
| 30 |
return chunks
|
| 31 |
|
| 32 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
def embed_and_index():
|
| 34 |
chunks = load_all_chunks()
|
| 35 |
if not chunks:
|
|
@@ -39,11 +50,17 @@ def embed_and_index():
|
|
| 39 |
texts = [c.get("text", json.dumps(c)) for c in chunks]
|
| 40 |
ids = [f"{c.get('case_id', 'unknown')}_{c.get('chunk_index', i)}" for i, c in enumerate(chunks)]
|
| 41 |
metadatas = [{k: str(v) for k, v in c.items() if k != "text"} for c in chunks]
|
|
|
|
| 42 |
|
| 43 |
-
# Use precomputed embeddings if IDs
|
| 44 |
if PRECOMPUTED_PATH.exists():
|
| 45 |
pre = json.loads(PRECOMPUTED_PATH.read_text())
|
| 46 |
-
if
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
print(f"Precomputed embeddings up to date ({len(ids)} chunks).")
|
| 48 |
return
|
| 49 |
print("Precomputed embeddings stale — recomputing...")
|
|
@@ -60,6 +77,7 @@ def embed_and_index():
|
|
| 60 |
"embeddings": embeddings,
|
| 61 |
"texts": texts,
|
| 62 |
"metadatas": metadatas,
|
|
|
|
| 63 |
}))
|
| 64 |
print(f"Saved precomputed embeddings for {len(texts)} chunks.")
|
| 65 |
print("\nNext: streamlit run app/app.py")
|
|
|
|
| 2 |
embed_and_index.py — Precompute and save embeddings for all case study chunks.
|
| 3 |
|
| 4 |
Saves data/extracted/precomputed_embeddings.json:
|
| 5 |
+
{ "ids": [...], "embeddings": [...], "texts": [...], "metadatas": [...],
|
| 6 |
+
"content_hash": "..." }
|
| 7 |
|
| 8 |
Retrieval uses numpy cosine similarity in rag/retriever.py (no ChromaDB —
|
| 9 |
it is broken on Python 3.13 / HuggingFace Spaces).
|
| 10 |
"""
|
| 11 |
import json
|
| 12 |
+
import hashlib
|
| 13 |
import sys
|
| 14 |
from pathlib import Path
|
| 15 |
from sentence_transformers import SentenceTransformer
|
|
|
|
| 32 |
return chunks
|
| 33 |
|
| 34 |
|
| 35 |
+
def _content_hash(ids: list[str], texts: list[str], metadatas: list[dict]) -> str:
|
| 36 |
+
payload = json.dumps(
|
| 37 |
+
{"ids": ids, "texts": texts, "metadatas": metadatas},
|
| 38 |
+
sort_keys=True,
|
| 39 |
+
ensure_ascii=False,
|
| 40 |
+
)
|
| 41 |
+
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
| 42 |
+
|
| 43 |
+
|
| 44 |
def embed_and_index():
|
| 45 |
chunks = load_all_chunks()
|
| 46 |
if not chunks:
|
|
|
|
| 50 |
texts = [c.get("text", json.dumps(c)) for c in chunks]
|
| 51 |
ids = [f"{c.get('case_id', 'unknown')}_{c.get('chunk_index', i)}" for i, c in enumerate(chunks)]
|
| 52 |
metadatas = [{k: str(v) for k, v in c.items() if k != "text"} for c in chunks]
|
| 53 |
+
content_hash = _content_hash(ids, texts, metadatas)
|
| 54 |
|
| 55 |
+
# Use precomputed embeddings only if chunk IDs and chunk contents match.
|
| 56 |
if PRECOMPUTED_PATH.exists():
|
| 57 |
pre = json.loads(PRECOMPUTED_PATH.read_text())
|
| 58 |
+
if (
|
| 59 |
+
pre.get("ids") == ids
|
| 60 |
+
and pre.get("texts") == texts
|
| 61 |
+
and pre.get("metadatas") == metadatas
|
| 62 |
+
and pre.get("content_hash") == content_hash
|
| 63 |
+
):
|
| 64 |
print(f"Precomputed embeddings up to date ({len(ids)} chunks).")
|
| 65 |
return
|
| 66 |
print("Precomputed embeddings stale — recomputing...")
|
|
|
|
| 77 |
"embeddings": embeddings,
|
| 78 |
"texts": texts,
|
| 79 |
"metadatas": metadatas,
|
| 80 |
+
"content_hash": content_hash,
|
| 81 |
}))
|
| 82 |
print(f"Saved precomputed embeddings for {len(texts)} chunks.")
|
| 83 |
print("\nNext: streamlit run app/app.py")
|
ingest/ingest_pipeline_outputs.py
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
"""
|
| 2 |
-
ingest_pipeline_outputs.py — Convert selected Ver2.md files into
|
| 3 |
|
| 4 |
Reads every data/raw/CSX/pipeline_output/CSX_Ver2.md (the canonical file written
|
| 5 |
by select_best_run.py), splits it into one chunk per questionnaire section, and
|
|
@@ -14,7 +14,7 @@ Usage:
|
|
| 14 |
python ingest/ingest_pipeline_outputs.py --all --force # overwrite existing JSON
|
| 15 |
|
| 16 |
The script is idempotent: re-running it for a case study overwrites its
|
| 17 |
-
data/extracted/CSX.json, so
|
| 18 |
"""
|
| 19 |
|
| 20 |
from __future__ import annotations
|
|
@@ -29,7 +29,7 @@ DATA_RAW = REPO_ROOT / "data" / "raw"
|
|
| 29 |
EXTRACTED_DIR = REPO_ROOT / "data" / "extracted"
|
| 30 |
|
| 31 |
# ---------------------------------------------------------------------------
|
| 32 |
-
# Section name →
|
| 33 |
# The keys match the field names used in CS1_IDJC.json so all chunks are
|
| 34 |
# queryable with the same metadata filters.
|
| 35 |
# ---------------------------------------------------------------------------
|
|
@@ -187,7 +187,7 @@ def ingest_case(case_id: str, force: bool = False) -> bool:
|
|
| 187 |
|
| 188 |
def main(argv: list[str] | None = None) -> None:
|
| 189 |
parser = argparse.ArgumentParser(
|
| 190 |
-
description="Convert selected Ver2.md files into
|
| 191 |
)
|
| 192 |
group = parser.add_mutually_exclusive_group()
|
| 193 |
group.add_argument("--case-id", nargs="+", metavar="ID",
|
|
|
|
| 1 |
"""
|
| 2 |
+
ingest_pipeline_outputs.py — Convert selected Ver2.md files into RAG JSON chunks.
|
| 3 |
|
| 4 |
Reads every data/raw/CSX/pipeline_output/CSX_Ver2.md (the canonical file written
|
| 5 |
by select_best_run.py), splits it into one chunk per questionnaire section, and
|
|
|
|
| 14 |
python ingest/ingest_pipeline_outputs.py --all --force # overwrite existing JSON
|
| 15 |
|
| 16 |
The script is idempotent: re-running it for a case study overwrites its
|
| 17 |
+
data/extracted/CSX.json, so the embedding store can reflect the latest selected Ver2.
|
| 18 |
"""
|
| 19 |
|
| 20 |
from __future__ import annotations
|
|
|
|
| 29 |
EXTRACTED_DIR = REPO_ROOT / "data" / "extracted"
|
| 30 |
|
| 31 |
# ---------------------------------------------------------------------------
|
| 32 |
+
# Section name → retrieval section key
|
| 33 |
# The keys match the field names used in CS1_IDJC.json so all chunks are
|
| 34 |
# queryable with the same metadata filters.
|
| 35 |
# ---------------------------------------------------------------------------
|
|
|
|
| 187 |
|
| 188 |
def main(argv: list[str] | None = None) -> None:
|
| 189 |
parser = argparse.ArgumentParser(
|
| 190 |
+
description="Convert selected Ver2.md files into RAG JSON chunks."
|
| 191 |
)
|
| 192 |
group = parser.add_mutually_exclusive_group()
|
| 193 |
group.add_argument("--case-id", nargs="+", metavar="ID",
|
rag/pipeline.py
CHANGED
|
@@ -31,21 +31,21 @@ def query_ollama(prompt: str) -> str:
|
|
| 31 |
return response.json()["response"].strip()
|
| 32 |
|
| 33 |
|
| 34 |
-
def query_claude(prompt: str) -> str:
|
| 35 |
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
|
| 36 |
message = client.messages.create(
|
| 37 |
model="claude-sonnet-4-6",
|
| 38 |
-
max_tokens=
|
| 39 |
messages=[{"role": "user", "content": prompt}],
|
| 40 |
)
|
| 41 |
return message.content[0].text.strip()
|
| 42 |
|
| 43 |
|
| 44 |
-
def query_openai(prompt: str) -> str:
|
| 45 |
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
|
| 46 |
response = client.chat.completions.create(
|
| 47 |
model="gpt-4o-mini",
|
| 48 |
-
max_completion_tokens=
|
| 49 |
messages=[{"role": "user", "content": prompt}],
|
| 50 |
)
|
| 51 |
return response.choices[0].message.content.strip()
|
|
@@ -86,9 +86,9 @@ def run_pipeline(
|
|
| 86 |
prompt = build_prompt(query, chunks, questionnaire_answers)
|
| 87 |
|
| 88 |
if model == "claude":
|
| 89 |
-
answer = query_claude(prompt)
|
| 90 |
elif model in ("openai", "gpt"):
|
| 91 |
-
answer = query_openai(prompt)
|
| 92 |
else:
|
| 93 |
answer = query_ollama(prompt)
|
| 94 |
|
|
|
|
| 31 |
return response.json()["response"].strip()
|
| 32 |
|
| 33 |
|
| 34 |
+
def query_claude(prompt: str, max_tokens: int = 4096) -> str:
|
| 35 |
client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
|
| 36 |
message = client.messages.create(
|
| 37 |
model="claude-sonnet-4-6",
|
| 38 |
+
max_tokens=max_tokens,
|
| 39 |
messages=[{"role": "user", "content": prompt}],
|
| 40 |
)
|
| 41 |
return message.content[0].text.strip()
|
| 42 |
|
| 43 |
|
| 44 |
+
def query_openai(prompt: str, max_tokens: int = 2048) -> str:
|
| 45 |
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
|
| 46 |
response = client.chat.completions.create(
|
| 47 |
model="gpt-4o-mini",
|
| 48 |
+
max_completion_tokens=max_tokens,
|
| 49 |
messages=[{"role": "user", "content": prompt}],
|
| 50 |
)
|
| 51 |
return response.choices[0].message.content.strip()
|
|
|
|
| 86 |
prompt = build_prompt(query, chunks, questionnaire_answers)
|
| 87 |
|
| 88 |
if model == "claude":
|
| 89 |
+
answer = query_claude(prompt, max_tokens=8192 if _OVERVIEW_RE.search(query) else 4096)
|
| 90 |
elif model in ("openai", "gpt"):
|
| 91 |
+
answer = query_openai(prompt, max_tokens=4096 if _OVERVIEW_RE.search(query) else 2048)
|
| 92 |
else:
|
| 93 |
answer = query_ollama(prompt)
|
| 94 |
|
rag/prompt_builder.py
CHANGED
|
@@ -1,30 +1,70 @@
|
|
| 1 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
|
| 4 |
def build_prompt(query: str, retrieved_chunks: list[dict], questionnaire_answers: dict | None = None) -> str:
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
for
|
| 8 |
)
|
| 9 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
answers_section = ""
|
| 11 |
if questionnaire_answers:
|
| 12 |
answers_section = "\n\nUser context from questionnaire:\n" + "\n".join(
|
| 13 |
f"- {k}: {v}" for k, v in questionnaire_answers.items()
|
| 14 |
)
|
| 15 |
|
| 16 |
-
is_overview = any(
|
| 17 |
-
kw in query.lower()
|
| 18 |
-
for kw in ("summar", "overview", "all case", "every case", "list", "compare", "across")
|
| 19 |
-
)
|
| 20 |
-
|
| 21 |
if is_overview:
|
|
|
|
| 22 |
instruction = (
|
| 23 |
"You are a research assistant. The user wants a summary of ALL case studies provided. "
|
| 24 |
-
"
|
| 25 |
-
"
|
| 26 |
-
"
|
| 27 |
-
"
|
|
|
|
| 28 |
)
|
| 29 |
else:
|
| 30 |
instruction = (
|
|
|
|
| 1 |
+
import re
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
_OVERVIEW_CONTEXT_CHARS = 1800
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
def _extract_label(text: str, label: str) -> str:
|
| 8 |
+
pattern = re.compile(
|
| 9 |
+
rf"\*\*{re.escape(label)}:\*\*\s*(.*?)(?=\n\n\*\*[^*\n]+:\*\*|\n---|\n#{1,6}\s|\Z)",
|
| 10 |
+
re.IGNORECASE | re.DOTALL,
|
| 11 |
+
)
|
| 12 |
+
match = pattern.search(text)
|
| 13 |
+
if not match:
|
| 14 |
+
return ""
|
| 15 |
+
return re.sub(r"\s+", " ", match.group(1)).strip()
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def _compact_overview_text(text: str) -> str:
|
| 19 |
+
"""Keep overview prompts small enough that all cases fit in the answer."""
|
| 20 |
+
fields = []
|
| 21 |
+
for label in ("Case Name and Location", "Hazard(s)", "Type of Retreat", "Key Outcomes"):
|
| 22 |
+
value = _extract_label(text, label)
|
| 23 |
+
if value:
|
| 24 |
+
fields.append(f"{label}: {value}")
|
| 25 |
+
|
| 26 |
+
compact = "\n".join(fields)
|
| 27 |
+
if not compact:
|
| 28 |
+
compact = re.sub(r"\s+", " ", text).strip()
|
| 29 |
+
|
| 30 |
+
if len(compact) > _OVERVIEW_CONTEXT_CHARS:
|
| 31 |
+
compact = compact[:_OVERVIEW_CONTEXT_CHARS].rsplit(" ", 1)[0] + " ..."
|
| 32 |
+
return compact
|
| 33 |
|
| 34 |
|
| 35 |
def build_prompt(query: str, retrieved_chunks: list[dict], questionnaire_answers: dict | None = None) -> str:
|
| 36 |
+
is_overview = any(
|
| 37 |
+
kw in query.lower()
|
| 38 |
+
for kw in ("summar", "overview", "all case", "every case", "list", "compare", "across")
|
| 39 |
)
|
| 40 |
|
| 41 |
+
if is_overview:
|
| 42 |
+
context_blocks = "\n\n".join(
|
| 43 |
+
f"[Source: {c['metadata'].get('source', 'unknown')} | Case: {c['metadata'].get('case_id', '?')}]\n"
|
| 44 |
+
f"{_compact_overview_text(c['text'])}"
|
| 45 |
+
for c in retrieved_chunks
|
| 46 |
+
)
|
| 47 |
+
else:
|
| 48 |
+
context_blocks = "\n\n".join(
|
| 49 |
+
f"[Source: {c['metadata'].get('source', 'unknown')} | Case: {c['metadata'].get('case_id', '?')}]\n{c['text']}"
|
| 50 |
+
for c in retrieved_chunks
|
| 51 |
+
)
|
| 52 |
+
|
| 53 |
answers_section = ""
|
| 54 |
if questionnaire_answers:
|
| 55 |
answers_section = "\n\nUser context from questionnaire:\n" + "\n".join(
|
| 56 |
f"- {k}: {v}" for k, v in questionnaire_answers.items()
|
| 57 |
)
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
if is_overview:
|
| 60 |
+
case_ids = ", ".join(c["metadata"].get("case_id", "?") for c in retrieved_chunks)
|
| 61 |
instruction = (
|
| 62 |
"You are a research assistant. The user wants a summary of ALL case studies provided. "
|
| 63 |
+
f"The provided cases are: {case_ids}. "
|
| 64 |
+
"Write exactly one numbered bullet for EVERY provided case, in the same order. "
|
| 65 |
+
"Each bullet must start with the bold case ID, then give one concise sentence covering "
|
| 66 |
+
"location, hazard or retreat type, and the key outcome or lesson. "
|
| 67 |
+
"Keep each bullet under 45 words. Do not skip cases. Do not invent missing case IDs."
|
| 68 |
)
|
| 69 |
else:
|
| 70 |
instruction = (
|