Anna Zhou commited on
Commit
318732d
·
1 Parent(s): 51b3964

Fix all-case summaries for Hugging Face Space

Browse files
AGENTS.md ADDED
@@ -0,0 +1,290 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # RFR Co-op: LLM-Assisted Managed Retreat Pipeline
2
+
3
+ > This file is read automatically by Codex. It provides full project context so every session starts with shared understanding.
4
+
5
+ ## What this project is
6
+
7
+ This is a 4-month co-op research project embedded in the **Retreat From Risk (RFR)** project — a NFRF-funded international research initiative (Canada, USA, Indonesia) studying **managed retreat (MR)** as a climate adaptation strategy for flood-prone communities.
8
+
9
+ The co-op goal is to build an **LLM-assisted framework** for extracting and synthesizing insights from MR case study literature, and to deliver a working **RAG pipeline** that helps decision-makers (municipal planners, First Nations leaders) query those insights through a guided questionnaire and persona-aware interface.
10
+
11
+ **Supervisors:** Dr. Costa (systems engineering, UW) and Dr. Doberstein (geography, UW)
12
+ **PhD collaborator:** Ana (has an existing human-coded extraction framework — this project compares LLM vs human extraction)
13
+ **GitHub:** https://github.com/AnnasBeef
14
+ **Linear:** https://linear.app/managed-retreat (team: Managed Retreat, issues prefixed MAN-)
15
+
16
+ ---
17
+
18
+ ## Working in Cowork (read this before running the pipeline here)
19
+
20
+ The Cowork sandbox is **ephemeral** (installed packages don't persist between sessions) and its **egress proxy blocks the LLM APIs**. To avoid re-diagnosing this every time:
21
+
22
+ - **Setup once per session:** `bash cowork_setup.sh` — installs only the minimal deps (`requirements-cowork.txt`), not the heavy RAG stack (`torch`, `sentence-transformers`) in `requirements.txt`.
23
+ - **The live API run does NOT work in Cowork.** `api.anthropic.com` returns a proxy `401`; `api.openai.com` is unreachable. Run the full 4-step pipeline on your **Mac**, not here.
24
+ - **If attempting outbound HTTPS from Python here,** the proxy uses a self-signed cert — prefix with `SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt`.
25
+ - **Works fine offline in Cowork:** PDF rendering (`pdf_utils.text_to_pdf`), source-text extraction (`pdfplumber`), manual Ver1/Ver2 drafting.
26
+
27
+ ### Finding & downloading source papers for a case study
28
+
29
+ 1. First try `python3 -m case_study_pipeline.fetch_papers --case-id CSxx` —
30
+ pulls open-access PDFs automatically (Unpaywall, Semantic Scholar, Sci-Hub).
31
+ Also blocked by the Cowork egress proxy; run on the Mac.
32
+ 2. For anything still paywalled, use UWaterloo's institutional library
33
+ access (EZproxy) via the Codex-in-Chrome browser extension — **this
34
+ requires a human to complete WatIAM login + Duo 2FA, it cannot be
35
+ scripted**. Full step-by-step procedure, including the one-time Chrome
36
+ download-folder setting, is in
37
+ `case_study_pipeline/README.md` under "Fetching paywalled papers via
38
+ UWaterloo EZproxy (Chrome-assisted)". Read that section before attempting
39
+ this — it documents the exact URL pattern (`ezproxy.py`) and the
40
+ inbox-sorting + Drive-upload script (`sort_inbox.py`).
41
+
42
+ ---
43
+
44
+ ## Core research questions
45
+
46
+ 1. Can LLMs reliably extract key information from MR case studies compared to human coders?
47
+ 2. How does question structure (the guided questionnaire) affect LLM output quality?
48
+ 3. What does a useful decision-support tool look like for a community leader considering managed retreat?
49
+
50
+ ---
51
+
52
+ ## What we're building
53
+
54
+ ### Pipeline architecture
55
+
56
+ ```
57
+ MR case studies (PDFs, spreadsheets)
58
+ ↓
59
+ Case-study source catalog (Ajibade et al. spreadsheet + local case_meta.json)
60
+ ↓
61
+ LLM extraction/audit pipeline (Ver1 → Check1 → Ver2 → Check2)
62
+ ↓
63
+ Section-level JSON chunks in data/extracted/
64
+ ↓
65
+ Precomputed embedding store (data/extracted/precomputed_embeddings.json)
66
+ ↓
67
+ NumPy cosine retrieval (semantic search)
68
+ ↓
69
+ Persona layer (decision-maker framing)
70
+ ↓
71
+ Guided questionnaire (shapes the query)
72
+ ↓
73
+ LLM response (grounded, source-cited)
74
+ ↓
75
+ Web interface (hosted on AWS / HuggingFace Spaces)
76
+ ```
77
+
78
+ ### Month 1 deliverables (by ~June 6)
79
+ - [ ] v0 RAG pipeline running locally (MAN-16)
80
+ - [ ] Decision-maker persona defined (MAN-11)
81
+ - [ ] Guided questionnaire v1 (MAN-12)
82
+ - [ ] Persona + questionnaire integrated into RAG (MAN-13)
83
+ - [ ] Cloud hosting decision made (MAN-17)
84
+
85
+ ---
86
+
87
+ ## Tech stack
88
+
89
+ | Layer | Tool | Notes |
90
+ |---|---|---|
91
+ | Language | Python 3.11+ | |
92
+ | Retrieval store | `data/extracted/precomputed_embeddings.json` | JSON file with chunk text, metadata, embeddings, and a content hash |
93
+ | Embeddings | sentence-transformers | Free, local (`all-MiniLM-L6-v2`) |
94
+ | Retrieval | NumPy cosine similarity | Runtime replacement for ChromaDB; see `rag/retriever.py` |
95
+ | LLM (local fallback) | Ollama | `rag/pipeline.py` currently calls Ollama model `gemma4` for non-Claude/OpenAI mode |
96
+ | Web interface | Streamlit | `app/app.py` |
97
+ | Hosting | HuggingFace Spaces (dev) → AWS EC2 (prod) | |
98
+ | Project tracking | Linear (MAN- issues) | |
99
+ | Version control | GitHub (AnnasBeef) | Branch names: `MAN-XX-short-description` |
100
+
101
+ ---
102
+
103
+ ## Repo structure
104
+
105
+ ```
106
+ rfr-rag/
107
+ ├── AGENTS.md ← you are here
108
+ ├── data/
109
+ │ ├── raw/ # original PDFs and spreadsheets
110
+ │ └── extracted/ # Ver2-derived JSON chunks + precomputed embeddings
111
+ ├── ingest/
112
+ │ ├── ingest_pipeline_outputs.py
113
+ │ ├── create_summary_chunks.py
114
+ │ └── embed_and_index.py
115
+ ├── rag/
116
+ │ ├── retriever.py
117
+ │ ├── prompt_builder.py
118
+ │ └── pipeline.py
119
+ ├── persona/
120
+ │ └── decision_maker.json # persona config
121
+ ├── questionnaire/
122
+ │ └── questions.json # guided questionnaire structure
123
+ ├── app/
124
+ │ └── app.py # Streamlit/Gradio interface
125
+ ├── case_study_pipeline/ # Ver1/Check1/Ver2/Check2 extraction workflow
126
+ ├── docs/
127
+ │ └── extraction_schema.md
128
+ ├── tests/
129
+ └── requirements.txt
130
+ ```
131
+
132
+ ---
133
+
134
+ ## Current extraction/indexing format
135
+
136
+ The original plan used a compact structured schema. The current RAG index uses
137
+ long-form Ver2 memo sections instead:
138
+
139
+ ```json
140
+ {
141
+ "case_id": "CS24",
142
+ "chunk_index": 0,
143
+ "section": "overview",
144
+ "location": "Queens, New York",
145
+ "country": "United States",
146
+ "source": "Pipeline Ver2 — Queens",
147
+ "text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
148
+ }
149
+ ```
150
+
151
+ The active index is built by:
152
+
153
+ 1. `python3 -m case_study_pipeline.select_best_run --case-id CSxx` if multiple runs exist.
154
+ 2. `python3 ingest/ingest_pipeline_outputs.py --case-id CSxx` to convert canonical `CSxx_Ver2.md` into `data/extracted/CSxx.json`.
155
+ 3. `python3 ingest/create_summary_chunks.py` to refresh one overview chunk per case in `data/extracted/case_summaries.json`.
156
+ 4. `python3 ingest/embed_and_index.py` to refresh `data/extracted/precomputed_embeddings.json`.
157
+
158
+ `embed_and_index.py` checks chunk IDs, text, metadata, and a content hash. If a
159
+ Ver2 memo or JSON chunk changes, rebuild the embedding file before testing the app.
160
+
161
+ ---
162
+
163
+ ## Case studies and source spreadsheet
164
+
165
+ The expanded case-study catalog is based on the Ajibade et al. spreadsheet:
166
+ `https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`
167
+
168
+ As of July 14, 2026, the local RAG knowledge base has canonical Ver2 outputs,
169
+ extracted JSON, and embeddings for 34 cases:
170
+
171
+ `CS1`-`CS31`, `CS33`, `CS34`, and `CS35`.
172
+
173
+ `CS32` exists in the spreadsheet/local metadata as Nono District / Jiru Gamachu
174
+ Resettlement Site, Ethiopia, but does not currently have a canonical Ver2 output
175
+ or extracted JSON in this repo.
176
+
177
+ Current CS1-CS20 local names from `case_meta.json` and canonical Ver2 outputs:
178
+
179
+ | ID | Current local name | Location | Country |
180
+ |----|--------------------|----------|---------|
181
+ | CS1 | Isle de Jean Charles Resettlement | Isle de Jean Charles, Terrebonne Parish, Louisiana | USA |
182
+ | CS2 | Shaanxi Ecological Resettlement | Shaanxi Province | China |
183
+ | CS3 | Oakwood Beach Buyout Program | Oakwood Beach, Staten Island, New York | USA |
184
+ | CS4 | Metro Manila Resettlement (Oplan LIKAS) | Metro Manila | Philippines |
185
+ | CS5 | Matata Managed Retreat | Matata, Bay of Plenty | New Zealand |
186
+ | CS6 | Pointe-Gatineau Home Buyout Program | Pointe-Gatineau, Quebec | Canada |
187
+ | CS7 | Hambantota Post-Tsunami Relocation | Hambantota | Sri Lanka |
188
+ | CS8 | Cedar Rapids Buyout Program | Cedar Rapids, Iowa | USA |
189
+ | CS9 | Grand Forks Floodplain Buyout | Grand Forks, North Dakota | USA |
190
+ | CS10 | San Juan Managed Retreat | San Juan, Puerto Rico | USA |
191
+ | CS11 | Terrebonne Parish Resettlement | Terrebonne Parish, Louisiana | USA |
192
+ | CS12 | Dhye Community Relocation | Dhye, Mustang District | Nepal |
193
+ | CS13 | Soldiers Grove Floodplain Relocation | Soldiers Grove, Wisconsin | USA |
194
+ | CS14 | Simbach am Inn Household Relocation | Simbach am Inn, Bavaria | Germany |
195
+ | CS15 | Tegua Island Planned Relocation | Tegua, Torba Province | Vanuatu |
196
+ | CS16 | Vunidogola Village Relocation | Vunidogola, Koralau Island | Fiji |
197
+ | CS17 | San Martin Planned Relocation | San Martin Region | Peru |
198
+ | CS18 | High River Buyout Program | High River, Alberta | Canada |
199
+ | CS19 | Nusa Hope Village Relocation | Nusa Hope, New Georgia | Solomon Islands |
200
+ | CS20 | East Riding Managed Retreat | East Riding of Yorkshire | UK |
201
+
202
+ ---
203
+
204
+ ## Decision-maker persona (v0)
205
+
206
+ The RAG pipeline frames responses for this user:
207
+
208
+ - **Role:** Municipal planner or First Nations community leader
209
+ - **Knowledge level:** Non-technical; familiar with community planning, not with ML or hydrology
210
+ - **Key concerns:** Equity, community buy-in, funding availability, cultural impacts, legal barriers
211
+ - **Preferred tone:** Plain language, practical, grounded in real examples
212
+ - **What "useful" looks like:** "Here's what worked in a similar community, here's what to watch out for, here are the key decisions you need to make"
213
+
214
+ ---
215
+
216
+ ## Linear issues (current)
217
+
218
+ | ID | Title | Due | Priority |
219
+ |----|-------|-----|----------|
220
+ | MAN-5 | Review human-coded extraction framework with Ana | May 16 | Urgent |
221
+ | MAN-6 | Read key MR case studies and synthesis papers | May 23 | High |
222
+ | MAN-7 | Build structured extraction schema | May 23 | High |
223
+ | MAN-8 | Apply framework to Isle de Jean Charles | May 23 | High |
224
+ | MAN-9 | Write Week 2 reflection memo | May 23 | Medium |
225
+ | MAN-10 | Build basic RAG pipeline | May 16 | High |
226
+ | MAN-11 | Define decision-maker persona | May 16 | High |
227
+ | MAN-12 | Design guided questionnaire | May 29 | High |
228
+ | MAN-13 | Integrate persona + questionnaire into RAG | May 29 | Medium |
229
+ | MAN-14 | Set up GitHub repo (AnnasBeef) | May 12 | Urgent |
230
+ | MAN-15 | Connect GitHub to Linear and Codex | May 12 | Urgent |
231
+ | MAN-16 | Build v0 RAG pipeline locally today | May 11 | Urgent |
232
+ | MAN-17 | Research cloud hosting strategy | May 16 | High |
233
+
234
+ Branch naming convention: `MAN-XX-short-description`
235
+ e.g. `MAN-16-rag-v0-local`
236
+
237
+ ---
238
+
239
+ ## Key files in this repo to read first
240
+
241
+ - `data/extracted/` — Ver2-derived JSON case study chunks and embedding index
242
+ - `case_study_pipeline/README.md` — end-to-end extraction/audit workflow
243
+ - `ingest/ingest_pipeline_outputs.py` — converts canonical Ver2 memos into JSON chunks
244
+ - `ingest/create_summary_chunks.py` — builds overview chunks for summary queries
245
+ - `ingest/embed_and_index.py` — builds `precomputed_embeddings.json`
246
+ - `rag/pipeline.py` — main RAG query loop
247
+ - `persona/decision_maker.json` — persona config used in prompt construction
248
+ - `questionnaire/questions.json` — guided questionnaire structure
249
+
250
+ ---
251
+
252
+ ## Quick start (local dev)
253
+
254
+ ```bash
255
+ # 1. Install dependencies
256
+ pip install -r requirements.txt
257
+
258
+ # 2. Optional: start local LLM fallback
259
+ ollama pull gemma4
260
+ ollama run gemma4
261
+
262
+ # 3. Convert canonical Ver2 memos to JSON chunks when case outputs change
263
+ python ingest/ingest_pipeline_outputs.py
264
+ python ingest/create_summary_chunks.py
265
+
266
+ # 4. Build or refresh the precomputed embedding index
267
+ python ingest/embed_and_index.py
268
+
269
+ # 5. Run the RAG pipeline
270
+ python rag/pipeline.py
271
+
272
+ # 6. Launch the web app
273
+ streamlit run app/app.py
274
+ ```
275
+
276
+ ---
277
+
278
+ ## Key concepts to know
279
+
280
+ **Managed retreat (MR):** The purposeful relocation of people, property, and infrastructure out of areas vulnerable to recurrent climatic hazards. Ranges from voluntary property buyouts to forced community resettlement.
281
+
282
+ **RAG (Retrieval-Augmented Generation):** An LLM pattern where relevant documents are retrieved from a vector store and included in the prompt, so the model's response is grounded in specific source material rather than relying on training data alone.
283
+
284
+ **Decision Support Framework (DSF):** The broader research output of the RFR project — a framework to help communities evaluate MR as a viable adaptation strategy. This pipeline is a technical tool to support the DSF.
285
+
286
+ **OCAP principles:** Ownership, Control, Access, Possession — Indigenous data governance principles that must be respected when working with First Nations case study data.
287
+
288
+ ---
289
+
290
+ *Last updated: May 11, 2026 — generated from Codex.ai project context*
README.md CHANGED
@@ -27,7 +27,9 @@ pip install -r requirements.txt
27
  export ANTHROPIC_API_KEY=your_key_here
28
  export OPENAI_API_KEY=your_key_here
29
 
30
- # 2. Ingest case studies into vector store
 
 
31
  python ingest/embed_and_index.py
32
 
33
  # 3. Run CLI pipeline
@@ -40,13 +42,13 @@ streamlit run app/app.py
40
  ## Ingesting documents
41
 
42
  ```bash
43
- # Parse a PDF case study
44
- python ingest/parse_pdf.py data/raw/isle_de_jean_charles.pdf CS1
45
 
46
- # Parse a spreadsheet (Ana's human-coded extractions)
47
- python ingest/parse_spreadsheet.py data/raw/extraction_framework.xlsx
48
 
49
- # Embed everything into ChromaDB
50
  python ingest/embed_and_index.py
51
  ```
52
 
@@ -55,8 +57,9 @@ python ingest/embed_and_index.py
55
  ```
56
  rfr-rag/
57
  ├── data/raw/ # Original PDFs and spreadsheets (not committed)
58
- ├── data/extracted/ # JSON outputs from parsing
59
- ├── ingest/ # PDF + spreadsheet parsers, embedding script
 
60
  ├── rag/ # Retriever, prompt builder, pipeline
61
  ├── persona/ # Decision-maker persona config
62
  ├── questionnaire/ # Guided questionnaire structure
 
27
  export ANTHROPIC_API_KEY=your_key_here
28
  export OPENAI_API_KEY=your_key_here
29
 
30
+ # 2. Convert canonical Ver2 memos to JSON chunks, then build embeddings
31
+ python ingest/ingest_pipeline_outputs.py
32
+ python ingest/create_summary_chunks.py
33
  python ingest/embed_and_index.py
34
 
35
  # 3. Run CLI pipeline
 
42
  ## Ingesting documents
43
 
44
  ```bash
45
+ # Convert canonical case-study memos into section chunks
46
+ python ingest/ingest_pipeline_outputs.py
47
 
48
+ # Refresh overview-summary chunks for broad comparison queries
49
+ python ingest/create_summary_chunks.py
50
 
51
+ # Build the precomputed embedding store used by rag/retriever.py
52
  python ingest/embed_and_index.py
53
  ```
54
 
 
57
  ```
58
  rfr-rag/
59
  ├── data/raw/ # Original PDFs and spreadsheets (not committed)
60
+ ├── data/extracted/ # Ver2-derived JSON chunks + embedding store
61
+ ├── ingest/ # Ver2 ingestion, summary chunks, embedding script
62
+ ├── case_study_pipeline/# Source extraction/audit workflow
63
  ├── rag/ # Retriever, prompt builder, pipeline
64
  ├── persona/ # Decision-maker persona config
65
  ├── questionnaire/ # Guided questionnaire structure
app/app.py CHANGED
@@ -10,14 +10,26 @@ sys.path.insert(0, str(Path(__file__).parent))
10
  from sheets_logger import log_query
11
 
12
 
13
- def _ensure_chroma_built():
14
  """Ensure precomputed_embeddings.json has texts+metadatas (needed by numpy retriever)."""
15
  precomputed = Path("data/extracted/precomputed_embeddings.json")
16
  needs_rebuild = True
17
  if precomputed.exists():
18
  import json as _json
19
  pre = _json.loads(precomputed.read_text())
20
- if "texts" in pre and "metadatas" in pre:
 
 
 
 
 
 
 
 
 
 
 
 
21
  print(f"Embeddings ready ({len(pre['ids'])} chunks).")
22
  needs_rebuild = False
23
  if needs_rebuild:
@@ -31,8 +43,8 @@ DATA_RAW = Path("data/raw")
31
 
32
 
33
  @st.cache_data
34
- def load_cs1_20_meta() -> dict[str, dict]:
35
- """Load case metadata for all case studies (CS1 onwards)."""
36
  meta = {}
37
  if not DATA_RAW.exists():
38
  return meta
@@ -68,8 +80,8 @@ def _clean_excerpt(text: str, max_chars: int = 800) -> str:
68
  return text
69
 
70
 
71
- # ── Build ChromaDB before any Streamlit calls ─────────────────────────────────
72
- _ensure_chroma_built()
73
 
74
  # ── Page config ───────────────────────────────────────────────────────────────
75
  st.set_page_config(page_title="AI-Assisted Knowledge Platform for Managed Retreat", layout="wide")
@@ -207,7 +219,7 @@ with main_col:
207
  st.warning("Please enter a question.")
208
  elif re.search(r"\b(what|which|list|show|how many).{0,30}case stud", query, re.I):
209
  # Meta question — list all case studies directly from extracted JSONs
210
- cs_meta = load_cs1_20_meta()
211
  extracted_dir = Path("data/extracted")
212
  all_cases = []
213
  for f in sorted(extracted_dir.glob("CS*.json"), key=lambda p: int(re.search(r'\d+', p.stem).group())):
@@ -264,7 +276,7 @@ with main_col:
264
  st.markdown(f"**{q_text}** {val}")
265
 
266
  # ── Case studies used ─────────────────────────────────────────
267
- cs_meta = load_cs1_20_meta()
268
 
269
  seen_ids: set[str] = set()
270
  retrieved_cs = []
@@ -275,7 +287,7 @@ with main_col:
275
  retrieved_cs.append(c)
276
 
277
  mentioned_ids = {
278
- f"CS{n}" for n in re.findall(r'\bCS([1-9]|1[0-9]|20)\b', answer)
279
  }
280
 
281
  if retrieved_cs:
 
10
  from sheets_logger import log_query
11
 
12
 
13
+ def _ensure_embedding_index_built():
14
  """Ensure precomputed_embeddings.json has texts+metadatas (needed by numpy retriever)."""
15
  precomputed = Path("data/extracted/precomputed_embeddings.json")
16
  needs_rebuild = True
17
  if precomputed.exists():
18
  import json as _json
19
  pre = _json.loads(precomputed.read_text())
20
+ from ingest.embed_and_index import _content_hash, load_all_chunks
21
+
22
+ chunks = load_all_chunks()
23
+ texts = [c.get("text", _json.dumps(c)) for c in chunks]
24
+ ids = [f"{c.get('case_id', 'unknown')}_{c.get('chunk_index', i)}" for i, c in enumerate(chunks)]
25
+ metadatas = [{k: str(v) for k, v in c.items() if k != "text"} for c in chunks]
26
+ content_hash = _content_hash(ids, texts, metadatas)
27
+ if (
28
+ pre.get("ids") == ids
29
+ and pre.get("texts") == texts
30
+ and pre.get("metadatas") == metadatas
31
+ and pre.get("content_hash") == content_hash
32
+ ):
33
  print(f"Embeddings ready ({len(pre['ids'])} chunks).")
34
  needs_rebuild = False
35
  if needs_rebuild:
 
43
 
44
 
45
  @st.cache_data
46
+ def load_case_meta() -> dict[str, dict]:
47
+ """Load case metadata for all case studies with local case_meta.json files."""
48
  meta = {}
49
  if not DATA_RAW.exists():
50
  return meta
 
80
  return text
81
 
82
 
83
+ # ── Build embedding index before any Streamlit calls ──────────────────────────
84
+ _ensure_embedding_index_built()
85
 
86
  # ── Page config ───────────────────────────────────────────────────────────────
87
  st.set_page_config(page_title="AI-Assisted Knowledge Platform for Managed Retreat", layout="wide")
 
219
  st.warning("Please enter a question.")
220
  elif re.search(r"\b(what|which|list|show|how many).{0,30}case stud", query, re.I):
221
  # Meta question — list all case studies directly from extracted JSONs
222
+ cs_meta = load_case_meta()
223
  extracted_dir = Path("data/extracted")
224
  all_cases = []
225
  for f in sorted(extracted_dir.glob("CS*.json"), key=lambda p: int(re.search(r'\d+', p.stem).group())):
 
276
  st.markdown(f"**{q_text}** {val}")
277
 
278
  # ── Case studies used ─────────────────────────────────────────
279
+ cs_meta = load_case_meta()
280
 
281
  seen_ids: set[str] = set()
282
  retrieved_cs = []
 
287
  retrieved_cs.append(c)
288
 
289
  mentioned_ids = {
290
+ f"CS{n}" for n in re.findall(r'\bCS(\d+)\b', answer)
291
  }
292
 
293
  if retrieved_cs:
case_study_pipeline/onboard_case_study.py CHANGED
@@ -326,7 +326,7 @@ def main() -> None:
326
  copied = copy_pdfs_from_drive_folder(service, source_id, folder_id, local_dest)
327
  print(f"✓ Copied {len(copied)} PDF(s) into the Drive folder and into {local_dest}/")
328
 
329
- # 3. Write case_meta.json so the ingestion bridge can tag ChromaDB chunks
330
  import json as _json
331
  _meta_path = REPO_ROOT / "data" / "raw" / case_id / "case_meta.json"
332
  _meta_path.write_text(_json.dumps({
 
326
  copied = copy_pdfs_from_drive_folder(service, source_id, folder_id, local_dest)
327
  print(f"✓ Copied {len(copied)} PDF(s) into the Drive folder and into {local_dest}/")
328
 
329
+ # 3. Write case_meta.json so the ingestion bridge can tag RAG chunks
330
  import json as _json
331
  _meta_path = REPO_ROOT / "data" / "raw" / case_id / "case_meta.json"
332
  _meta_path.write_text(_json.dumps({
case_study_pipeline/select_best_run.py CHANGED
@@ -6,7 +6,7 @@ reads the Check 2 (or Check 1) grade from each, selects the winner, copies its
6
  Ver2.md to the canonical location (pipeline_output/CSX_Ver2.md), and writes
7
  pipeline_output/selected.json.
8
 
9
- The canonical Ver2.md is what ingest_pipeline_outputs.py reads into ChromaDB.
10
 
11
  Usage:
12
  python -m case_study_pipeline.select_best_run --case-id CS1
@@ -167,7 +167,7 @@ def main(argv: list[str] | None = None) -> None:
167
  for case_id in case_ids:
168
  select_for_case(case_id, force=args.force)
169
 
170
- print("\nDone. Run ingest/ingest_pipeline_outputs.py to push selected Ver2 files into ChromaDB.")
171
 
172
 
173
  if __name__ == "__main__":
 
6
  Ver2.md to the canonical location (pipeline_output/CSX_Ver2.md), and writes
7
  pipeline_output/selected.json.
8
 
9
+ The canonical Ver2.md is what ingest_pipeline_outputs.py reads into JSON chunks.
10
 
11
  Usage:
12
  python -m case_study_pipeline.select_best_run --case-id CS1
 
167
  for case_id in case_ids:
168
  select_for_case(case_id, force=args.force)
169
 
170
+ print("\nDone. Run ingest/ingest_pipeline_outputs.py to convert selected Ver2 files into JSON chunks.")
171
 
172
 
173
  if __name__ == "__main__":
data/extracted/case_summaries.json CHANGED
@@ -89,6 +89,15 @@
89
  "source": "Pipeline Ver2 — San Juan Managed Retreat",
90
  "text": "Case study: San Juan Managed Retreat (San Juan, Puerto Rico, USA)\nSection: Case Study Overview (Summary)\n\n## Case Name and Location\n**El Caño Martín Peña (CMP) — Eight Communities Aledañas al Caño Martín Peña Special Planning District**\nSan Juan, Puerto Rico — a U.S. territory with a distinct colonial governance context (classified as Global North by project codebook definition, though Puerto Rico's territorial and colonial status introduces important caveats)\n\n## Hazard(s)\nPrimary hazard: **riverine/tidal flooding**, driven by the progressive narrowing and blockage of a 3.75-mile tidal channel (El Caño Martín Peña) connecting the San José Lagoon to the San Juan Bay Estuary. The channel was severely obstructed by decades of informal landfill (debris, trash, wetland conversion), leading to chronic flooding that historically affected approximately 70% of the area [Source 1]. Secondary hazard: **hurricane events**, including the 1928 San Felipe and 1932 San Ciprián hurricanes (which originally accelerated migration into the area) and Hurricanes Irma and María (September 2017), after which 1,200 homes lost their roofs and 100 homes were destroyed within the eight communities [Source 1].\n\n## Type of Retreat\nA **community-driven, participatory managed retreat** embedded within a broader ecosystem restoration and community redevelopment initiative. The retreat component requires approximately 1,000 families who live along the canal right-of-way to relocate to allow for channel widening and dredging. Households are offered a choice between relocating within the eight communities (within the CLT footprint) or accepting a housing relocation voucher to move elsewhere. It is important to note that relocation is not purely voluntary in the conventional sense: households along the right-of-way *must* move for dredging to proceed; what is choice-based is the *destination and form* of their new housing, not whether relocation occurs at all [Source 1, Source 2]. The retreat is explicitly framed as inseparable from environmental restoration, flood mitigation, and long-term community resilience. It is not a standalone buyout program.\n\n## Scale\n- **Population**: Approximately 10,997 current residents; the population had previously been estimated at approximately 15,693–18,103. Sources identify a range of possible contributing factors to this decline, including governmental discrimination and stigmatization of the communities, though causation is not established [Source 1].\n- **Housing units**: 6,946 total units in the eight communities; approximately 1,000 families required to relocate for the dredging right-of-way; approximately 600 families had already moved into safe housing as of 2017 [Source 1, Source 2].\n- **Land/tenure**: The Community Land Trust (*Fideicomiso de la Tierra del Caño Martín Peña*) holds collective tenure on behalf of approximately 2,000 families (around 16.51% of the area's households), who share collective clear title [Source 1]. Source 2 states that \"the community collectively owns 200 acres (80 ha) of land\" [Source 2]; this figure is cited from Source 2 (Yarina et al. 2020) and is not independently confirmed in Source 1.\n- **Geographic scope**: Eight neighborhoods — Barrio Obrero Oeste, Barrio Obrero, San Ciprián, Parada 27, Buena Vista Santurce, Las Monjas, Buena Vista Hato Rey, and Israel–Bitumul — forming the Martín Peña Special Planning District [Source 1].\n\n## Key Actors\n- **G8 Coalition** (*Grupo de las Ocho Comunidades Aledañas al Caño Martín Peña*): Community coalition of the eight neighborhoods; central community governance body. Formed in 2004 [Source 1].\n- **Proyecto ENLACE del Caño Martín Peña**: Public corporation established by Puerto Rico Law 489 (2004) to manage environmental restoration and community development initiatives; serves as institutional intermediary between community and government [Source 1, Source 2].\n- **Fideicomiso de la Tierra del Caño Martín Peña (CLT)**: Community Land Trust; a private, non-profit organization created to manage the land, secure collective tenure, and prevent residents from being priced out [Source 2]. Provides housing opportunities for relocating families and protects residents from speculative displacement [Source 1, Source 2].\n- **Government of Puerto Rico / Puerto Rico Department of Housing**: State-level authority administering CDBG-DR and CDBG-MIT funds; responsible for Program R3 (Repair, Reconstruction, or Relocation) [Source 3].\n- **U.S. Army Corps of Engineers**: Dredging project partner; government of San Juan partnered with the Corps in 2001 to plan canal restoration; $163 million assigned to the Corps for dredging in 2022 [Source 1].\n- **U.S. Department of Housing and Urban Development (HUD)**: Federal funder; provided $46 million in CDBG-MIT funds for relocation [Source 1].\n- **U.S. Environmental Protection Agency (EPA)**: Urban Waters Federal Partnership participant [Source 1].\n- **Residents / community members**: Primary decision-makers and participants throughout all phases; the community-driven process ensured fair treatment and sought to minimize displacement-related risks as a main goal [Source 1].\n\n## Timeline\n- **1928, 1932**: San Felipe and San Ciprián hurricanes accelerate rural-to-urban migration; people settle along the tidal channel wetlands [Source 1].\n- **1930s–1950s**: Informal settlement of wetlands; rapid environmental degradation; mangrove swamps progressively filled in [Source 1].\n- **1940s–1960s**: Operation Bootstrap industrialization; urban renewal programs displace thousands, some settling near the Caño [Source 1].\n- **2001**: City of San Juan partners with U.S. Army Corps of Engineers on canal dredging proposal; residents fear displacement and begin to organize [Source 1].\n- **2002**: Residents begin organized coalition meetings; San Juan designates the eight communities as a Special Planning District [Source 1].\n- **2004**: Eight communities form the G8 Coalition; Puerto Rico Law 489 enacted, creating ENLACE as a public corporation; CLT established; Caño Martín Peña Comprehensive Infrastructure Master Plan (CIMP) developed [Source 1, Source 2].\n- **2016**: UN World Habitat Award received by the CLT/G8/ENLACE model [Source 1].\n- **2017**: Hurricanes Irma and María devastate Puerto Rico; 1,200 homes lost roofs and 100 homes destroyed in CMP; ENLACE identified opportunity to accelerate project but encountered CDBG-DR action plan constraints that, by not allowing new infrastructure or construction/rehabilitation of housing within the floodplain, promoted displacement rather than community cohesion [Source 1, Source 2]; approximately 600 families had already relocated by this point [Source 2].\n- **2022**: $163 million assigned to the U.S. Army Corps of Engineers for dredging; $46 million in CDBG-MIT provided by HUD for relocation [Source 1].\n- **2023**: Dredging work begins; updated CIMP issued (OLIN, 2023) [Source 1].\n- **Current stage**: **Ongoing** — dredging and relocation underway; CLT continues to operate; future land use planning for vacated corridor areas in progress.\n\n## Key Outcomes\n- **Achieved**: Collective tenure regularized for approximately 2,000 low-income families through the CLT; flooding incidence reduced through initial infrastructure works (prior to full dredging, flooding historically affected ~70% of the area); approximately 600 families successfully relocated with choice as of 2017; community governance and social cohesion preserved; international recognition (2016 UN World Habitat Award); legal institutionalization of community governance through Law 489 [Sources 1, 2].\n- **In progress**: Full-scale dredging and canal restoration; relocation of remaining families from right-of-way; ongoing CLT governance [Source 1].\n- **Unresolved/contested**: Tensions between CDBG-DR action plan restrictions (which, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than on-site community cohesion) and ENLACE's goals [Source 2]; population decline from approximately 15,693–18,103 to approximately 10,997, linked by community leaders to governmental discrimination and stigmatization, though causal attribution is uncertain [Source 1].\n\n---\n\n## Narrative Summary\n\nEl Caño Martín Peña is a 3.75-mile tidal channel running through the heart of San Juan, Puerto Rico, surrounded by eight densely populated informal neighborhoods that developed from the 1930s onward as rural migrants — many displaced by the 1928 and 1932 hurricanes and by government urban renewal programs — settled in the wetlands. Over decades, the canal became severely degraded: mangrove swamps approximately 250 feet wide were progressively filled in, narrowing the channel dramatically, and communities were chronically flooded (historically affecting about 70% of the area), lacked basic sewage and water infrastructure, and faced persistent threats of top-down displacement given the high real estate value of their land. With median household incomes of $17,817 and over 61% of residents living below the poverty line, these communities were among Puerto Rico's most economically marginalized [Source 1].\n\nIn 2001, when the City of San Juan and the U.S. Army Corps of Engineers announced plans to dredge the canal, residents feared forced removal. Rather than accepting displacement, the eight communities organized. By 2004, they had formed the G8 Coalition and successfully lobbied the Puerto Rican government to pass Law 489, creating Proyecto ENLACE — a public corporation tasked with developing and implementing a comprehensive community-driven plan for the canal and surrounding neighborhoods. Simultaneously, the communities established the Fideicomiso de la Tierra (Community Land Trust), a private non-profit organization that secured collective tenure for approximately 2,000 families and protected residents from speculative displacement [Sources 1, 2]. The CLT and ENLACE together produced a Comprehensive Development and Land Use Plan, developed through a sustained participatory process involving over 700 community engagement activities, that integrated dredging, flood mitigation, housing stabilization, social services, and managed retreat into a single cohesive vision [Source 2].\n\nThe managed retreat component requires approximately 1,000 families to relocate to allow for canal widening and dredging. This is choice-based within a constrained setting: households along the right-of-way are required to move, but they select housing within the CLT footprint (preserving community membership and social networks) or accept a relocation voucher [Source 1]. As relocated resident Vicky Castro described: \"They give you the option to choose, they don't choose for you, you make your own decisions\" [Source 2]. The relocation process is explained clearly and repeatedly at community meetings; ENLACE provides mental health services and comprehensive support throughout [Source 2]. One constraint prompting relocation is that the Government of Puerto Rico does not allow elevation of homes in the floodplain [Source 1]. By 2017, approximately 600 families had already relocated [Source 2]. Funded by $46 million in CDBG-MIT from HUD and $163 million assigned to the Army Corps for dredging, construction began in 2023. The case has received international recognition, including the 2016 UN World Habitat Award, as an example of participatory, equity-driven managed retreat that achieves resilience goals without reproducing the top-down displacement historically imposed on Puerto Rico's informal communities. However, post-Hurricane María complications — including CDBG-DR action plan restrictions that, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than community cohesion — have created tensions between federal recovery frameworks and ENLACE's vision [Source 2].\n\n---"
91
  },
 
 
 
 
 
 
 
 
 
92
  {
93
  "case_id": "CS12",
94
  "chunk_index": 999,
 
89
  "source": "Pipeline Ver2 — San Juan Managed Retreat",
90
  "text": "Case study: San Juan Managed Retreat (San Juan, Puerto Rico, USA)\nSection: Case Study Overview (Summary)\n\n## Case Name and Location\n**El Caño Martín Peña (CMP) — Eight Communities Aledañas al Caño Martín Peña Special Planning District**\nSan Juan, Puerto Rico — a U.S. territory with a distinct colonial governance context (classified as Global North by project codebook definition, though Puerto Rico's territorial and colonial status introduces important caveats)\n\n## Hazard(s)\nPrimary hazard: **riverine/tidal flooding**, driven by the progressive narrowing and blockage of a 3.75-mile tidal channel (El Caño Martín Peña) connecting the San José Lagoon to the San Juan Bay Estuary. The channel was severely obstructed by decades of informal landfill (debris, trash, wetland conversion), leading to chronic flooding that historically affected approximately 70% of the area [Source 1]. Secondary hazard: **hurricane events**, including the 1928 San Felipe and 1932 San Ciprián hurricanes (which originally accelerated migration into the area) and Hurricanes Irma and María (September 2017), after which 1,200 homes lost their roofs and 100 homes were destroyed within the eight communities [Source 1].\n\n## Type of Retreat\nA **community-driven, participatory managed retreat** embedded within a broader ecosystem restoration and community redevelopment initiative. The retreat component requires approximately 1,000 families who live along the canal right-of-way to relocate to allow for channel widening and dredging. Households are offered a choice between relocating within the eight communities (within the CLT footprint) or accepting a housing relocation voucher to move elsewhere. It is important to note that relocation is not purely voluntary in the conventional sense: households along the right-of-way *must* move for dredging to proceed; what is choice-based is the *destination and form* of their new housing, not whether relocation occurs at all [Source 1, Source 2]. The retreat is explicitly framed as inseparable from environmental restoration, flood mitigation, and long-term community resilience. It is not a standalone buyout program.\n\n## Scale\n- **Population**: Approximately 10,997 current residents; the population had previously been estimated at approximately 15,693–18,103. Sources identify a range of possible contributing factors to this decline, including governmental discrimination and stigmatization of the communities, though causation is not established [Source 1].\n- **Housing units**: 6,946 total units in the eight communities; approximately 1,000 families required to relocate for the dredging right-of-way; approximately 600 families had already moved into safe housing as of 2017 [Source 1, Source 2].\n- **Land/tenure**: The Community Land Trust (*Fideicomiso de la Tierra del Caño Martín Peña*) holds collective tenure on behalf of approximately 2,000 families (around 16.51% of the area's households), who share collective clear title [Source 1]. Source 2 states that \"the community collectively owns 200 acres (80 ha) of land\" [Source 2]; this figure is cited from Source 2 (Yarina et al. 2020) and is not independently confirmed in Source 1.\n- **Geographic scope**: Eight neighborhoods — Barrio Obrero Oeste, Barrio Obrero, San Ciprián, Parada 27, Buena Vista Santurce, Las Monjas, Buena Vista Hato Rey, and Israel–Bitumul — forming the Martín Peña Special Planning District [Source 1].\n\n## Key Actors\n- **G8 Coalition** (*Grupo de las Ocho Comunidades Aledañas al Caño Martín Peña*): Community coalition of the eight neighborhoods; central community governance body. Formed in 2004 [Source 1].\n- **Proyecto ENLACE del Caño Martín Peña**: Public corporation established by Puerto Rico Law 489 (2004) to manage environmental restoration and community development initiatives; serves as institutional intermediary between community and government [Source 1, Source 2].\n- **Fideicomiso de la Tierra del Caño Martín Peña (CLT)**: Community Land Trust; a private, non-profit organization created to manage the land, secure collective tenure, and prevent residents from being priced out [Source 2]. Provides housing opportunities for relocating families and protects residents from speculative displacement [Source 1, Source 2].\n- **Government of Puerto Rico / Puerto Rico Department of Housing**: State-level authority administering CDBG-DR and CDBG-MIT funds; responsible for Program R3 (Repair, Reconstruction, or Relocation) [Source 3].\n- **U.S. Army Corps of Engineers**: Dredging project partner; government of San Juan partnered with the Corps in 2001 to plan canal restoration; $163 million assigned to the Corps for dredging in 2022 [Source 1].\n- **U.S. Department of Housing and Urban Development (HUD)**: Federal funder; provided $46 million in CDBG-MIT funds for relocation [Source 1].\n- **U.S. Environmental Protection Agency (EPA)**: Urban Waters Federal Partnership participant [Source 1].\n- **Residents / community members**: Primary decision-makers and participants throughout all phases; the community-driven process ensured fair treatment and sought to minimize displacement-related risks as a main goal [Source 1].\n\n## Timeline\n- **1928, 1932**: San Felipe and San Ciprián hurricanes accelerate rural-to-urban migration; people settle along the tidal channel wetlands [Source 1].\n- **1930s–1950s**: Informal settlement of wetlands; rapid environmental degradation; mangrove swamps progressively filled in [Source 1].\n- **1940s–1960s**: Operation Bootstrap industrialization; urban renewal programs displace thousands, some settling near the Caño [Source 1].\n- **2001**: City of San Juan partners with U.S. Army Corps of Engineers on canal dredging proposal; residents fear displacement and begin to organize [Source 1].\n- **2002**: Residents begin organized coalition meetings; San Juan designates the eight communities as a Special Planning District [Source 1].\n- **2004**: Eight communities form the G8 Coalition; Puerto Rico Law 489 enacted, creating ENLACE as a public corporation; CLT established; Caño Martín Peña Comprehensive Infrastructure Master Plan (CIMP) developed [Source 1, Source 2].\n- **2016**: UN World Habitat Award received by the CLT/G8/ENLACE model [Source 1].\n- **2017**: Hurricanes Irma and María devastate Puerto Rico; 1,200 homes lost roofs and 100 homes destroyed in CMP; ENLACE identified opportunity to accelerate project but encountered CDBG-DR action plan constraints that, by not allowing new infrastructure or construction/rehabilitation of housing within the floodplain, promoted displacement rather than community cohesion [Source 1, Source 2]; approximately 600 families had already relocated by this point [Source 2].\n- **2022**: $163 million assigned to the U.S. Army Corps of Engineers for dredging; $46 million in CDBG-MIT provided by HUD for relocation [Source 1].\n- **2023**: Dredging work begins; updated CIMP issued (OLIN, 2023) [Source 1].\n- **Current stage**: **Ongoing** — dredging and relocation underway; CLT continues to operate; future land use planning for vacated corridor areas in progress.\n\n## Key Outcomes\n- **Achieved**: Collective tenure regularized for approximately 2,000 low-income families through the CLT; flooding incidence reduced through initial infrastructure works (prior to full dredging, flooding historically affected ~70% of the area); approximately 600 families successfully relocated with choice as of 2017; community governance and social cohesion preserved; international recognition (2016 UN World Habitat Award); legal institutionalization of community governance through Law 489 [Sources 1, 2].\n- **In progress**: Full-scale dredging and canal restoration; relocation of remaining families from right-of-way; ongoing CLT governance [Source 1].\n- **Unresolved/contested**: Tensions between CDBG-DR action plan restrictions (which, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than on-site community cohesion) and ENLACE's goals [Source 2]; population decline from approximately 15,693–18,103 to approximately 10,997, linked by community leaders to governmental discrimination and stigmatization, though causal attribution is uncertain [Source 1].\n\n---\n\n## Narrative Summary\n\nEl Caño Martín Peña is a 3.75-mile tidal channel running through the heart of San Juan, Puerto Rico, surrounded by eight densely populated informal neighborhoods that developed from the 1930s onward as rural migrants — many displaced by the 1928 and 1932 hurricanes and by government urban renewal programs — settled in the wetlands. Over decades, the canal became severely degraded: mangrove swamps approximately 250 feet wide were progressively filled in, narrowing the channel dramatically, and communities were chronically flooded (historically affecting about 70% of the area), lacked basic sewage and water infrastructure, and faced persistent threats of top-down displacement given the high real estate value of their land. With median household incomes of $17,817 and over 61% of residents living below the poverty line, these communities were among Puerto Rico's most economically marginalized [Source 1].\n\nIn 2001, when the City of San Juan and the U.S. Army Corps of Engineers announced plans to dredge the canal, residents feared forced removal. Rather than accepting displacement, the eight communities organized. By 2004, they had formed the G8 Coalition and successfully lobbied the Puerto Rican government to pass Law 489, creating Proyecto ENLACE — a public corporation tasked with developing and implementing a comprehensive community-driven plan for the canal and surrounding neighborhoods. Simultaneously, the communities established the Fideicomiso de la Tierra (Community Land Trust), a private non-profit organization that secured collective tenure for approximately 2,000 families and protected residents from speculative displacement [Sources 1, 2]. The CLT and ENLACE together produced a Comprehensive Development and Land Use Plan, developed through a sustained participatory process involving over 700 community engagement activities, that integrated dredging, flood mitigation, housing stabilization, social services, and managed retreat into a single cohesive vision [Source 2].\n\nThe managed retreat component requires approximately 1,000 families to relocate to allow for canal widening and dredging. This is choice-based within a constrained setting: households along the right-of-way are required to move, but they select housing within the CLT footprint (preserving community membership and social networks) or accept a relocation voucher [Source 1]. As relocated resident Vicky Castro described: \"They give you the option to choose, they don't choose for you, you make your own decisions\" [Source 2]. The relocation process is explained clearly and repeatedly at community meetings; ENLACE provides mental health services and comprehensive support throughout [Source 2]. One constraint prompting relocation is that the Government of Puerto Rico does not allow elevation of homes in the floodplain [Source 1]. By 2017, approximately 600 families had already relocated [Source 2]. Funded by $46 million in CDBG-MIT from HUD and $163 million assigned to the Army Corps for dredging, construction began in 2023. The case has received international recognition, including the 2016 UN World Habitat Award, as an example of participatory, equity-driven managed retreat that achieves resilience goals without reproducing the top-down displacement historically imposed on Puerto Rico's informal communities. However, post-Hurricane María complications — including CDBG-DR action plan restrictions that, by prohibiting reconstruction or rehabilitation within the floodplain, promoted displacement rather than community cohesion — have created tensions between federal recovery frameworks and ENLACE's vision [Source 2].\n\n---"
91
  },
92
+ {
93
+ "case_id": "CS11",
94
+ "chunk_index": 999,
95
+ "section": "overview_summary",
96
+ "location": "Terrebonne Parish, Louisiana",
97
+ "country": "USA",
98
+ "source": "Pipeline Ver2 — Terrebonne Parish Resettlement",
99
+ "text": "Case study: Terrebonne Parish Resettlement (Terrebonne Parish, Louisiana, USA)\nSection: Case Study Overview (Summary)\n\n## Case Identification\n\n**Case study title:** LA SAFE Terrebonne Parish Adaptation Strategy and the Buyout/Optional Home Owner Assistance for Permanent Resident Households program (with reference to the related, but administratively separate, Isle de Jean Charles resettlement), Terrebonne Parish, Louisiana, United States.\n\n**Global North or South:** North. Terrebonne Parish is located in the United States, a Global North country, consistent with the codebook's definition of \"North\" as broadly describing countries in Europe, North America, and the Oceania nations of Australia and New Zealand [2].\n\n---\n\n## General Information\n\n### Is there any evidence of a proactive or reactive trigger in the MR decision process? (Reactive/Proactive)\n\n**Answer:** Both Reactive and Proactive elements are supported, with the overall LA SAFE program best characterized as Proactive (anticipatory), layered on top of a long history of Reactive, incremental household-level adaptation.\n\n**Narrative:** The LA SAFE (Louisiana's Strategic Adaptations for Future Environments) planning process that produced the Terrebonne Parish Adaptation Strategy (2019) was directly enabled by funding from the U.S. Department of Housing and Urban Development's National Disaster Resilience Competition (NDRC) and the Rockefeller Foundation, which became available in response to Hurricane Isaac (2012) [1]. In this sense, the program's origin is reactive — it was a \"Qualifying Disaster\" (Hurricane Isaac) that created the funding opportunity [1]. However, once funded, the LA SAFE process was explicitly framed as anticipatory: it sought to plan for \"future environments\" and used 50-year flood-risk projections (CPRA's 2017 Coastal Master Plan modeling, Medium Environmental Scenario, 2067) to define high-, moderate-, and low-risk zones and to develop strategies before the projected risk materializes [1]. The Terrebonne Adaptation Strategy explicitly states that LA SAFE \"complements\" reactive structural efforts (the Morganza to the Gulf levee system) with \"community-driven, smaller-scale adaptation strategies\" oriented toward the future [1].\n\nAt the household level, however, Nelson et al. [4] document a long pattern of reactive adaptation: residents in lower Terrebonne Parish communities (Chauvin, Dulac, Isle de Jean Charles, Montegut, Theriot) have historically responded to specific flood events by elevating their homes after the fact (e.g., \"Resident 28\" raised their home after Hurricane Juan in 1985, and again experienced flooding after Hurricane Lili in 2002, eventually relocating through the Isle de Jean Charles resettlement program) [4]. The Isle de Jean Charles resettlement itself, funded through a $48.3 million CDBG-DR award in 2016, was also tied to a \"Qualifying Disaster\" / NDRC funding cycle, and followed decades of land loss and repeated storm damage (including Tropical Storm Lili in 2002, documented by Burley et al. [6]) — making it reactive in its funding trigger but framed by the State as a proactive, anticipatory resettlement given the island's documented loss of more than 98% of its land area over six decades [4].\n\n**Category rationale:** Both categories are selected because the evidence shows a mixed picture: program funding triggers are reactive (tied to disaster declarations and post-disaster federal competitions), while the planning content and stated intent of the LA SAFE Adaptation Strategy and the Isle de Jean Charles resettlement are explicitly proactive/anticipatory, using future risk projections (2067 horizon) rather than only addressing damage already incurred [1], [4].\n\n**Uncertainty/Alternative interpretations:** An alternative interpretation could weight the funding origin more heavily and code the case primarily Reactive, since post-Isaac NDRC/CDBG-DR funding was the proximate trigger that made the LA SAFE process possible [1]. However, while post-Isaac funding was a critical trigger, the LA SAFE program encompasses both proactive planning and responses to past disaster impacts: it is built on the explicit forward-looking framing and the 50-year (2067) risk-modeling basis for zoning described above, and states a goal of acting \"before major impacts occur\" for most of the parish — only approximately 7-12 households are currently being relocated [1], [3], [5], none of which have yet experienced the catastrophic land loss seen at Isle de Jean Charles [4]. The working interpretation therefore favors Proactive as the dominant trigger type for the decision process itself, with Reactive noted as the funding-origin trigger, rather than treating the case as driven solely by reactive, post-disaster motivations.\n\n---\n\n### Date, Number of households affected, Number of population affected, Total funding\n\n**Answer:** LA SAFE planning process: 2016-2019 (Adaptation Strategy published April 2019). Buyout/Optional Home Owner Assistance for Permanent Resident Households: approximately 7-12 households, reflecting inconsistencies in source reporting — the LA SAFE Adaptation Strategy and Summary documents state approximately seven households [1], [3], while Birch et al. report approximately 12 for what appears to be the same or a closely related program [5]. Isle de Jean Charles resettlement (related but distinct program): funded 2016, ongoing through the 2020s. Total funding: the only total funding figure documented in the reviewed sources is the $48.3 million CDBG-DR award, which is specifically attributed to the Isle de Jean Charles resettlement [4]; no total funding figure distinct to the Terrebonne Buyout/Optional Home Owner Assistance program was found.\n\n**Narrative:** The LA SAFE program for Terrebonne Parish ran its community engagement process across five rounds of meetings between roughly 2016 and 2018, engaging 371 residents and community leaders, with the final Adaptation Strategy document published in April 2019 [1]. The program was funded through HUD's National Disaster Resilience Competition and the Rockefeller Fou"
100
+ },
101
  {
102
  "case_id": "CS12",
103
  "chunk_index": 999,
data/extracted/precomputed_embeddings.json CHANGED
The diff for this file is too large to render. See raw diff
 
docs/extraction_schema.md CHANGED
@@ -1,22 +1,79 @@
1
- # Extraction Schema
2
-
3
- Fields captured from each managed retreat case study:
4
-
5
- | Field | Description | Values |
6
- |-------|-------------|--------|
7
- | `case_id` | Unique identifier (CS1–CS8) | string |
8
- | `location` | Place name | string |
9
- | `country` | Canada / USA / Indonesia | string |
10
- | `hazard_type` | Primary climate hazard | string |
11
- | `trigger` | What initiated the retreat | string |
12
- | `year_initiated` | Year retreat process began | integer |
13
- | `stage` | Where the case is in the process | `pre-retreat`, `during`, `post-retreat` |
14
- | `governance` | Governing body (municipal, federal, First Nation, etc.) | string |
15
- | `funding_mechanism` | How the retreat was/is funded | string |
16
- | `compensation_type` | What residents received | string |
17
- | `voluntariness` | Degree of choice for residents | `voluntary`, `incentivized`, `involuntary` |
18
- | `equity_outcomes` | Who benefited/was harmed | string |
19
- | `implementation_barriers` | List of obstacles encountered | array |
20
- | `community_engagement` | How the community was involved | string |
21
- | `success_rating` | Researcher assessment | `successful`, `failed`, `ongoing` |
22
- | `source` | Citation for the case study | string |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Current Extraction and Indexing Format
2
+
3
+ The original project sketch used a compact structured extraction schema for a
4
+ small pilot set of managed-retreat cases. The current RAG knowledge base uses
5
+ long-form case-study memo sections generated by the LLM extraction/audit
6
+ pipeline.
7
+
8
+ ## Source Lineage
9
+
10
+ Case-study candidates come from the Ajibade et al. spreadsheet:
11
+
12
+ `https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`
13
+
14
+ Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and
15
+ pipeline outputs. A case becomes queryable only after it has:
16
+
17
+ 1. A canonical `data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md`.
18
+ 2. An extracted `data/extracted/CSxx.json`.
19
+ 3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`.
20
+
21
+ ## Extracted Chunk Shape
22
+
23
+ Each `data/extracted/CSxx.json` file is a list of section chunks:
24
+
25
+ ```json
26
+ {
27
+ "case_id": "CS24",
28
+ "chunk_index": 0,
29
+ "section": "overview",
30
+ "location": "Queens, New York",
31
+ "country": "United States",
32
+ "source": "Pipeline Ver2 — Queens",
33
+ "text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
34
+ }
35
+ ```
36
+
37
+ Field meanings:
38
+
39
+ | Field | Meaning |
40
+ |-------|---------|
41
+ | `case_id` | Case identifier such as `CS24`. |
42
+ | `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. |
43
+ | `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. |
44
+ | `location` | Location copied from `case_meta.json` when available. |
45
+ | `country` | Country copied from `case_meta.json` when available. |
46
+ | `source` | Human-readable source label for retrieved evidence. |
47
+ | `text` | Self-contained chunk text sent to the retriever and LLM prompt. |
48
+
49
+ `data/extracted/case_summaries.json` contains one `overview_summary` chunk per
50
+ case when a usable overview section can be extracted. These chunks support
51
+ overview and comparison queries.
52
+
53
+ ## Refresh Commands
54
+
55
+ Run these after adding or changing canonical Ver2 outputs:
56
+
57
+ ```bash
58
+ python3 ingest/ingest_pipeline_outputs.py
59
+ python3 ingest/create_summary_chunks.py
60
+ python3 ingest/embed_and_index.py
61
+ ```
62
+
63
+ `embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It
64
+ checks chunk IDs, text, metadata, and a content hash so changed Ver2 content
65
+ triggers a rebuild.
66
+
67
+ ## Current State Checked July 14, 2026
68
+
69
+ - Canonical Ver2 cases: 34.
70
+ - Extracted case JSON files: 34.
71
+ - Embedded cases: 34.
72
+ - Embedded chunks: 231.
73
+ - Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`.
74
+ - `CS32` is present in the spreadsheet/local metadata but does not currently
75
+ have a canonical Ver2 output or extracted JSON.
76
+ - `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the
77
+ opening `Case Identification` / `General Information` sections as a fallback
78
+ because its Ver2 document does not have a literal `Case Study Overview`
79
+ heading.
ingest/create_summary_chunks.py CHANGED
@@ -34,6 +34,9 @@ _PREAMBLE_RE = re.compile(
34
  re.IGNORECASE | re.DOTALL,
35
  )
36
 
 
 
 
37
 
38
  def _find_case_folder(case_id: str) -> Path | None:
39
  for p in DATA_RAW.iterdir():
@@ -61,6 +64,28 @@ def _all_case_ids() -> list[str]:
61
  return sorted(set(ids), key=lambda x: int(x[2:]))
62
 
63
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
64
  def build_summary_chunks() -> list[dict]:
65
  chunks = []
66
  for case_id in _all_case_ids():
@@ -79,6 +104,8 @@ def build_summary_chunks() -> list[dict]:
79
  # Fallback: grab preamble before Q&A questions
80
  m2 = _PREAMBLE_RE.match(text)
81
  overview_text = m2.group(1).strip() if m2 else ""
 
 
82
 
83
  if not overview_text:
84
  print(f" [{case_id}] No CASE STUDY OVERVIEW section found — skipping.")
 
34
  re.IGNORECASE | re.DOTALL,
35
  )
36
 
37
+ _H2_RE = re.compile(r"^##\s+(.+)$", re.MULTILINE)
38
+ _FALLBACK_SECTION_LIMIT = 6000
39
+
40
 
41
  def _find_case_folder(case_id: str) -> Path | None:
42
  for p in DATA_RAW.iterdir():
 
64
  return sorted(set(ids), key=lambda x: int(x[2:]))
65
 
66
 
67
+ def _fallback_opening_sections(text: str) -> str:
68
+ """Use opening memo sections when a Ver2 lacks a Case Study Overview heading."""
69
+ matches = list(_H2_RE.finditer(text))
70
+ if not matches:
71
+ return ""
72
+
73
+ pieces = []
74
+ for i, match in enumerate(matches):
75
+ heading = match.group(1).strip()
76
+ if not re.search(r"case identification|general information", heading, re.I):
77
+ continue
78
+ start = match.end()
79
+ end = matches[i + 1].start() if i + 1 < len(matches) else len(text)
80
+ body = text[start:end].strip()
81
+ if body:
82
+ pieces.append(f"## {heading}\n\n{body}")
83
+ if len("\n\n".join(pieces)) >= _FALLBACK_SECTION_LIMIT:
84
+ break
85
+
86
+ return "\n\n".join(pieces)[:_FALLBACK_SECTION_LIMIT].strip()
87
+
88
+
89
  def build_summary_chunks() -> list[dict]:
90
  chunks = []
91
  for case_id in _all_case_ids():
 
104
  # Fallback: grab preamble before Q&A questions
105
  m2 = _PREAMBLE_RE.match(text)
106
  overview_text = m2.group(1).strip() if m2 else ""
107
+ if len(overview_text) < 100:
108
+ overview_text = _fallback_opening_sections(text)
109
 
110
  if not overview_text:
111
  print(f" [{case_id}] No CASE STUDY OVERVIEW section found — skipping.")
ingest/embed_and_index.py CHANGED
@@ -2,12 +2,14 @@
2
  embed_and_index.py — Precompute and save embeddings for all case study chunks.
3
 
4
  Saves data/extracted/precomputed_embeddings.json:
5
- { "ids": [...], "embeddings": [...], "texts": [...], "metadatas": [...] }
 
6
 
7
  Retrieval uses numpy cosine similarity in rag/retriever.py (no ChromaDB —
8
  it is broken on Python 3.13 / HuggingFace Spaces).
9
  """
10
  import json
 
11
  import sys
12
  from pathlib import Path
13
  from sentence_transformers import SentenceTransformer
@@ -30,6 +32,15 @@ def load_all_chunks() -> list[dict]:
30
  return chunks
31
 
32
 
 
 
 
 
 
 
 
 
 
33
  def embed_and_index():
34
  chunks = load_all_chunks()
35
  if not chunks:
@@ -39,11 +50,17 @@ def embed_and_index():
39
  texts = [c.get("text", json.dumps(c)) for c in chunks]
40
  ids = [f"{c.get('case_id', 'unknown')}_{c.get('chunk_index', i)}" for i, c in enumerate(chunks)]
41
  metadatas = [{k: str(v) for k, v in c.items() if k != "text"} for c in chunks]
 
42
 
43
- # Use precomputed embeddings if IDs match — nothing to do
44
  if PRECOMPUTED_PATH.exists():
45
  pre = json.loads(PRECOMPUTED_PATH.read_text())
46
- if pre.get("ids") == ids and "texts" in pre and "metadatas" in pre:
 
 
 
 
 
47
  print(f"Precomputed embeddings up to date ({len(ids)} chunks).")
48
  return
49
  print("Precomputed embeddings stale — recomputing...")
@@ -60,6 +77,7 @@ def embed_and_index():
60
  "embeddings": embeddings,
61
  "texts": texts,
62
  "metadatas": metadatas,
 
63
  }))
64
  print(f"Saved precomputed embeddings for {len(texts)} chunks.")
65
  print("\nNext: streamlit run app/app.py")
 
2
  embed_and_index.py — Precompute and save embeddings for all case study chunks.
3
 
4
  Saves data/extracted/precomputed_embeddings.json:
5
+ { "ids": [...], "embeddings": [...], "texts": [...], "metadatas": [...],
6
+ "content_hash": "..." }
7
 
8
  Retrieval uses numpy cosine similarity in rag/retriever.py (no ChromaDB —
9
  it is broken on Python 3.13 / HuggingFace Spaces).
10
  """
11
  import json
12
+ import hashlib
13
  import sys
14
  from pathlib import Path
15
  from sentence_transformers import SentenceTransformer
 
32
  return chunks
33
 
34
 
35
+ def _content_hash(ids: list[str], texts: list[str], metadatas: list[dict]) -> str:
36
+ payload = json.dumps(
37
+ {"ids": ids, "texts": texts, "metadatas": metadatas},
38
+ sort_keys=True,
39
+ ensure_ascii=False,
40
+ )
41
+ return hashlib.sha256(payload.encode("utf-8")).hexdigest()
42
+
43
+
44
  def embed_and_index():
45
  chunks = load_all_chunks()
46
  if not chunks:
 
50
  texts = [c.get("text", json.dumps(c)) for c in chunks]
51
  ids = [f"{c.get('case_id', 'unknown')}_{c.get('chunk_index', i)}" for i, c in enumerate(chunks)]
52
  metadatas = [{k: str(v) for k, v in c.items() if k != "text"} for c in chunks]
53
+ content_hash = _content_hash(ids, texts, metadatas)
54
 
55
+ # Use precomputed embeddings only if chunk IDs and chunk contents match.
56
  if PRECOMPUTED_PATH.exists():
57
  pre = json.loads(PRECOMPUTED_PATH.read_text())
58
+ if (
59
+ pre.get("ids") == ids
60
+ and pre.get("texts") == texts
61
+ and pre.get("metadatas") == metadatas
62
+ and pre.get("content_hash") == content_hash
63
+ ):
64
  print(f"Precomputed embeddings up to date ({len(ids)} chunks).")
65
  return
66
  print("Precomputed embeddings stale — recomputing...")
 
77
  "embeddings": embeddings,
78
  "texts": texts,
79
  "metadatas": metadatas,
80
+ "content_hash": content_hash,
81
  }))
82
  print(f"Saved precomputed embeddings for {len(texts)} chunks.")
83
  print("\nNext: streamlit run app/app.py")
ingest/ingest_pipeline_outputs.py CHANGED
@@ -1,5 +1,5 @@
1
  """
2
- ingest_pipeline_outputs.py — Convert selected Ver2.md files into ChromaDB-ready JSON.
3
 
4
  Reads every data/raw/CSX/pipeline_output/CSX_Ver2.md (the canonical file written
5
  by select_best_run.py), splits it into one chunk per questionnaire section, and
@@ -14,7 +14,7 @@ Usage:
14
  python ingest/ingest_pipeline_outputs.py --all --force # overwrite existing JSON
15
 
16
  The script is idempotent: re-running it for a case study overwrites its
17
- data/extracted/CSX.json, so ChromaDB always reflects the latest selected Ver2.
18
  """
19
 
20
  from __future__ import annotations
@@ -29,7 +29,7 @@ DATA_RAW = REPO_ROOT / "data" / "raw"
29
  EXTRACTED_DIR = REPO_ROOT / "data" / "extracted"
30
 
31
  # ---------------------------------------------------------------------------
32
- # Section name → ChromaDB section key
33
  # The keys match the field names used in CS1_IDJC.json so all chunks are
34
  # queryable with the same metadata filters.
35
  # ---------------------------------------------------------------------------
@@ -187,7 +187,7 @@ def ingest_case(case_id: str, force: bool = False) -> bool:
187
 
188
  def main(argv: list[str] | None = None) -> None:
189
  parser = argparse.ArgumentParser(
190
- description="Convert selected Ver2.md files into ChromaDB-ready JSON chunks."
191
  )
192
  group = parser.add_mutually_exclusive_group()
193
  group.add_argument("--case-id", nargs="+", metavar="ID",
 
1
  """
2
+ ingest_pipeline_outputs.py — Convert selected Ver2.md files into RAG JSON chunks.
3
 
4
  Reads every data/raw/CSX/pipeline_output/CSX_Ver2.md (the canonical file written
5
  by select_best_run.py), splits it into one chunk per questionnaire section, and
 
14
  python ingest/ingest_pipeline_outputs.py --all --force # overwrite existing JSON
15
 
16
  The script is idempotent: re-running it for a case study overwrites its
17
+ data/extracted/CSX.json, so the embedding store can reflect the latest selected Ver2.
18
  """
19
 
20
  from __future__ import annotations
 
29
  EXTRACTED_DIR = REPO_ROOT / "data" / "extracted"
30
 
31
  # ---------------------------------------------------------------------------
32
+ # Section name → retrieval section key
33
  # The keys match the field names used in CS1_IDJC.json so all chunks are
34
  # queryable with the same metadata filters.
35
  # ---------------------------------------------------------------------------
 
187
 
188
  def main(argv: list[str] | None = None) -> None:
189
  parser = argparse.ArgumentParser(
190
+ description="Convert selected Ver2.md files into RAG JSON chunks."
191
  )
192
  group = parser.add_mutually_exclusive_group()
193
  group.add_argument("--case-id", nargs="+", metavar="ID",
rag/pipeline.py CHANGED
@@ -31,21 +31,21 @@ def query_ollama(prompt: str) -> str:
31
  return response.json()["response"].strip()
32
 
33
 
34
- def query_claude(prompt: str) -> str:
35
  client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
36
  message = client.messages.create(
37
  model="claude-sonnet-4-6",
38
- max_tokens=4096,
39
  messages=[{"role": "user", "content": prompt}],
40
  )
41
  return message.content[0].text.strip()
42
 
43
 
44
- def query_openai(prompt: str) -> str:
45
  client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
46
  response = client.chat.completions.create(
47
  model="gpt-4o-mini",
48
- max_completion_tokens=2048,
49
  messages=[{"role": "user", "content": prompt}],
50
  )
51
  return response.choices[0].message.content.strip()
@@ -86,9 +86,9 @@ def run_pipeline(
86
  prompt = build_prompt(query, chunks, questionnaire_answers)
87
 
88
  if model == "claude":
89
- answer = query_claude(prompt)
90
  elif model in ("openai", "gpt"):
91
- answer = query_openai(prompt)
92
  else:
93
  answer = query_ollama(prompt)
94
 
 
31
  return response.json()["response"].strip()
32
 
33
 
34
+ def query_claude(prompt: str, max_tokens: int = 4096) -> str:
35
  client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
36
  message = client.messages.create(
37
  model="claude-sonnet-4-6",
38
+ max_tokens=max_tokens,
39
  messages=[{"role": "user", "content": prompt}],
40
  )
41
  return message.content[0].text.strip()
42
 
43
 
44
+ def query_openai(prompt: str, max_tokens: int = 2048) -> str:
45
  client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
46
  response = client.chat.completions.create(
47
  model="gpt-4o-mini",
48
+ max_completion_tokens=max_tokens,
49
  messages=[{"role": "user", "content": prompt}],
50
  )
51
  return response.choices[0].message.content.strip()
 
86
  prompt = build_prompt(query, chunks, questionnaire_answers)
87
 
88
  if model == "claude":
89
+ answer = query_claude(prompt, max_tokens=8192 if _OVERVIEW_RE.search(query) else 4096)
90
  elif model in ("openai", "gpt"):
91
+ answer = query_openai(prompt, max_tokens=4096 if _OVERVIEW_RE.search(query) else 2048)
92
  else:
93
  answer = query_ollama(prompt)
94
 
rag/prompt_builder.py CHANGED
@@ -1,30 +1,70 @@
1
- from pathlib import Path
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
 
3
 
4
  def build_prompt(query: str, retrieved_chunks: list[dict], questionnaire_answers: dict | None = None) -> str:
5
- context_blocks = "\n\n".join(
6
- f"[Source: {c['metadata'].get('source', 'unknown')} | Case: {c['metadata'].get('case_id', '?')}]\n{c['text']}"
7
- for c in retrieved_chunks
8
  )
9
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  answers_section = ""
11
  if questionnaire_answers:
12
  answers_section = "\n\nUser context from questionnaire:\n" + "\n".join(
13
  f"- {k}: {v}" for k, v in questionnaire_answers.items()
14
  )
15
 
16
- is_overview = any(
17
- kw in query.lower()
18
- for kw in ("summar", "overview", "all case", "every case", "list", "compare", "across")
19
- )
20
-
21
  if is_overview:
 
22
  instruction = (
23
  "You are a research assistant. The user wants a summary of ALL case studies provided. "
24
- "For EVERY case study in the excerpts below, write exactly 2-3 sentences covering: "
25
- "location, hazard type, and key outcome or lesson. "
26
- "Do not skip any case. Format as a numbered list with the case ID bolded. "
27
- "Be concise — do not write long paragraphs."
 
28
  )
29
  else:
30
  instruction = (
 
1
+ import re
2
+
3
+
4
+ _OVERVIEW_CONTEXT_CHARS = 1800
5
+
6
+
7
+ def _extract_label(text: str, label: str) -> str:
8
+ pattern = re.compile(
9
+ rf"\*\*{re.escape(label)}:\*\*\s*(.*?)(?=\n\n\*\*[^*\n]+:\*\*|\n---|\n#{1,6}\s|\Z)",
10
+ re.IGNORECASE | re.DOTALL,
11
+ )
12
+ match = pattern.search(text)
13
+ if not match:
14
+ return ""
15
+ return re.sub(r"\s+", " ", match.group(1)).strip()
16
+
17
+
18
+ def _compact_overview_text(text: str) -> str:
19
+ """Keep overview prompts small enough that all cases fit in the answer."""
20
+ fields = []
21
+ for label in ("Case Name and Location", "Hazard(s)", "Type of Retreat", "Key Outcomes"):
22
+ value = _extract_label(text, label)
23
+ if value:
24
+ fields.append(f"{label}: {value}")
25
+
26
+ compact = "\n".join(fields)
27
+ if not compact:
28
+ compact = re.sub(r"\s+", " ", text).strip()
29
+
30
+ if len(compact) > _OVERVIEW_CONTEXT_CHARS:
31
+ compact = compact[:_OVERVIEW_CONTEXT_CHARS].rsplit(" ", 1)[0] + " ..."
32
+ return compact
33
 
34
 
35
  def build_prompt(query: str, retrieved_chunks: list[dict], questionnaire_answers: dict | None = None) -> str:
36
+ is_overview = any(
37
+ kw in query.lower()
38
+ for kw in ("summar", "overview", "all case", "every case", "list", "compare", "across")
39
  )
40
 
41
+ if is_overview:
42
+ context_blocks = "\n\n".join(
43
+ f"[Source: {c['metadata'].get('source', 'unknown')} | Case: {c['metadata'].get('case_id', '?')}]\n"
44
+ f"{_compact_overview_text(c['text'])}"
45
+ for c in retrieved_chunks
46
+ )
47
+ else:
48
+ context_blocks = "\n\n".join(
49
+ f"[Source: {c['metadata'].get('source', 'unknown')} | Case: {c['metadata'].get('case_id', '?')}]\n{c['text']}"
50
+ for c in retrieved_chunks
51
+ )
52
+
53
  answers_section = ""
54
  if questionnaire_answers:
55
  answers_section = "\n\nUser context from questionnaire:\n" + "\n".join(
56
  f"- {k}: {v}" for k, v in questionnaire_answers.items()
57
  )
58
 
 
 
 
 
 
59
  if is_overview:
60
+ case_ids = ", ".join(c["metadata"].get("case_id", "?") for c in retrieved_chunks)
61
  instruction = (
62
  "You are a research assistant. The user wants a summary of ALL case studies provided. "
63
+ f"The provided cases are: {case_ids}. "
64
+ "Write exactly one numbered bullet for EVERY provided case, in the same order. "
65
+ "Each bullet must start with the bold case ID, then give one concise sentence covering "
66
+ "location, hazard or retreat type, and the key outcome or lesson. "
67
+ "Keep each bullet under 45 words. Do not skip cases. Do not invent missing case IDs."
68
  )
69
  else:
70
  instruction = (