File size: 2,855 Bytes
318732d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
# Current Extraction and Indexing Format

The original project sketch used a compact structured extraction schema for a
small pilot set of managed-retreat cases. The current RAG knowledge base uses
long-form case-study memo sections generated by the LLM extraction/audit
pipeline.

## Source Lineage

Case-study candidates come from the Ajibade et al. spreadsheet:

`https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`

Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and
pipeline outputs. A case becomes queryable only after it has:

1. A canonical `data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md`.
2. An extracted `data/extracted/CSxx.json`.
3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`.

## Extracted Chunk Shape

Each `data/extracted/CSxx.json` file is a list of section chunks:

```json
{
  "case_id": "CS24",
  "chunk_index": 0,
  "section": "overview",
  "location": "Queens, New York",
  "country": "United States",
  "source": "Pipeline Ver2 — Queens",
  "text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
}
```

Field meanings:

| Field | Meaning |
|-------|---------|
| `case_id` | Case identifier such as `CS24`. |
| `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. |
| `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. |
| `location` | Location copied from `case_meta.json` when available. |
| `country` | Country copied from `case_meta.json` when available. |
| `source` | Human-readable source label for retrieved evidence. |
| `text` | Self-contained chunk text sent to the retriever and LLM prompt. |

`data/extracted/case_summaries.json` contains one `overview_summary` chunk per
case when a usable overview section can be extracted. These chunks support
overview and comparison queries.

## Refresh Commands

Run these after adding or changing canonical Ver2 outputs:

```bash
python3 ingest/ingest_pipeline_outputs.py
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```

`embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It
checks chunk IDs, text, metadata, and a content hash so changed Ver2 content
triggers a rebuild.

## Current State Checked July 14, 2026

- Canonical Ver2 cases: 34.
- Extracted case JSON files: 34.
- Embedded cases: 34.
- Embedded chunks: 231.
- Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`.
- `CS32` is present in the spreadsheet/local metadata but does not currently
  have a canonical Ver2 output or extracted JSON.
- `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the
  opening `Case Identification` / `General Information` sections as a fallback
  because its Ver2 document does not have a literal `Case Study Overview`
  heading.