Spaces:
Sleeping
Sleeping
File size: 2,855 Bytes
318732d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | # Current Extraction and Indexing Format
The original project sketch used a compact structured extraction schema for a
small pilot set of managed-retreat cases. The current RAG knowledge base uses
long-form case-study memo sections generated by the LLM extraction/audit
pipeline.
## Source Lineage
Case-study candidates come from the Ajibade et al. spreadsheet:
`https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`
Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and
pipeline outputs. A case becomes queryable only after it has:
1. A canonical `data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md`.
2. An extracted `data/extracted/CSxx.json`.
3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`.
## Extracted Chunk Shape
Each `data/extracted/CSxx.json` file is a list of section chunks:
```json
{
"case_id": "CS24",
"chunk_index": 0,
"section": "overview",
"location": "Queens, New York",
"country": "United States",
"source": "Pipeline Ver2 — Queens",
"text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
}
```
Field meanings:
| Field | Meaning |
|-------|---------|
| `case_id` | Case identifier such as `CS24`. |
| `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. |
| `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. |
| `location` | Location copied from `case_meta.json` when available. |
| `country` | Country copied from `case_meta.json` when available. |
| `source` | Human-readable source label for retrieved evidence. |
| `text` | Self-contained chunk text sent to the retriever and LLM prompt. |
`data/extracted/case_summaries.json` contains one `overview_summary` chunk per
case when a usable overview section can be extracted. These chunks support
overview and comparison queries.
## Refresh Commands
Run these after adding or changing canonical Ver2 outputs:
```bash
python3 ingest/ingest_pipeline_outputs.py
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```
`embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It
checks chunk IDs, text, metadata, and a content hash so changed Ver2 content
triggers a rebuild.
## Current State Checked July 14, 2026
- Canonical Ver2 cases: 34.
- Extracted case JSON files: 34.
- Embedded cases: 34.
- Embedded chunks: 231.
- Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`.
- `CS32` is present in the spreadsheet/local metadata but does not currently
have a canonical Ver2 output or extracted JSON.
- `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the
opening `Case Identification` / `General Information` sections as a fallback
because its Ver2 document does not have a literal `Case Study Overview`
heading.
|