# Current Extraction and Indexing Format The original project sketch used a compact structured extraction schema for a small pilot set of managed-retreat cases. The current RAG knowledge base uses long-form case-study memo sections generated by the LLM extraction/audit pipeline. ## Source Lineage Case-study candidates come from the Ajibade et al. spreadsheet: `https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit` Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and pipeline outputs. A case becomes queryable only after it has: 1. A canonical `data/raw//pipeline_output/CSxx_Ver2.md`. 2. An extracted `data/extracted/CSxx.json`. 3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`. ## Extracted Chunk Shape Each `data/extracted/CSxx.json` file is a list of section chunks: ```json { "case_id": "CS24", "chunk_index": 0, "section": "overview", "location": "Queens, New York", "country": "United States", "source": "Pipeline Ver2 — Queens", "text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..." } ``` Field meanings: | Field | Meaning | |-------|---------| | `case_id` | Case identifier such as `CS24`. | | `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. | | `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. | | `location` | Location copied from `case_meta.json` when available. | | `country` | Country copied from `case_meta.json` when available. | | `source` | Human-readable source label for retrieved evidence. | | `text` | Self-contained chunk text sent to the retriever and LLM prompt. | `data/extracted/case_summaries.json` contains one `overview_summary` chunk per case when a usable overview section can be extracted. These chunks support overview and comparison queries. ## Refresh Commands Run these after adding or changing canonical Ver2 outputs: ```bash python3 ingest/ingest_pipeline_outputs.py python3 ingest/create_summary_chunks.py python3 ingest/embed_and_index.py ``` `embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It checks chunk IDs, text, metadata, and a content hash so changed Ver2 content triggers a rebuild. ## Current State Checked July 14, 2026 - Canonical Ver2 cases: 34. - Extracted case JSON files: 34. - Embedded cases: 34. - Embedded chunks: 231. - Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`. - `CS32` is present in the spreadsheet/local metadata but does not currently have a canonical Ver2 output or extracted JSON. - `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the opening `Case Identification` / `General Information` sections as a fallback because its Ver2 document does not have a literal `Case Study Overview` heading.