UWaterlooManagedRetreat / docs /extraction_schema.md
Anna Zhou
Fix all-case summaries for Hugging Face Space
318732d
|
Raw History Blame Contribute Delete
2.86 kB

A newer version of the Streamlit SDK is available: 1.67.0

Upgrade

Current Extraction and Indexing Format

The original project sketch used a compact structured extraction schema for a small pilot set of managed-retreat cases. The current RAG knowledge base uses long-form case-study memo sections generated by the LLM extraction/audit pipeline.

Source Lineage

Case-study candidates come from the Ajibade et al. spreadsheet:

https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit

Local case folders under data/raw/ contain source PDFs, case_meta.json, and pipeline outputs. A case becomes queryable only after it has:

  1. A canonical data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md.
  2. An extracted data/extracted/CSxx.json.
  3. Fresh embeddings in data/extracted/precomputed_embeddings.json.

Extracted Chunk Shape

Each data/extracted/CSxx.json file is a list of section chunks:

{
  "case_id": "CS24",
  "chunk_index": 0,
  "section": "overview",
  "location": "Queens, New York",
  "country": "United States",
  "source": "Pipeline Ver2 — Queens",
  "text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
}

Field meanings:

Field Meaning
case_id Case identifier such as CS24.
chunk_index Stable chunk number within that case. Overview-summary chunks use 999.
section Normalized Ver2 memo section name, such as overview, planning_processes, or community_engagement.
location Location copied from case_meta.json when available.
country Country copied from case_meta.json when available.
source Human-readable source label for retrieved evidence.
text Self-contained chunk text sent to the retriever and LLM prompt.

data/extracted/case_summaries.json contains one overview_summary chunk per case when a usable overview section can be extracted. These chunks support overview and comparison queries.

Refresh Commands

Run these after adding or changing canonical Ver2 outputs:

python3 ingest/ingest_pipeline_outputs.py
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py

embed_and_index.py writes a JSON embedding store rather than ChromaDB. It checks chunk IDs, text, metadata, and a content hash so changed Ver2 content triggers a rebuild.

Current State Checked July 14, 2026

  • Canonical Ver2 cases: 34.
  • Extracted case JSON files: 34.
  • Embedded cases: 34.
  • Embedded chunks: 231.
  • Indexed cases: CS1-CS31, CS33, CS34, CS35.
  • CS32 is present in the spreadsheet/local metadata but does not currently have a canonical Ver2 output or extracted JSON.
  • case_summaries.json has 34 overview-summary chunks. CS11 uses the opening Case Identification / General Information sections as a fallback because its Ver2 document does not have a literal Case Study Overview heading.