Download docs/extraction_schema.md from UWaterlooRFR/UWaterlooManagedRetreat: direct link, hf CLI and curl.
- Browser
- Download file 2.86 kB
-
https://huggingface.co/spaces/UWaterlooRFR/UWaterlooManagedRetreat/resolve/main/docs/extraction_schema.md
- Command line
-
hf download hf://spaces/UWaterlooRFR/UWaterlooManagedRetreat/docs/extraction_schema.md
-
curl -L -o extraction_schema.md https://huggingface.co/spaces/UWaterlooRFR/UWaterlooManagedRetreat/resolve/main/docs/extraction_schema.md
A newer version of the Streamlit SDK is available: 1.67.0
Current Extraction and Indexing Format
The original project sketch used a compact structured extraction schema for a small pilot set of managed-retreat cases. The current RAG knowledge base uses long-form case-study memo sections generated by the LLM extraction/audit pipeline.
Source Lineage
Case-study candidates come from the Ajibade et al. spreadsheet:
https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit
Local case folders under data/raw/ contain source PDFs, case_meta.json, and
pipeline outputs. A case becomes queryable only after it has:
- A canonical
data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md. - An extracted
data/extracted/CSxx.json. - Fresh embeddings in
data/extracted/precomputed_embeddings.json.
Extracted Chunk Shape
Each data/extracted/CSxx.json file is a list of section chunks:
{
"case_id": "CS24",
"chunk_index": 0,
"section": "overview",
"location": "Queens, New York",
"country": "United States",
"source": "Pipeline Ver2 — Queens",
"text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
}
Field meanings:
| Field | Meaning |
|---|---|
case_id |
Case identifier such as CS24. |
chunk_index |
Stable chunk number within that case. Overview-summary chunks use 999. |
section |
Normalized Ver2 memo section name, such as overview, planning_processes, or community_engagement. |
location |
Location copied from case_meta.json when available. |
country |
Country copied from case_meta.json when available. |
source |
Human-readable source label for retrieved evidence. |
text |
Self-contained chunk text sent to the retriever and LLM prompt. |
data/extracted/case_summaries.json contains one overview_summary chunk per
case when a usable overview section can be extracted. These chunks support
overview and comparison queries.
Refresh Commands
Run these after adding or changing canonical Ver2 outputs:
python3 ingest/ingest_pipeline_outputs.py
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
embed_and_index.py writes a JSON embedding store rather than ChromaDB. It
checks chunk IDs, text, metadata, and a content hash so changed Ver2 content
triggers a rebuild.
Current State Checked July 14, 2026
- Canonical Ver2 cases: 34.
- Extracted case JSON files: 34.
- Embedded cases: 34.
- Embedded chunks: 231.
- Indexed cases:
CS1-CS31,CS33,CS34,CS35. CS32is present in the spreadsheet/local metadata but does not currently have a canonical Ver2 output or extracted JSON.case_summaries.jsonhas 34 overview-summary chunks.CS11uses the openingCase Identification/General Informationsections as a fallback because its Ver2 document does not have a literalCase Study Overviewheading.