RFR-Knowledge-Platform / docs /extraction_schema.md
Anna Zhou
Fix all-case summaries for Hugging Face Space
318732d
|
Raw History Blame Contribute Delete
2.86 kB
# Current Extraction and Indexing Format
The original project sketch used a compact structured extraction schema for a
small pilot set of managed-retreat cases. The current RAG knowledge base uses
long-form case-study memo sections generated by the LLM extraction/audit
pipeline.
## Source Lineage
Case-study candidates come from the Ajibade et al. spreadsheet:
`https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit`
Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and
pipeline outputs. A case becomes queryable only after it has:
1. A canonical `data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md`.
2. An extracted `data/extracted/CSxx.json`.
3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`.
## Extracted Chunk Shape
Each `data/extracted/CSxx.json` file is a list of section chunks:
```json
{
"case_id": "CS24",
"chunk_index": 0,
"section": "overview",
"location": "Queens, New York",
"country": "United States",
"source": "Pipeline Ver2 — Queens",
"text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..."
}
```
Field meanings:
| Field | Meaning |
|-------|---------|
| `case_id` | Case identifier such as `CS24`. |
| `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. |
| `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. |
| `location` | Location copied from `case_meta.json` when available. |
| `country` | Country copied from `case_meta.json` when available. |
| `source` | Human-readable source label for retrieved evidence. |
| `text` | Self-contained chunk text sent to the retriever and LLM prompt. |
`data/extracted/case_summaries.json` contains one `overview_summary` chunk per
case when a usable overview section can be extracted. These chunks support
overview and comparison queries.
## Refresh Commands
Run these after adding or changing canonical Ver2 outputs:
```bash
python3 ingest/ingest_pipeline_outputs.py
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```
`embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It
checks chunk IDs, text, metadata, and a content hash so changed Ver2 content
triggers a rebuild.
## Current State Checked July 14, 2026
- Canonical Ver2 cases: 34.
- Extracted case JSON files: 34.
- Embedded cases: 34.
- Embedded chunks: 231.
- Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`.
- `CS32` is present in the spreadsheet/local metadata but does not currently
have a canonical Ver2 output or extracted JSON.
- `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the
opening `Case Identification` / `General Information` sections as a fallback
because its Ver2 document does not have a literal `Case Study Overview`
heading.