Spaces:
Sleeping
Sleeping
|
Download docs/extraction_schema.md from annasbeef/RFR-Knowledge-Platform: direct link, hf CLI and curl.
- Browser
- Download file 2.86 kB
-
https://huggingface.co/spaces/annasbeef/RFR-Knowledge-Platform/resolve/main/docs/extraction_schema.md
- Command line
-
hf download hf://spaces/annasbeef/RFR-Knowledge-Platform/docs/extraction_schema.md
-
curl -L -o extraction_schema.md https://huggingface.co/spaces/annasbeef/RFR-Knowledge-Platform/resolve/main/docs/extraction_schema.md
2.86 kB
| # Current Extraction and Indexing Format | |
| The original project sketch used a compact structured extraction schema for a | |
| small pilot set of managed-retreat cases. The current RAG knowledge base uses | |
| long-form case-study memo sections generated by the LLM extraction/audit | |
| pipeline. | |
| ## Source Lineage | |
| Case-study candidates come from the Ajibade et al. spreadsheet: | |
| `https://docs.google.com/spreadsheets/d/1PGvxXlBUP-DFTuaYKhO5xSesHnPGhMlM/edit` | |
| Local case folders under `data/raw/` contain source PDFs, `case_meta.json`, and | |
| pipeline outputs. A case becomes queryable only after it has: | |
| 1. A canonical `data/raw/<CSxx>/pipeline_output/CSxx_Ver2.md`. | |
| 2. An extracted `data/extracted/CSxx.json`. | |
| 3. Fresh embeddings in `data/extracted/precomputed_embeddings.json`. | |
| ## Extracted Chunk Shape | |
| Each `data/extracted/CSxx.json` file is a list of section chunks: | |
| ```json | |
| { | |
| "case_id": "CS24", | |
| "chunk_index": 0, | |
| "section": "overview", | |
| "location": "Queens, New York", | |
| "country": "United States", | |
| "source": "Pipeline Ver2 — Queens", | |
| "text": "Case study: Queens ... Section: CASE STUDY OVERVIEW ..." | |
| } | |
| ``` | |
| Field meanings: | |
| | Field | Meaning | | |
| |-------|---------| | |
| | `case_id` | Case identifier such as `CS24`. | | |
| | `chunk_index` | Stable chunk number within that case. Overview-summary chunks use `999`. | | |
| | `section` | Normalized Ver2 memo section name, such as `overview`, `planning_processes`, or `community_engagement`. | | |
| | `location` | Location copied from `case_meta.json` when available. | | |
| | `country` | Country copied from `case_meta.json` when available. | | |
| | `source` | Human-readable source label for retrieved evidence. | | |
| | `text` | Self-contained chunk text sent to the retriever and LLM prompt. | | |
| `data/extracted/case_summaries.json` contains one `overview_summary` chunk per | |
| case when a usable overview section can be extracted. These chunks support | |
| overview and comparison queries. | |
| ## Refresh Commands | |
| Run these after adding or changing canonical Ver2 outputs: | |
| ```bash | |
| python3 ingest/ingest_pipeline_outputs.py | |
| python3 ingest/create_summary_chunks.py | |
| python3 ingest/embed_and_index.py | |
| ``` | |
| `embed_and_index.py` writes a JSON embedding store rather than ChromaDB. It | |
| checks chunk IDs, text, metadata, and a content hash so changed Ver2 content | |
| triggers a rebuild. | |
| ## Current State Checked July 14, 2026 | |
| - Canonical Ver2 cases: 34. | |
| - Extracted case JSON files: 34. | |
| - Embedded cases: 34. | |
| - Embedded chunks: 231. | |
| - Indexed cases: `CS1`-`CS31`, `CS33`, `CS34`, `CS35`. | |
| - `CS32` is present in the spreadsheet/local metadata but does not currently | |
| have a canonical Ver2 output or extracted JSON. | |
| - `case_summaries.json` has 34 overview-summary chunks. `CS11` uses the | |
| opening `Case Identification` / `General Information` sections as a fallback | |
| because its Ver2 document does not have a literal `Case Study Overview` | |
| heading. | |