Anna Zhou
Add case study request intake workflow
52c5c96
|
Raw History Blame Contribute Delete
12 kB
# Case Study Processing Pipeline
Automates the 4-step managed retreat case study workflow:
1. **Claude** generates a narrative case-study questionnaire document (**Version 1**)
from the RFR questionnaire, codebook, cross-cutting themes guide, note-taking
guidelines, and a folder of source documents for one case study.
2. **ChatGPT** audits Version 1 against the sources and codebook (**Check 1**),
producing a scored report (0-100).
3. **Claude** revises Version 1 into **Version 2** using the Check 1 feedback.
4. **ChatGPT** audits Version 2 in a fresh, standalone call with no memory of
Check 1 (**Check 2**), producing another scored report so you can compare
Check 1 vs. Check 2 to measure improvement.
The default workflow still uses Claude for drafting/revision and ChatGPT for
audits. For shared-account use, pass `--llm-provider claude` or
`--llm-provider openai` to run all four steps with one provider/key.
## Setup
1. Make sure `rfr-rag/.env` has the key or keys needed for your workflow:
```
ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...
```
For `--llm-provider claude`, only `ANTHROPIC_API_KEY` is required. For
`--llm-provider openai`, only `OPENAI_API_KEY` is required. The default
mixed workflow requires both.
2. Install dependencies (adds `fpdf2` to the existing requirements):
```bash
pip install -r requirements.txt
```
## Context documents
The `context/` folder contains the shared reference material used in every run:
- `ai_questionnaire.md` β€” the RFR coding questions
- `codebook.md` β€” the RFR codebook (categories, codes, descriptions)
- `cross_cutting_guide.md` β€” the RFR cross-cutting themes guide
- `note_taking_guidelines.md` β€” the RFR note-taking guidelines for LLMs
These were generated from the RFR Drive docs. If the team updates those docs,
re-export them (File > Download > Plain text for Google Docs, or re-export the
codebook spreadsheet) and overwrite the corresponding file here.
## Usage
For any case study, put its source documents (PDFs of papers/reports; `.txt`/`.md`
also supported) in their own folder, then run:
```bash
python -m case_study_pipeline.run_case_study --case-folder data/raw/CS6
```
This writes to `data/raw/CS6/pipeline_output/` by default:
- `CS6_Ver1.md` / `CS6_Ver1.pdf` β€” initial questionnaire document
- `CS6_Check1_report.md` / `.pdf` β€” Check 1 audit report + grade
- `CS6_Ver2.md` / `.pdf` β€” revised questionnaire document
- `CS6_Check2_report.md` / `.pdf` β€” Check 2 audit report + grade
A summary with both grades and the change is printed at the end.
### Useful options
```bash
python -m case_study_pipeline.run_case_study \
--case-folder data/raw/CS6 \
--case-id CS6 \ # defaults to the folder name
--output-dir outputs/CS6 \ # defaults to <case-folder>/pipeline_output/
--llm-provider claude \ # mixed, claude, or openai
--claude-model claude-sonnet-4-6 \
--openai-model gpt-5.4-mini \
--max-tokens 8000 \ # max output tokens per LLM call
--max-chars-per-source 120000 \ # truncate very long source files (0 = no limit)
--force # overwrite existing output files without asking
```
By default, if an output PDF already exists you'll be asked to confirm before it's
overwritten (the `.md` files are always (re)written so you don't lose the run, but
existing PDFs are kept unless you confirm or pass `--force`).
### Shared one-provider workflow
Use this when the team wants a simple process that only needs one LLM account.
The Hugging Face app can collect requests and uploaded papers without any API
calls; a project member then runs the extraction locally:
```bash
python3 -m case_study_pipeline.run_case_study \
--case-folder data/raw/CSxx-ShortName \
--case-id CSxx \
--llm-provider claude \
--max-chars-per-source 150000 \
--force
python3 -m case_study_pipeline.select_best_run --case-id CSxx
python3 ingest/ingest_pipeline_outputs.py --case-id CSxx
python3 ingest/create_summary_chunks.py
python3 ingest/embed_and_index.py
```
## Running just the audit step (Check 1 / Check 2 only)
Sometimes Version 1 or Version 2 of a case study's questionnaire document is written
by hand (e.g. Claude generated it directly in a chat, without calling the Anthropic
API). In that situation you don't need to re-run the whole pipeline -- you just need
ChatGPT/OpenAI to audit the existing document and produce a scored report.
```bash
# Audit data/raw/CS11/pipeline_output/CS11_Ver1.md
python -m case_study_pipeline.run_check1 --case-id CS11
# Audit data/raw/CS11/pipeline_output/CS11_Ver2.md (also compares against the
# Check 1 grade, if a CS11_Check1_report.md is already there)
python -m case_study_pipeline.run_check2 --case-id CS11
```
`--case-id` is the only required option -- it's how you tell the script which case
study to audit (e.g. `CS6`, `CS11`, ...). Both scripts accept the same `--case-folder`,
`--output-dir`, `--openai-model`, `--max-tokens`, and `--max-chars-per-source` options
as `run_case_study.py`. Each writes `<CASE_ID>_Check1_report.{md,pdf}` or
`<CASE_ID>_Check2_report.{md,pdf}` into the case study's `pipeline_output/` folder.
Requires `OPENAI_API_KEY` (in `rfr-rag/.env` or the environment) -- no Anthropic key
needed for these two scripts.
## High-precision paper discovery
Use `discover_sources.py` before running the extraction pipeline when a case
needs source cleanup, DOI lookup, or supplemental paper discovery.
```bash
python3 -m case_study_pipeline.discover_sources --case-id CS36
python3 -m case_study_pipeline.discover_sources \
--case-id CS36 \
--alias "Pitt County" \
--alias "Hurricane Floyd"
```
The discovery command is deliberately conservative:
- Existing `sources.txt` / `dois.txt` entries are treated as accepted seed
sources, but still resolved through scholarly metadata APIs where possible.
- New search results are not silently accepted. They are classified as
`review_required` or `rejected_by_identity_filter`.
- Direct case evidence must match the core place identity, such as the village,
city, resettlement name, province/state, and country from the spreadsheet or
`case_meta.json`.
- National/provincial/local legal and policy papers can be surfaced as
`candidate_policy_context`, but they stay separate from direct case evidence.
The command writes:
```text
data/raw/<CASE>/
β”œβ”€β”€ sources.json # structured accepted seed sources
β”œβ”€β”€ paper_candidates.json # all candidates with evidence/scoring
└── paper_discovery_report.md # human-readable review report
```
Recommended review rule: only use `accepted_seed` sources and manually approved
`review_required` candidates in the LLM extraction. Do not use rejected/background
papers unless a human reclassifies them after reading the full text.
## Automated source acquisition
After discovery, run `acquire_sources.py` to populate the case folder with source
files the extraction pipeline can read:
```bash
python3 -m case_study_pipeline.acquire_sources --case-id CS39
python3 -m case_study_pipeline.acquire_sources \
--case-id CS38 \
--extra-url "https://www.academia.edu/37237146/Heritage_and_Postdisaster_Recovery_Indigenous_Community_Resilience"
```
This command prefers publisher/open-access PDFs, then falls back to a
provenance-labelled `.md` capture when the PDF is blocked but the article/report
text is publicly reachable. That fallback is intentional: the extraction
pipeline reads `.pdf`, `.txt`, and `.md`, so a full-text markdown capture is
better than stopping the pipeline on a missing PDF. Use the generated
`source_acquisition_report.json` to see what was acquired.
If a publisher PDF is paywalled and no public full-text route is available, the
script will not bypass access controls. In that case, the non-manual route is to
add an authorized/open URL with `--extra-url`, or use the EZproxy/browser flow
below for institutional access.
## Session notes
`session_notes/` holds one-off "continue this work" notes written for specific case
studies (for example, a note describing how CS11's Version 1 document was produced
by hand). These are historical records, not part of the reusable pipeline -- the
scripts above work for any case study ID.
## Notes
- Source PDFs are read with `pdfplumber` (already a project dependency).
- Output PDFs are generated with `fpdf2` from the LLM's narrative text. Smart
quotes/dashes are converted to plain ASCII for compatibility with the built-in
PDF fonts.
- "No memory" for Check 2: each ChatGPT call in this pipeline is a single
standalone request (system + user message only, no prior turns or
`previous_response_id`). The Chat Completions API has no per-conversation memory
by default, so Check 2 is independent of Check 1 by construction β€” there's no
separate toggle to flip.
## Fetching paywalled papers via UWaterloo EZproxy (Chrome-assisted)
For papers that `fetch_papers.py` can't get through Unpaywall / Sci-Hub /
Semantic Scholar (i.e. paywalled, behind a major publisher), use UWaterloo's
institutional library access. **This cannot be fully scripted** β€” WatIAM
requires Duo two-factor authentication, which only a human can complete. The
workflow below uses the Claude-in-Chrome browser extension to drive an
already-logged-in browser session, with the actual login/Duo step done by
the user once per browser session (sessions last several hours).
### One-time setup (per machine)
In Chrome, go to `chrome://settings/downloads` and set:
- **Location:** `/Users/AnnaZhou/rfr-rag/data/raw/_inbox`
- **"Ask where to save each file before downloading"** β€” turn this **OFF**
This makes every PDF download save automatically into the repo's inbox
folder with no save dialog, so an agent driving the browser can trigger a
download and immediately find the file on disk (this folder is gitignored
via `data/raw/` in `.gitignore`).
### Recurring procedure (every session, every case study)
1. Run `fetch_papers.py` first β€” it handles every paper it can without
needing institutional access:
```
python3 -m case_study_pipeline.fetch_papers --case-id CS13
```
2. For DOIs that still fail, use UWaterloo EZproxy through Chrome:
- Navigate to `https://login.proxy.lib.uwaterloo.ca/login?qurl=<any target URL>`
and click "Using WatIAM". **Ask the user to enter credentials and
approve Duo themselves** β€” never enter passwords for them.
- Once logged in, for each remaining DOI:
a. Navigate to `https://doi.org/<DOI>` β€” this redirects to the
publisher's real article URL.
b. Convert that URL to its EZproxy form using the dash-domain rule
(see `case_study_pipeline/ezproxy.py`):
`link.springer.com` β†’ `link-springer-com.proxy.lib.uwaterloo.ca`
`agupubs.onlinelibrary.wiley.com` β†’ `agupubs-onlinelibrary-wiley-com.proxy.lib.uwaterloo.ca`
(any publisher domain follows this same pattern)
c. Navigate to the EZproxy URL, use `find` to locate the "Download PDF"
link, and click it. The PDF lands in `data/raw/_inbox/` automatically
(no dialog, because of the one-time Chrome setting above).
d. Check the inbox and file it away:
```
python3 -m case_study_pipeline.sort_inbox --list
python3 -m case_study_pipeline.sort_inbox \
--filename "<exact name in inbox>" \
--case-id CS13 \
--rename "Author_Year_Journal_FULLTEXT.pdf"
```
This moves the PDF into `data/raw/CS13/` and uploads it to the
correct Drive folder automatically (via `drive_folders.json` +
`drive_upload.py` β€” same OAuth token already set up for the
pipeline's own output uploads).
3. The EZproxy session stays alive in the browser for the rest of the
working session β€” no need to re-login for subsequent papers/case studies
unless the session expires (multi-hour timeout) or Chrome is restarted.