project_proposal/proposal.md

# Candle-Fire: ALS Research Landscape Tool

## Project Initiative

Candle-fire is a physician-facing ALS research intelligence platform — a sibling to beacon. Where beacon helps patients find clinical trials, candle-fire helps physicians understand the *research evidence behind* those trials and ALS biology more broadly.

**The problem**: A physician encounters a clinical trial for an antisense oligonucleotide targeting SOD1. To evaluate it, they need to know: What is the evidence base for SOD1 as a target? What are the known mechanisms? What else has been tried? Today this requires hours of manual literature review.

**The solution**: A physician types a free-text question ("What's the evidence for tofersen targeting SOD1 in ALS?") and gets a synthesized, cited answer grounded in ~500 curated ALS papers, enriched by a knowledge graph linking genes, proteins, compounds, pathways, and clinical trials — delivered in under 30 seconds.

**End state**: A Gradio web app deployable on HuggingFace Spaces. Two layers of intelligence: (1) a vector RAG layer for semantic paper retrieval, (2) a knowledge graph layer for entity-level relationship traversal. Claude Sonnet synthesizes both into a structured research landscape report with citations, mechanism summaries, and related trial links.

---

## Critique of Original Plan

1. **RAG alone is semantically blind.** Query for "riluzole" must surface "glutamate excitotoxicity" via the KG (riluzole → INHIBITS → glutamate excitotoxicity). KG expansion *precedes* RAG retrieval — not optional.

2. **Entity normalization is a prerequisite for KG construction.** TDP-43 / TARDBP / TDP43 must resolve to one canonical node before graph build. The normalizer runs as part of extraction and writes `canonical_ids.json` as the single source of truth.

3. **All heavy compute is offline batch.** Ingestion, extraction, KG build, RAG indexing — run once. Query time: NetworkX pickle (read) + ChromaDB (read) + Anthropic API only.

4. **Citation counts weight evidence quality.** A paper cited 500 times is stronger evidence than one cited 5 times. Fetch citation counts from Semantic Scholar API (free, no auth). Use to re-rank RAG results and weight KG edge confidence.

5. **PMC XML full text, not raw PDFs.** ~50-60% of recent ALS papers are PMC Open Access. Ingest structured XML (section-labeled: Introduction, Methods, Results, Discussion) via PubMed Entrez. Always fall back to abstract. Avoid PDF parsing — too brittle and legally ambiguous for paywalled papers.

---

## Architecture

### Tech Stack
- **Language**: Python 3.11+, uv package manager
- **LLM**: Claude Sonnet 4-6 (entity extraction + research synthesis)
- **Vector store**: ChromaDB (SQLite-backed, persistent, no separate service)
- **Graph**: NetworkX DiGraph (pickled, loaded once at startup, scales to 20K+ papers)
- **Embeddings**: `all-MiniLM-L6-v2` via sentence-transformers (local, no API needed)
- **UI**: Gradio 6 (HuggingFace Spaces deployment, mirrors beacon)
- **Paper source**: PubMed Entrez API + PMC XML full text (hybrid)
- **Citation counts**: Semantic Scholar API (free, by DOI/PMID)

### Data Flow

```
OFFLINE PIPELINE (run once, in order):

  Stage 1: Ingestion
    scripts/ingest_papers.py
      → ingestion/pubmed.py (PubMed Entrez, batch 200 PMIDs, abstract always)
      → ingestion/pmc.py (PMC XML full text for OA papers, ~50-60% coverage)
      → ingestion/semantic_scholar.py (citation counts per PMID/DOI)
      → data/papers/papers.jsonl  [ALSPaper: title, abstract, full_text?, citation_count, ...]

    scripts/ingest_trials.py  [parallel with above]
      → ingestion/clinicaltrials.py (ClinicalTrials.gov v2, condition=ALS)
      → data/trials/trials.jsonl

  Stage 2: Entity Extraction
    scripts/extract_entities.py
      → extraction/extractor.py (Claude Sonnet, 10 papers/call, retry+backoff, resumable)
      → extraction/normalizer.py (HGNC alias table + PubChem + HGNC REST fallback)
      → data/extracted/entities.jsonl
      → data/extracted/canonical_ids.json

  Stage 3: Knowledge Graph Build
    scripts/build_graph.py
      → graph/builder.py (NetworkX DiGraph, upsert nodes+edges, weight by citation_count)
      → data/graph/als_graph.pkl + als_graph.json

  Stage 4: RAG Index Build
    scripts/build_index.py
      → rag/indexer.py (ChromaDB "als_papers", chunk by section if full text, else abstract)
      → data/chroma/

ONLINE QUERY PIPELINE (per physician request):

  Physician free-text question
    → agents/research_agent.py
        1. Claude: extract query entities → ["SOD1", "antisense oligonucleotide"]
        2. graph/query.py: 1-hop KG expansion → ["SOD1", "TARDBP", "tofersen", "RNA splicing", ...]
        3. rag/retriever.py: semantic search + entity-filtered search → top 15 papers
           (re-ranked by: ChromaDB distance × log(citation_count + 1))
        4. graph/query.py: find_trials_for_target() for each query entity
        5. Claude Sonnet (streaming): synthesize ResearchLandscape
    → Gradio UI (streaming response with citations)
```

---

## File Structure

```
candle-fire/
├── pyproject.toml
├── .env.example             # ANTHROPIC_API_KEY, ENTREZ_EMAIL, NCBI_API_KEY
├── CLAUDE.md                # Architecture guide (module responsibilities)
│
├── config.py                # Model names, paths, ALS seed entities, API endpoints
├── models.py                # Dataclasses: ALSPaper, ExtractedEntity, EntityRelationship, ResearchLandscape
├── prompts.py               # System prompts: extraction + synthesis
├── tools.py                 # Tool schema loader (mirrors beacon/tools.py)
├── llm.py                   # LLM provider abstraction (copied from beacon/llm.py)
├── logging_config.py        # Structured JSON logging (adapted from beacon/beacon_logging.py)
├── app.py                   # Gradio UI entry point (graph + ChromaDB loaded at startup)
├── main.py                  # CLI entry point (Rich console)
│
├── ingestion/
│   ├── pubmed.py            # PubMed Entrez: fetch abstracts + metadata by query or PMID list
│   ├── pmc.py               # PMC XML full text: fetch & parse structured sections for OA papers
│   ├── clinicaltrials.py    # ClinicalTrials.gov v2 (adapted from beacon/trials_api.py, no geo)
│   └── semantic_scholar.py  # Citation counts by PMID/DOI (batch API, free tier)
│
├── extraction/
│   ├── extractor.py         # Claude Sonnet NER: batch 10 papers, retry+backoff, .progress.json
│   └── normalizer.py        # Canonical ID mapping (HGNC alias table + PubChem + REST fallback)
│
├── graph/
│   ├── builder.py           # NetworkX DiGraph: upsert nodes+edges, seed ALS entities, citation weighting
│   ├── query.py             # Traversal: expand_query_entities, find_trials_for_target, get_entity_evidence
│   └── serializer.py        # Save/load: pickle (fast) + JSON (human-readable)
│
├── rag/
│   ├── indexer.py           # ChromaDB collection builder: section-aware chunking, citation_count metadata
│   └── retriever.py         # search(), search_by_entities(), citation-weighted re-ranking
│
├── agents/
│   └── research_agent.py    # Multi-step synthesis agent (streaming, mirrors beacon/agents/research.py)
│
├── scripts/
│   ├── ingest_papers.py     # CLI: PubMed + PMC XML + citation counts → papers.jsonl
│   ├── ingest_trials.py     # CLI: ClinicalTrials.gov → trials.jsonl
│   ├── extract_entities.py  # CLI: papers.jsonl → entities.jsonl (resumable)
│   ├── build_graph.py       # CLI: entities.jsonl + trials.jsonl → als_graph.pkl
│   └── build_index.py       # CLI: papers.jsonl + entities.jsonl → data/chroma/
│
├── data/
│   ├── papers/papers.jsonl
│   ├── trials/trials.jsonl
│   ├── extracted/
│   │   ├── entities.jsonl
│   │   ├── canonical_ids.json
│   │   └── .progress.json   # Extraction resumability tracker
│   ├── graph/
│   │   ├── als_graph.pkl
│   │   └── als_graph.json
│   ├── chroma/              # ChromaDB SQLite store
│   └── tools/
│       ├── extract_entities.json
│       └── search_landscape.json
│
└── tests/
    ├── test_pubmed.py
    ├── test_pmc.py
    ├── test_semantic_scholar.py
    ├── test_extractor.py
    ├── test_normalizer.py
    ├── test_graph_builder.py
    ├── test_graph_query.py
    ├── test_retriever.py
    └── test_research_agent.py
```

---

## Staged Implementation Plan

### Stage 1 — Project Foundation
**Goal**: Runnable skeleton with all dependencies wired.

- `pyproject.toml` with all dependencies (anthropic, gradio, chromadb, networkx, biopython, httpx, sentence-transformers, rich, python-dotenv)
- `config.py`, `models.py`, `prompts.py` (stubs), `tools.py`, `llm.py` (copied from beacon), `logging_config.py`
- `data/tools/extract_entities.json` and `search_landscape.json` tool schemas
- All `__init__.py` files, `.env.example`, `CLAUDE.md`

**Done when**: `uv run python -c "import anthropic, chromadb, networkx, Bio"` passes with no errors.

---

### Stage 2 — Paper Ingestion Pipeline
**Goal**: Populate `data/papers/papers.jsonl` with ~500 ALS papers including citation counts.

- `ingestion/pubmed.py`: PubMed Entrez client (fetch by MeSH query or PMID file, batch 200)
- `ingestion/pmc.py`: PMC XML full-text fetcher for OA papers (parse by section)
- `ingestion/semantic_scholar.py`: Citation count enrichment (batch by PMID)
- `scripts/ingest_papers.py`: Orchestrates all three, writes `papers.jsonl`
- `scripts/ingest_trials.py` + `ingestion/clinicaltrials.py`: ALS trials → `trials.jsonl`

PubMed seed query: `"amyotrophic lateral sclerosis"[MeSH Major Topic] AND ("2018"[PDAT]:"2024"[PDAT]) AND hasabstract[text]`

**Done when**: `papers.jsonl` has 500 lines with `citation_count` populated; `trials.jsonl` has 20+ records.

---

### Stage 3 — RAG Pipeline + Working v0
**Goal**: End-to-end working query pipeline using RAG only (no KG yet).

- `rag/indexer.py`: ChromaDB collection builder (section-aware chunking, citation_count in metadata)
- `rag/retriever.py`: `search()`, `search_by_entities()`, citation-weighted re-ranking
- `scripts/build_index.py`
- `agents/research_agent.py` (RAG-only version, no KG expansion step yet)
- `prompts.py` synthesis prompt
- `main.py`: CLI interface

**Done when**: `uv run python main.py "What compounds target glutamate excitotoxicity in ALS?"` returns a synthesized answer with cited PMIDs.

---

### Stage 4 — Entity Extraction + Knowledge Graph
**Goal**: Offline pipeline produces a populated KG; query agent upgrades to KG+RAG.

- `extraction/extractor.py`: Claude Sonnet NER, 10 papers/call, resumable via `.progress.json`
- `extraction/normalizer.py`: HGNC alias table (~50 ALS genes) + PubChem fallback + REST fallback
- `scripts/extract_entities.py`
- `graph/builder.py`: NetworkX DiGraph with upsert, citation-weighted edge confidence, seed entities
- `graph/query.py`: `expand_query_entities()`, `find_trials_for_target()`, `get_entity_evidence()`
- `graph/serializer.py`
- `scripts/build_graph.py`
- Upgrade `agents/research_agent.py` to include KG expansion step before RAG

**Done when**: `G.number_of_nodes() > 100`; query for "tofersen" surfaces SOD1 as the mechanism link (not just literal tofersen matches).

---

### Stage 5 — Gradio UI + Deployment Polish
**Goal**: Browser-accessible web app ready for HuggingFace Spaces.

- `app.py`: Gradio UI with streaming, example questions sidebar, citation display, disclaimer
- Module-level graph + ChromaDB client initialization (once at startup)
- `requirements.txt` (HF Spaces mirror of pyproject.toml)
- `README.md`: setup instructions, offline pipeline run order, env vars
- Tests: `uv run pytest` all pass

**Done when**: `uv run gradio app.py` → physician asks "What's the mechanism of tofersen in ALS?" → streaming response includes SOD1 mechanism, 3+ cited papers, at least 1 NCT trial link, and a disclaimer.

---

## Key Reuse from Beacon

| Beacon file | Candle-fire file | Change |
|---|---|---|
| `beacon/llm.py` | `llm.py` | Copy verbatim; model constants from `config.py` |
| `beacon/beacon_logging.py` | `logging_config.py` | Namespace → `candle_fire` |
| `beacon/trials_api.py` | `ingestion/clinicaltrials.py` | Remove geo/distance; add `extract_target_entities()` |
| `beacon/tools.py` | `tools.py` | Copy pattern; update tool names |
| `beacon/agents/research.py` | `agents/research_agent.py` | Adapt streaming loop; replace tool handlers |
| `beacon/app.py` | `app.py` | Single-turn instead of multi-turn state machine |

---

## Estimated Costs

| Item | Cost |
|---|---|
| Entity extraction (500 papers, 50 Sonnet calls ~10K tokens each) | ~$1.50 one-time |
| Semantic Scholar citation counts | Free |
| PMC XML full text | Free |
| Per physician query (Sonnet, ~15K tokens in+out) | ~$0.05/query |