# Phase 3 Readiness Assessment — Retrieval **Date**: 2026-08-26 **Status**: Data/evaluation sources assessed. Retrieval pipeline **NOT built**. Phase 2 baseline **NOT measured** (blocked on `ANTHROPIC_API_KEY`). This document exists because the project's own execution policy requires Phase 3 to not be considered complete (or even started) until the Phase 2 baseline is actually measured, and requires any Phase 3 infrastructure prepared beforehand to be clearly labeled as preparation, not implementation. Nothing in this document should be read as "Phase 3 is underway" — it is a readiness check performed while Phase 2 sits idle on a credential. --- ## Why this exists now, before Phase 2 is measured `ANTHROPIC_API_KEY` is required for Phase 2 and is not available in this environment. Rather than block all progress, this session did the highest- value work that requires no API calls and does not depend on the baseline number: verifying the actual sources Phase 3 will need. This is explicitly scoped as **preparation**, per the project's standing instruction to "produce a Phase-3 data/evaluation readiness assessment" and "STOP at the Phase-3 implementation boundary." No retrieval code (BM25, dense embeddings, hybrid fusion, reranking) has been written. --- ## Source-by-source readiness | Source | License | Download | Schema | Verdict | |---|---|---|---|---| | CUAD | CC BY 4.0 (claimed on project site; no LICENSE file in repo) | ✓ Downloaded (18.3MB, SHA256 recorded) | ✓ Verified against real data | **READY** — already in use for Phase 2 | | ContractNLI | CC BY 4.0 (confirmed: LICENSE+TERMS shipped in archive) | ✓ Downloaded (65.4MB, SHA256 recorded) | ✓ Verified against real data (loader's original guess was correct) | **READY** — could supply evidence-span examples for Phase 5's verifier; not itself a retrieval benchmark | | MAUD | CC BY 4.0 (confirmed via HF dataset card tag) | ⚠️ Not downloaded (schema checked via API only) | ⚠️ Loader rewritten to match verified flat-CSV schema, not execution-tested | **DEFERRED** — optional per `docs/ROADMAP.md` (Phase 6+); no action taken this pass, consistent with not installing `datasets`/`pandas`/`pyarrow` for a dataset three phases away from relevance | | LegalBench-RAG | MIT (confirmed via GitHub API structured field) | ❌ Blocked — `dropbox.com` unreachable from this environment (SSLConnectError); `github.com` reachable in the same session, so this is a network restriction, not a broken or unofficial link (see ADR-020) | ⚠️ Plausible per README description, not verified against a real file | **BLOCKED** on environment network access, not on data/licensing | ## The actual Phase 3 blocker LegalBench-RAG is the dataset the project's own `docs/EVAL_PLAN.md` and `docs/ROADMAP.md` designate as the gold retrieval benchmark (Recall@5, Recall@10, nDCG@10). Its source and license are now verified, but the data itself cannot be downloaded from this sandbox because `dropbox.com` is unreachable here. This is a genuine hard blocker for Phase 3's *retrieval evaluation* specifically — not for Phase 3 preparation work in general, and not something to route around by fabricating a substitute dataset or silently swapping to a different (weaker) evaluation source, which the project's evaluation-methodology-change rule (any metric/dataset change requires justification + ADR + test + doc + comparison) does not permit doing quietly. **What this does NOT block**: building the retrieval mechanisms themselves (BM25, dense embeddings via BGE-M3, hybrid fusion, bge-reranker-v2-m3) can proceed without LegalBench-RAG, since those are local, open-weight components with no external data dependency. What genuinely needs LegalBench-RAG is the *scientific evaluation* of whether retrieval helps — Recall@k and nDCG@k require its gold query/snippet judgments. Building retrieval mechanics without a way to measure them would violate "never optimize before measuring" and "never add architecture merely because it sounds impressive" — so retrieval implementation is being held, not just the evaluation of it. ## What Phase 3 will need once unblocked From the retrieval architecture already specified in `docs/ARCHITECTURE.md` and `docs/DECISIONS.md` (ADR-005): BGE-M3 for dense embeddings and bge-reranker-v2-m3 for reranking, both open-weight and runnable on the available RTX 2000 Ada (16GB VRAM, confirmed present via `nvidia-smi` in the initial environment audit). `rank-bm25` is already listed in `requirements.txt` for the lexical baseline. None of these require an API key — only LegalBench-RAG's gold judgments (for measuring recall/nDCG) and CUAD (already local) as retrieval corpus/queries are external dependencies, and CUAD is already satisfied. ## Explicit non-claims - This document does NOT claim retrieval improves extraction quality. That is an empirical question Phase 3 exists to answer, and it cannot be answered without both a measured Phase 2 baseline (for the extraction-F1 delta) and LegalBench-RAG (for the retrieval-quality numbers). - This document does NOT claim LegalBench-RAG's schema is confirmed. The loader's field-name assumptions are plausible based on the README's prose description but have not been checked against an actual file. - This document does NOT constitute Phase 3 sign-off. Per the project's own phase gate, Phase 3 cannot begin in earnest until Phase 2's baseline is measured and found scientifically valid. ## Update 2026-08-26: Phase 2 baseline measured — STATUS = RED The paid-API blocker above no longer applies: `ANTHROPIC_API_KEY` was never obtained, and was made unnecessary by replacing Claude with a local, open-weight model (Qwen2.5-7B-Instruct via Ollama, $0 cost -- `docs/DECISIONS.md` ADR-022). The real 77-contract baseline was measured (twice: v3, then v4 after fixing a diagnosed JSON-escaping defect, ADR-024) and is **RED** against the pre-registered gate: Macro F1 0.117 vs. a classical TF-IDF baseline's 0.4532. Full detail in `docs/PHASE2_BASELINE.md` and `docs/ERROR_ANALYSIS_PHASE2.md`. Per this project's own phase-gate rule, **Phase 3 retrieval implementation remains gated** -- a RED Phase 2 does not license moving to Phase 3 optimization. However, the specific, evidence-based root cause identified (recall collapses with document length; the model doesn't scale its search effort to content, not a formatting or category- confusion problem) is direct, mechanistic motivation for exactly what Phase 3 proposes: pre-localizing relevant sections before extraction targets this exact failure mode. This readiness document's data-source findings (CUAD/ContractNLI ready, LegalBench-RAG blocked by this environment's network) stand unchanged; what changed is that there is now a concrete, measured reason to expect retrieval could help, rather than an assumption that it would. ## Update 2026-08-29: Retrieval recall measured WITHOUT LegalBench-RAG The LegalBench-RAG blocker above (dropbox.com unreachable) turned out not to block the actual retrieval-recall question for this project's own dataset: CUAD's own SQuAD-style annotations already carry gold `answer_start` character offsets (verified in `data/processed/cuad_test_contracts.json`), which is exactly what's needed to score "did retrieval find the right section" -- the same thing LegalBench-RAG's snippet character ranges would provide. `src/retrieval.py` implements a BM25 section retriever plus `compute_retrieval_recall()`, tested in `tests/test_retrieval.py` (including an explicit no-gold- leakage check on retrieval queries), and measured on the real 77-contract test set at **zero API cost, zero new external data**: Recall@1/3/5/10 = 0.327 / 0.548 / 0.646 / 0.749, MRR = 0.469 — far above the LLM's own effective micro-recall of 0.158 when given the whole document. Full detail, literature context, candidate architectures, and the resulting pilot experiment are in `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md`, which supersedes item 3 below as the live decision document. LegalBench-RAG itself remains un-downloaded (still blocked in this environment) and would still add value as an externally-standardized cross-dataset comparison, but is no longer a blocker for testing the retrieval hypothesis on this project's own data. ## Next gated actions (in order) 1. ~~Obtain `ANTHROPIC_API_KEY`~~ — superseded; local model used instead, $0 cost. 2. ~~Determine baseline scientific validity~~ — done: valid protocol, genuinely RED result, root-caused (not a suspicious ceiling/floor artifact -- the classical baseline on the identical protocol scored 0.4532, ruling out an evaluation-methodology explanation). 3. ~~Either run a larger local model or measure retrieval recall~~ — retrieval recall measured directly (above); see `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md` for the full analysis, literature comparison, and the resulting small pilot experiment (category-batched extraction with and without per-batch BM25 localization) run before committing to any further full-77-contract experiment. 4. Full Phase 3 implementation (BM25 → dense → hybrid → reranked retrieval evaluated end-to-end) remains contingent on `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md`'s final decision (GREEN/YELLOW/RED/STOP), which is itself contingent on the pilot's measured result -- not assumed in advance.