docs/PHASE3_READINESS.md

Phase 3 Readiness Assessment — Retrieval

Date: 2026-08-26 Status: Data/evaluation sources assessed. Retrieval pipeline NOT built. Phase 2 baseline NOT measured (blocked on ANTHROPIC_API_KEY).

This document exists because the project's own execution policy requires Phase 3 to not be considered complete (or even started) until the Phase 2 baseline is actually measured, and requires any Phase 3 infrastructure prepared beforehand to be clearly labeled as preparation, not implementation. Nothing in this document should be read as "Phase 3 is underway" — it is a readiness check performed while Phase 2 sits idle on a credential.

---

Why this exists now, before Phase 2 is measured

ANTHROPIC_API_KEY is required for Phase 2 and is not available in this environment. Rather than block all progress, this session did the highest- value work that requires no API calls and does not depend on the baseline number: verifying the actual sources Phase 3 will need. This is explicitly scoped as preparation, per the project's standing instruction to "produce a Phase-3 data/evaluation readiness assessment" and "STOP at the Phase-3 implementation boundary." No retrieval code (BM25, dense embeddings, hybrid fusion, reranking) has been written.

---

Source-by-source readiness

Source	License	Download	Schema	Verdict
CUAD	CC BY 4.0 (claimed on project site; no LICENSE file in repo)	✓ Downloaded (18.3MB, SHA256 recorded)	✓ Verified against real data	READY — already in use for Phase 2
ContractNLI	CC BY 4.0 (confirmed: LICENSE+TERMS shipped in archive)	✓ Downloaded (65.4MB, SHA256 recorded)	✓ Verified against real data (loader's original guess was correct)	READY — could supply evidence-span examples for Phase 5's verifier; not itself a retrieval benchmark
MAUD	CC BY 4.0 (confirmed via HF dataset card tag)	⚠️ Not downloaded (schema checked via API only)	⚠️ Loader rewritten to match verified flat-CSV schema, not execution-tested	DEFERRED — optional per docs/ROADMAP.md (Phase 6+); no action taken this pass, consistent with not installing datasets/pandas/pyarrow for a dataset three phases away from relevance
LegalBench-RAG	MIT (confirmed via GitHub API structured field)	❌ Blocked — dropbox.com unreachable from this environment (SSLConnectError); github.com reachable in the same session, so this is a network restriction, not a broken or unofficial link (see ADR-020)	⚠️ Plausible per README description, not verified against a real file	BLOCKED on environment network access, not on data/licensing

The actual Phase 3 blocker

LegalBench-RAG is the dataset the project's own docs/EVAL_PLAN.md and docs/ROADMAP.md designate as the gold retrieval benchmark (Recall@5, Recall@10, nDCG@10). Its source and license are now verified, but the data itself cannot be downloaded from this sandbox because dropbox.com is unreachable here. This is a genuine hard blocker for Phase 3's retrieval evaluation specifically — not for Phase 3 preparation work in general, and not something to route around by fabricating a substitute dataset or silently swapping to a different (weaker) evaluation source, which the project's evaluation-methodology-change rule (any metric/dataset change requires justification + ADR + test + doc + comparison) does not permit doing quietly.

What this does NOT block: building the retrieval mechanisms themselves (BM25, dense embeddings via BGE-M3, hybrid fusion, bge-reranker-v2-m3) can proceed without LegalBench-RAG, since those are local, open-weight components with no external data dependency. What genuinely needs LegalBench-RAG is the scientific evaluation of whether retrieval helps — Recall@k and nDCG@k require its gold query/snippet judgments. Building retrieval mechanics without a way to measure them would violate "never optimize before measuring" and "never add architecture merely because it sounds impressive" — so retrieval implementation is being held, not just the evaluation of it.

What Phase 3 will need once unblocked

From the retrieval architecture already specified in docs/ARCHITECTURE.md and docs/DECISIONS.md (ADR-005): BGE-M3 for dense embeddings and bge-reranker-v2-m3 for reranking, both open-weight and runnable on the available RTX 2000 Ada (16GB VRAM, confirmed present via nvidia-smi in the initial environment audit). rank-bm25 is already listed in requirements.txt for the lexical baseline. None of these require an API key — only LegalBench-RAG's gold judgments (for measuring recall/nDCG) and CUAD (already local) as retrieval corpus/queries are external dependencies, and CUAD is already satisfied.

Explicit non-claims

• This document does NOT claim retrieval improves extraction quality. That is an empirical question Phase 3 exists to answer, and it cannot be answered without both a measured Phase 2 baseline (for the extraction-F1 delta) and LegalBench-RAG (for the retrieval-quality numbers).
• This document does NOT claim LegalBench-RAG's schema is confirmed. The loader's field-name assumptions are plausible based on the README's prose description but have not been checked against an actual file.
• This document does NOT constitute Phase 3 sign-off. Per the project's own phase gate, Phase 3 cannot begin in earnest until Phase 2's baseline is measured and found scientifically valid.

Update 2026-08-26: Phase 2 baseline measured — STATUS = RED

The paid-API blocker above no longer applies: ANTHROPIC_API_KEY was never obtained, and was made unnecessary by replacing Claude with a local, open-weight model (Qwen2.5-7B-Instruct via Ollama, $0 cost -- docs/DECISIONS.md ADR-022). The real 77-contract baseline was measured (twice: v3, then v4 after fixing a diagnosed JSON-escaping defect, ADR-024) and is RED against the pre-registered gate: Macro F1 0.117 vs. a classical TF-IDF baseline's 0.4532. Full detail in docs/PHASE2_BASELINE.md and docs/ERROR_ANALYSIS_PHASE2.md.

Per this project's own phase-gate rule, Phase 3 retrieval implementation remains gated -- a RED Phase 2 does not license moving to Phase 3 optimization. However, the specific, evidence-based root cause identified (recall collapses with document length; the model doesn't scale its search effort to content, not a formatting or category- confusion problem) is direct, mechanistic motivation for exactly what Phase 3 proposes: pre-localizing relevant sections before extraction targets this exact failure mode. This readiness document's data-source findings (CUAD/ContractNLI ready, LegalBench-RAG blocked by this environment's network) stand unchanged; what changed is that there is now a concrete, measured reason to expect retrieval could help, rather than an assumption that it would.

Update 2026-08-29: Retrieval recall measured WITHOUT LegalBench-RAG

The LegalBench-RAG blocker above (dropbox.com unreachable) turned out not to block the actual retrieval-recall question for this project's own dataset: CUAD's own SQuAD-style annotations already carry gold answer_start character offsets (verified in data/processed/cuad_test_contracts.json), which is exactly what's needed to score "did retrieval find the right section" -- the same thing LegalBench-RAG's snippet character ranges would provide. src/retrieval.py implements a BM25 section retriever plus compute_retrieval_recall(), tested in tests/test_retrieval.py (including an explicit no-gold- leakage check on retrieval queries), and measured on the real 77-contract test set at zero API cost, zero new external data:

Recall@1/3/5/10 = 0.327 / 0.548 / 0.646 / 0.749, MRR = 0.469 — far above the LLM's own effective micro-recall of 0.158 when given the whole document. Full detail, literature context, candidate architectures, and the resulting pilot experiment are in docs/RESEARCH_AND_ARCHITECTURE_DECISION.md, which supersedes item 3 below as the live decision document. LegalBench-RAG itself remains un-downloaded (still blocked in this environment) and would still add value as an externally-standardized cross-dataset comparison, but is no longer a blocker for testing the retrieval hypothesis on this project's own data.

Next gated actions (in order)

1. Obtain ANTHROPIC_API_KEY — superseded; local model used instead, $0 cost.
2. Determine baseline scientific validity — done: valid protocol, genuinely RED result, root-caused (not a suspicious ceiling/floor artifact -- the classical baseline on the identical protocol scored 0.4532, ruling out an evaluation-methodology explanation).
3. Either run a larger local model or measure retrieval recall — retrieval recall measured directly (above); see docs/RESEARCH_AND_ARCHITECTURE_DECISION.md for the full analysis, literature comparison, and the resulting small pilot experiment (category-batched extraction with and without per-batch BM25 localization) run before committing to any further full-77-contract experiment.
4. Full Phase 3 implementation (BM25 → dense → hybrid → reranked retrieval evaluated end-to-end) remains contingent on docs/RESEARCH_AND_ARCHITECTURE_DECISION.md's final decision (GREEN/YELLOW/RED/STOP), which is itself contingent on the pilot's measured result -- not assumed in advance.