docs/PHASE3_READINESS.md

# Phase 3 Readiness Assessment — Retrieval

**Date**: 2026-08-26
**Status**: Data/evaluation sources assessed. Retrieval pipeline **NOT built**. Phase 2 baseline **NOT measured** (blocked on `ANTHROPIC_API_KEY`).

This document exists because the project's own execution policy requires
Phase 3 to not be considered complete (or even started) until the Phase 2
baseline is actually measured, and requires any Phase 3 infrastructure
prepared beforehand to be clearly labeled as preparation, not
implementation. Nothing in this document should be read as "Phase 3 is
underway" — it is a readiness check performed while Phase 2 sits idle on a
credential.

---

## Why this exists now, before Phase 2 is measured

`ANTHROPIC_API_KEY` is required for Phase 2 and is not available in this
environment. Rather than block all progress, this session did the highest-
value work that requires no API calls and does not depend on the baseline
number: verifying the actual sources Phase 3 will need. This is explicitly
scoped as **preparation**, per the project's standing instruction to
"produce a Phase-3 data/evaluation readiness assessment" and "STOP at the
Phase-3 implementation boundary." No retrieval code (BM25, dense
embeddings, hybrid fusion, reranking) has been written.

---

## Source-by-source readiness

| Source | License | Download | Schema | Verdict |
|---|---|---|---|---|
| CUAD | CC BY 4.0 (claimed on project site; no LICENSE file in repo) | ✓ Downloaded (18.3MB, SHA256 recorded) | ✓ Verified against real data | **READY** — already in use for Phase 2 |
| ContractNLI | CC BY 4.0 (confirmed: LICENSE+TERMS shipped in archive) | ✓ Downloaded (65.4MB, SHA256 recorded) | ✓ Verified against real data (loader's original guess was correct) | **READY** — could supply evidence-span examples for Phase 5's verifier; not itself a retrieval benchmark |
| MAUD | CC BY 4.0 (confirmed via HF dataset card tag) | ⚠️ Not downloaded (schema checked via API only) | ⚠️ Loader rewritten to match verified flat-CSV schema, not execution-tested | **DEFERRED** — optional per `docs/ROADMAP.md` (Phase 6+); no action taken this pass, consistent with not installing `datasets`/`pandas`/`pyarrow` for a dataset three phases away from relevance |
| LegalBench-RAG | MIT (confirmed via GitHub API structured field) | ❌ Blocked — `dropbox.com` unreachable from this environment (SSLConnectError); `github.com` reachable in the same session, so this is a network restriction, not a broken or unofficial link (see ADR-020) | ⚠️ Plausible per README description, not verified against a real file | **BLOCKED** on environment network access, not on data/licensing |

## The actual Phase 3 blocker

LegalBench-RAG is the dataset the project's own `docs/EVAL_PLAN.md` and
`docs/ROADMAP.md` designate as the gold retrieval benchmark (Recall@5,
Recall@10, nDCG@10). Its source and license are now verified, but the data
itself cannot be downloaded from this sandbox because `dropbox.com` is
unreachable here. This is a genuine hard blocker for Phase 3's *retrieval
evaluation* specifically — not for Phase 3 preparation work in general, and
not something to route around by fabricating a substitute dataset or
silently swapping to a different (weaker) evaluation source, which the
project's evaluation-methodology-change rule (any metric/dataset change
requires justification + ADR + test + doc + comparison) does not permit
doing quietly.

**What this does NOT block**: building the retrieval mechanisms themselves
(BM25, dense embeddings via BGE-M3, hybrid fusion, bge-reranker-v2-m3) can
proceed without LegalBench-RAG, since those are local, open-weight
components with no external data dependency. What genuinely needs
LegalBench-RAG is the *scientific evaluation* of whether retrieval helps —
Recall@k and nDCG@k require its gold query/snippet judgments. Building
retrieval mechanics without a way to measure them would violate "never
optimize before measuring" and "never add architecture merely because it
sounds impressive" — so retrieval implementation is being held, not just
the evaluation of it.

## What Phase 3 will need once unblocked

From the retrieval architecture already specified in `docs/ARCHITECTURE.md`
and `docs/DECISIONS.md` (ADR-005): BGE-M3 for dense embeddings and
bge-reranker-v2-m3 for reranking, both open-weight and runnable on the
available RTX 2000 Ada (16GB VRAM, confirmed present via `nvidia-smi` in
the initial environment audit). `rank-bm25` is already listed in
`requirements.txt` for the lexical baseline. None of these require an API
key — only LegalBench-RAG's gold judgments (for measuring recall/nDCG) and
CUAD (already local) as retrieval corpus/queries are external dependencies,
and CUAD is already satisfied.

## Explicit non-claims

- This document does NOT claim retrieval improves extraction quality. That
  is an empirical question Phase 3 exists to answer, and it cannot be
  answered without both a measured Phase 2 baseline (for the extraction-F1
  delta) and LegalBench-RAG (for the retrieval-quality numbers).
- This document does NOT claim LegalBench-RAG's schema is confirmed. The
  loader's field-name assumptions are plausible based on the README's prose
  description but have not been checked against an actual file.
- This document does NOT constitute Phase 3 sign-off. Per the project's own
  phase gate, Phase 3 cannot begin in earnest until Phase 2's baseline is
  measured and found scientifically valid.

## Update 2026-08-26: Phase 2 baseline measured — STATUS = RED

The paid-API blocker above no longer applies: `ANTHROPIC_API_KEY` was
never obtained, and was made unnecessary by replacing Claude with a
local, open-weight model (Qwen2.5-7B-Instruct via Ollama, $0 cost --
`docs/DECISIONS.md` ADR-022). The real 77-contract baseline was measured
(twice: v3, then v4 after fixing a diagnosed JSON-escaping defect,
ADR-024) and is **RED** against the pre-registered gate: Macro F1 0.117
vs. a classical TF-IDF baseline's 0.4532. Full detail in
`docs/PHASE2_BASELINE.md` and `docs/ERROR_ANALYSIS_PHASE2.md`.

Per this project's own phase-gate rule, **Phase 3 retrieval
implementation remains gated** -- a RED Phase 2 does not license moving to
Phase 3 optimization. However, the specific, evidence-based root cause
identified (recall collapses with document length; the model doesn't
scale its search effort to content, not a formatting or category-
confusion problem) is direct, mechanistic motivation for exactly what
Phase 3 proposes: pre-localizing relevant sections before extraction
targets this exact failure mode. This readiness document's
data-source findings (CUAD/ContractNLI ready, LegalBench-RAG blocked by
this environment's network) stand unchanged; what changed is that there
is now a concrete, measured reason to expect retrieval could help, rather
than an assumption that it would.

## Update 2026-08-29: Retrieval recall measured WITHOUT LegalBench-RAG

The LegalBench-RAG blocker above (dropbox.com unreachable) turned out not
to block the actual retrieval-recall question for this project's own
dataset: CUAD's own SQuAD-style annotations already carry gold
`answer_start` character offsets (verified in
`data/processed/cuad_test_contracts.json`), which is exactly what's
needed to score "did retrieval find the right section" -- the same thing
LegalBench-RAG's snippet character ranges would provide. `src/retrieval.py`
implements a BM25 section retriever plus `compute_retrieval_recall()`,
tested in `tests/test_retrieval.py` (including an explicit no-gold-
leakage check on retrieval queries), and measured on the real 77-contract
test set at **zero API cost, zero new external data**:

Recall@1/3/5/10 = 0.327 / 0.548 / 0.646 / 0.749, MRR = 0.469 — far above
the LLM's own effective micro-recall of 0.158 when given the whole
document. Full detail, literature context, candidate architectures, and
the resulting pilot experiment are in
`docs/RESEARCH_AND_ARCHITECTURE_DECISION.md`, which supersedes item 3
below as the live decision document. LegalBench-RAG itself remains
un-downloaded (still blocked in this environment) and would still add
value as an externally-standardized cross-dataset comparison, but is no
longer a blocker for testing the retrieval hypothesis on this project's
own data.

## Next gated actions (in order)

1. ~~Obtain `ANTHROPIC_API_KEY`~~ — superseded; local model used instead, $0 cost.
2. ~~Determine baseline scientific validity~~ — done: valid protocol,
   genuinely RED result, root-caused (not a suspicious ceiling/floor
   artifact -- the classical baseline on the identical protocol scored
   0.4532, ruling out an evaluation-methodology explanation).
3. ~~Either run a larger local model or measure retrieval recall~~ —
   retrieval recall measured directly (above); see
   `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md` for the full analysis,
   literature comparison, and the resulting small pilot experiment
   (category-batched extraction with and without per-batch BM25
   localization) run before committing to any further full-77-contract
   experiment.
4. Full Phase 3 implementation (BM25 → dense → hybrid → reranked
   retrieval evaluated end-to-end) remains contingent on
   `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md`'s final decision
   (GREEN/YELLOW/RED/STOP), which is itself contingent on the pilot's
   measured result -- not assumed in advance.