Spaces:
Running
Running
research: BM25 retrieval-recall measurement + category-batched extraction (Phase 3 hypothesis test)
4f47f31 | # Phase 3 Readiness Assessment — Retrieval | |
| **Date**: 2026-08-26 | |
| **Status**: Data/evaluation sources assessed. Retrieval pipeline **NOT built**. Phase 2 baseline **NOT measured** (blocked on `ANTHROPIC_API_KEY`). | |
| This document exists because the project's own execution policy requires | |
| Phase 3 to not be considered complete (or even started) until the Phase 2 | |
| baseline is actually measured, and requires any Phase 3 infrastructure | |
| prepared beforehand to be clearly labeled as preparation, not | |
| implementation. Nothing in this document should be read as "Phase 3 is | |
| underway" — it is a readiness check performed while Phase 2 sits idle on a | |
| credential. | |
| --- | |
| ## Why this exists now, before Phase 2 is measured | |
| `ANTHROPIC_API_KEY` is required for Phase 2 and is not available in this | |
| environment. Rather than block all progress, this session did the highest- | |
| value work that requires no API calls and does not depend on the baseline | |
| number: verifying the actual sources Phase 3 will need. This is explicitly | |
| scoped as **preparation**, per the project's standing instruction to | |
| "produce a Phase-3 data/evaluation readiness assessment" and "STOP at the | |
| Phase-3 implementation boundary." No retrieval code (BM25, dense | |
| embeddings, hybrid fusion, reranking) has been written. | |
| --- | |
| ## Source-by-source readiness | |
| | Source | License | Download | Schema | Verdict | | |
| |---|---|---|---|---| | |
| | CUAD | CC BY 4.0 (claimed on project site; no LICENSE file in repo) | ✓ Downloaded (18.3MB, SHA256 recorded) | ✓ Verified against real data | **READY** — already in use for Phase 2 | | |
| | ContractNLI | CC BY 4.0 (confirmed: LICENSE+TERMS shipped in archive) | ✓ Downloaded (65.4MB, SHA256 recorded) | ✓ Verified against real data (loader's original guess was correct) | **READY** — could supply evidence-span examples for Phase 5's verifier; not itself a retrieval benchmark | | |
| | MAUD | CC BY 4.0 (confirmed via HF dataset card tag) | ⚠️ Not downloaded (schema checked via API only) | ⚠️ Loader rewritten to match verified flat-CSV schema, not execution-tested | **DEFERRED** — optional per `docs/ROADMAP.md` (Phase 6+); no action taken this pass, consistent with not installing `datasets`/`pandas`/`pyarrow` for a dataset three phases away from relevance | | |
| | LegalBench-RAG | MIT (confirmed via GitHub API structured field) | ❌ Blocked — `dropbox.com` unreachable from this environment (SSLConnectError); `github.com` reachable in the same session, so this is a network restriction, not a broken or unofficial link (see ADR-020) | ⚠️ Plausible per README description, not verified against a real file | **BLOCKED** on environment network access, not on data/licensing | | |
| ## The actual Phase 3 blocker | |
| LegalBench-RAG is the dataset the project's own `docs/EVAL_PLAN.md` and | |
| `docs/ROADMAP.md` designate as the gold retrieval benchmark (Recall@5, | |
| Recall@10, nDCG@10). Its source and license are now verified, but the data | |
| itself cannot be downloaded from this sandbox because `dropbox.com` is | |
| unreachable here. This is a genuine hard blocker for Phase 3's *retrieval | |
| evaluation* specifically — not for Phase 3 preparation work in general, and | |
| not something to route around by fabricating a substitute dataset or | |
| silently swapping to a different (weaker) evaluation source, which the | |
| project's evaluation-methodology-change rule (any metric/dataset change | |
| requires justification + ADR + test + doc + comparison) does not permit | |
| doing quietly. | |
| **What this does NOT block**: building the retrieval mechanisms themselves | |
| (BM25, dense embeddings via BGE-M3, hybrid fusion, bge-reranker-v2-m3) can | |
| proceed without LegalBench-RAG, since those are local, open-weight | |
| components with no external data dependency. What genuinely needs | |
| LegalBench-RAG is the *scientific evaluation* of whether retrieval helps — | |
| Recall@k and nDCG@k require its gold query/snippet judgments. Building | |
| retrieval mechanics without a way to measure them would violate "never | |
| optimize before measuring" and "never add architecture merely because it | |
| sounds impressive" — so retrieval implementation is being held, not just | |
| the evaluation of it. | |
| ## What Phase 3 will need once unblocked | |
| From the retrieval architecture already specified in `docs/ARCHITECTURE.md` | |
| and `docs/DECISIONS.md` (ADR-005): BGE-M3 for dense embeddings and | |
| bge-reranker-v2-m3 for reranking, both open-weight and runnable on the | |
| available RTX 2000 Ada (16GB VRAM, confirmed present via `nvidia-smi` in | |
| the initial environment audit). `rank-bm25` is already listed in | |
| `requirements.txt` for the lexical baseline. None of these require an API | |
| key — only LegalBench-RAG's gold judgments (for measuring recall/nDCG) and | |
| CUAD (already local) as retrieval corpus/queries are external dependencies, | |
| and CUAD is already satisfied. | |
| ## Explicit non-claims | |
| - This document does NOT claim retrieval improves extraction quality. That | |
| is an empirical question Phase 3 exists to answer, and it cannot be | |
| answered without both a measured Phase 2 baseline (for the extraction-F1 | |
| delta) and LegalBench-RAG (for the retrieval-quality numbers). | |
| - This document does NOT claim LegalBench-RAG's schema is confirmed. The | |
| loader's field-name assumptions are plausible based on the README's prose | |
| description but have not been checked against an actual file. | |
| - This document does NOT constitute Phase 3 sign-off. Per the project's own | |
| phase gate, Phase 3 cannot begin in earnest until Phase 2's baseline is | |
| measured and found scientifically valid. | |
| ## Update 2026-08-26: Phase 2 baseline measured — STATUS = RED | |
| The paid-API blocker above no longer applies: `ANTHROPIC_API_KEY` was | |
| never obtained, and was made unnecessary by replacing Claude with a | |
| local, open-weight model (Qwen2.5-7B-Instruct via Ollama, $0 cost -- | |
| `docs/DECISIONS.md` ADR-022). The real 77-contract baseline was measured | |
| (twice: v3, then v4 after fixing a diagnosed JSON-escaping defect, | |
| ADR-024) and is **RED** against the pre-registered gate: Macro F1 0.117 | |
| vs. a classical TF-IDF baseline's 0.4532. Full detail in | |
| `docs/PHASE2_BASELINE.md` and `docs/ERROR_ANALYSIS_PHASE2.md`. | |
| Per this project's own phase-gate rule, **Phase 3 retrieval | |
| implementation remains gated** -- a RED Phase 2 does not license moving to | |
| Phase 3 optimization. However, the specific, evidence-based root cause | |
| identified (recall collapses with document length; the model doesn't | |
| scale its search effort to content, not a formatting or category- | |
| confusion problem) is direct, mechanistic motivation for exactly what | |
| Phase 3 proposes: pre-localizing relevant sections before extraction | |
| targets this exact failure mode. This readiness document's | |
| data-source findings (CUAD/ContractNLI ready, LegalBench-RAG blocked by | |
| this environment's network) stand unchanged; what changed is that there | |
| is now a concrete, measured reason to expect retrieval could help, rather | |
| than an assumption that it would. | |
| ## Update 2026-08-29: Retrieval recall measured WITHOUT LegalBench-RAG | |
| The LegalBench-RAG blocker above (dropbox.com unreachable) turned out not | |
| to block the actual retrieval-recall question for this project's own | |
| dataset: CUAD's own SQuAD-style annotations already carry gold | |
| `answer_start` character offsets (verified in | |
| `data/processed/cuad_test_contracts.json`), which is exactly what's | |
| needed to score "did retrieval find the right section" -- the same thing | |
| LegalBench-RAG's snippet character ranges would provide. `src/retrieval.py` | |
| implements a BM25 section retriever plus `compute_retrieval_recall()`, | |
| tested in `tests/test_retrieval.py` (including an explicit no-gold- | |
| leakage check on retrieval queries), and measured on the real 77-contract | |
| test set at **zero API cost, zero new external data**: | |
| Recall@1/3/5/10 = 0.327 / 0.548 / 0.646 / 0.749, MRR = 0.469 — far above | |
| the LLM's own effective micro-recall of 0.158 when given the whole | |
| document. Full detail, literature context, candidate architectures, and | |
| the resulting pilot experiment are in | |
| `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md`, which supersedes item 3 | |
| below as the live decision document. LegalBench-RAG itself remains | |
| un-downloaded (still blocked in this environment) and would still add | |
| value as an externally-standardized cross-dataset comparison, but is no | |
| longer a blocker for testing the retrieval hypothesis on this project's | |
| own data. | |
| ## Next gated actions (in order) | |
| 1. ~~Obtain `ANTHROPIC_API_KEY`~~ — superseded; local model used instead, $0 cost. | |
| 2. ~~Determine baseline scientific validity~~ — done: valid protocol, | |
| genuinely RED result, root-caused (not a suspicious ceiling/floor | |
| artifact -- the classical baseline on the identical protocol scored | |
| 0.4532, ruling out an evaluation-methodology explanation). | |
| 3. ~~Either run a larger local model or measure retrieval recall~~ — | |
| retrieval recall measured directly (above); see | |
| `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md` for the full analysis, | |
| literature comparison, and the resulting small pilot experiment | |
| (category-batched extraction with and without per-batch BM25 | |
| localization) run before committing to any further full-77-contract | |
| experiment. | |
| 4. Full Phase 3 implementation (BM25 → dense → hybrid → reranked | |
| retrieval evaluated end-to-end) remains contingent on | |
| `docs/RESEARCH_AND_ARCHITECTURE_DECISION.md`'s final decision | |
| (GREEN/YELLOW/RED/STOP), which is itself contingent on the pilot's | |
| measured result -- not assumed in advance. | |