Spaces:
Running
Running
Commit History
-
research: BM25 retrieval-recall measurement + category-batched extraction (Phase 3 hypothesis test)
Before committing further GPU time to either a bigger local model or a full Phase 3 retrieval build, this measures two things directly: 1. src/retrieval.py: BM25 section retriever + compute_retrieval_recall(), scored against CUAD's own gold answer_start offsets -- no LegalBench-RAG download needed (still blocked in this environment). Real result on the full 77-contract test set: Recall@10=0.749, MRR=0.469, vs. the LLM's own effective recall of 0.158 -- retrieval is not the bottleneck the LLM itself is. Also measured: localizing by unioning top-k sections across all 41 categories barely compresses long documents (ratio 0.70-0.89), motivating per-batch-scoped retrieval instead (tested in the pilot below). 2. src/baseline.py: added extract_clauses_batched() (category batching, isolates whether one-shot 41-category enumeration is itself the bottleneck -- directly motivated by ContractEval 2025 showing similar-size open models score far higher on a per-que…4f47f31
-
experiment: measure real v4 baseline (RED); root-cause the remaining gap
REAL MEASURED RESULT, v4 prompt (schema-constrained JSON), full 77-contract test set: Macro F1: 0.117 (up from v3's 0.077, but still far below the classical baseline's 0.4532) Micro F1: 0.259 | Micro precision/recall: 0.720 / 0.158 TP=152, FP=59, FN=811 76/77 contracts (98.7%) succeeded with zero errors -- JSON decode errors and hallucinated categories both dropped to ZERO (confirmed fix), only 1 edge case remains (no JSON array found) 229 total clauses extracted (mean 2.97/contract vs. real gold mean 27.1/contract) Mean confidence 0.965, Brier score 0.243, Spearman rho 0.304 Runtime 14,695.9s (~4h) for 77 contracts Per the pre-registered gate: STATUS = RED, unchanged from v3. Root-caused the remaining gap (not just re-measured and stopped): correlated real contract length against gold category count and clauses actually found across all 77 contracts. Pearson r(length, gold)=0.685 (longer contracts genuinely have more to find) vs. r(length, found)=0.093…a04c5e3 -
docs: Phase 3 data/evaluation readiness assessment
Consolidates this session's dataset verification findings into an explicit readiness check, per the standing instruction to produce this before any Phase 3 implementation and to stop at the Phase 3 implementation boundary. States plainly: CUAD and ContractNLI are ready (downloaded, schema-verified); MAUD is deferred (optional per docs/ROADMAP.md, Phase 6+, not worth installing datasets/pandas for yet); LegalBench-RAG's source and MIT license are verified but the actual download is blocked by dropbox.com being unreachable from this environment (a network restriction, not a licensing or dataset problem). No retrieval code has been written. Phase 2 remains the gating blocker for Phase 3 proper, per the project's own phase-gate policy.45539a4