Spaces:
Running
Running
Commit History
-
research: record Phase 3 pilot results (H beats G beats A); launch full v5 retrieval-batched experiment
Pilot (docs/RESEARCH_AND_ARCHITECTURE_DECISION.md Section 9) measured a monotonic A < G < H improvement on macro F1, micro F1, and recall across 9 real test contracts -- category batching and per-batch BM25 retrieval each add real, additive value to local-LLM extraction. Decision is YELLOW, not GREEN: H's best pilot macro F1 (0.2128) is still well below the classical baseline (0.4532) measured on the full 77-contract set, so the full-scale run (scripts/run_v5_full_experiment.py, already running) is the next and deciding step, per the pre-registered experimental plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>343aa03 -
research: BM25 retrieval-recall measurement + category-batched extraction (Phase 3 hypothesis test)
Before committing further GPU time to either a bigger local model or a full Phase 3 retrieval build, this measures two things directly: 1. src/retrieval.py: BM25 section retriever + compute_retrieval_recall(), scored against CUAD's own gold answer_start offsets -- no LegalBench-RAG download needed (still blocked in this environment). Real result on the full 77-contract test set: Recall@10=0.749, MRR=0.469, vs. the LLM's own effective recall of 0.158 -- retrieval is not the bottleneck the LLM itself is. Also measured: localizing by unioning top-k sections across all 41 categories barely compresses long documents (ratio 0.70-0.89), motivating per-batch-scoped retrieval instead (tested in the pilot below). 2. src/baseline.py: added extract_clauses_batched() (category batching, isolates whether one-shot 41-category enumeration is itself the bottleneck -- directly motivated by ContractEval 2025 showing similar-size open models score far higher on a per-que…4f47f31
-
test: add Phase 2 pipeline sanity checks and a real-load smoke test
Stage 10: automated coverage for what's practically testable in code without a running model server -- invalid/hallucinated category names are rejected (not silently accepted into presence scoring), empty/ malformed responses are recorded as errors rather than silently swallowed, duplicate clauses from overlapping chunk regions are deduplicated, a long contract is actually split into multiple provider calls and merged back into one result (exercised end-to-end with a stubbed provider, not just chunking.py in isolation), and the real test split on disk is confirmed to be exactly 77 contracts, disjoint from train. Properties that are procedural claims about how an experiment is run (no test labels used for model selection, no gold spans in prompts) are documented rather than faked as unit tests -- see docs/PHASE2_BASELINE.md. scripts/smoke_test_local_model.py: Stage 4's required smoke test. Loads the real provider, checks real server/model availability, and runs one real generation aga…b9442ea -
experiment: add classical TF-IDF baseline script
Stage 12: "Do not assume an LLM is automatically superior." Fits TF-IDF (20K features, 1-2 grams, min_df=2, fit on train text ONLY) + One-vs-Rest balanced Logistic Regression on the real 356-contract train split, scores against the real 77-contract test split using the identical BaselineEvaluator.compute_f1() protocol the local-LLM baseline uses -- same (contract, category) presence granularity, same zero-support exclusion, no protocol changes between the two comparisons. Actually run against real data this session -- see docs/PHASE2_BASELINE.md for the measured numbers (data/processed/classical_baseline_results.json is gitignored like the rest of data/processed/, so the real result is recorded in the committed doc, not only in the local, regenerable artifact).68f2a2d -
refactor: switch baseline pipeline to local provider; drop Claude requirement
src/baseline.py::BaselineExtractor now takes a ModelProvider instead of constructing an Anthropic client directly, and defaults to get_provider() (local Ollama). PROMPT_VERSION bumped to v3 (provider-agnostic prompt, chunking-aware system prompt noting the text may be a partial excerpt). extract_clauses() now: chunks contracts exceeding the provider's context via src/chunking.py, extracts each chunk independently, and merges results with exact-duplicate removal across chunk overlaps. _parse_response now rejects hallucinated/invalid category names outright (previously these could slip through) and handles empty responses explicitly. scripts/run_baseline.py: replaced the ANTHROPIC_API_KEY presence check with provider.is_available() (is Ollama actually running and is the model actually pulled) -- the equivalent readiness gate for a local setup. Report template now shows "$0 -- local inference, no paid API used" instead of a deferred cost calculation, records the actual GPU name (nvidia-s…ee02d8b
-
fix: render calibration metrics; stamp real provenance into reports
Two gaps found while verifying the eval report actually contains measured values rather than placeholders: 1. compute_confidence_calibration() was already being called and its result stashed in metrics["calibration"], but generate_eval_report() never rendered it into the markdown -- computed values that never reached the output. Now shown in a dedicated Confidence Calibration section. 2. Model name, seed, and split fractions were hardcoded literal strings in the report templates, independent of what actually ran. Added _git_commit_hash() and _split_provenance() helpers; reports now read the real values from extraction_results (model, prompt_version, temperature -- set by src/baseline.py's extract_baseline()) and from data/processed/cuad_splits.json's own recorded seed/fractions, so a report can't silently drift out of sync with the code/data that produced it. Also fixed a crash the evaluator's new (contract, category) presence granularity would have ca…f5ec49d -
fix: persist full contract data across splits; make run_baseline usable
prepare_datasets.py previously wrote only contract IDs to cuad_splits.json. That's enough for leakage checks but not enough to actually run extraction -- there was no way to reconstruct a test contract's text/clauses from an ID alone without re-querying the (broken) loader. Now also writes cuad_{train,val,test}_contracts.json with full text + gold clauses per split. run_baseline.py had two defects that made it non-functional even with data and credentials present: 1. load_test_contracts() was a stub that always returned [] regardless of whether data existed (a known issue flagged before this change). 2. generate_eval_report() wrote a hardcoded "NOT MEASURED" markdown template unconditionally -- it never read its `results` argument, so even a fully successful extraction run would still have produced a report claiming nothing was measured. Fixed both: load_test_contracts() now reads the persisted cuad_test_contracts.json; generate_eval_report() takes an optional `metrics` d…cec37e6 -
phase-2-baseline: Baseline extraction architecture and scripts
- src/baseline.py: Zero-shot extraction using Claude 3.5 Sonnet * No retrieval, no agents, no optimization * Structured output (Pydantic) with validation * Deterministic (temperature=0) * Per-contract extraction + confidence scores - src/evaluator.py: Baseline evaluation metrics * Macro F1 (primary): average across all categories * Micro F1 (secondary): global metric * Per-category F1: diagnostic breakdown * Confidence calibration (Brier score, Spearman rho) - scripts/run_baseline.py: End-to-end baseline execution * Loads test contracts from Phase 1 splits * Calls Claude API for extraction * Computes all metrics * Generates evaluation report - docs/EVAL_REPORT.md: Phase 0-2 progress tracker * Current status: Phase 2 ready but NOT YET MEASURED * All metrics explicitly marked as NOT MEASURED * No fabrication, no assumptions * Clear blocking factors Status: Phase 2 complete (scaffolding). Ready to execute once: 1. ANTHROPIC_API_KEY is set (requires user…31c05a4 -
phase-1-data-pipeline: Dataset loaders, splitting, and leakage tests
- data/loaders.py: Unified loaders for CUAD, ContractNLI, MAUD, LegalBench-RAG * Load from official sources only (verified licenses) * Support for all major legal datasets * Contract-level data access (critical for split integrity) - data/split_utils.py: Contract-level train/val/test splitting * Guarantees zero leakage (no contract in multiple splits) * Reproducible with fixed seeds * Dataset statistics generation - tests/test_data_leakage.py: Critical leakage prevention tests * test_no_contract_leakage: MUST PASS before evaluation * test_no_duplicate_text_across_splits: Catch identical contracts * test_split_fractions: Verify split distribution * test_reproducibility_with_seed: Ensure deterministic splits - src/schemas.py: Pydantic models for validation * ClauseExtraction: Clause with category, confidence, source span * RiskFlag: Risk with verification status and citation * ContractExtractionResult: Complete extraction for a contract * EvaluationMetrics:…9c26600