Spaces:
Running
Running
Commit History
-
docs: add project defense document closing out the Phase 3 research cycle
Written for a skeptical reviewer: directly addresses the objections this project's own negative results invite (modest classical-baseline F1, local-model choice, retrieval "failing," pivoting away from LLM-only extraction, GPU-hours spent on failed hypotheses, overfitting/cherry- picking concerns, and "this isn't a finished product") with evidence already recorded in this repo -- leakage tests, fixed split/seed, pre-registered stop conditions, and the Candidate 4 probe. Also states plainly what it does not claim, matching this project's own never-fabricate-conclusions rule. Linked from README.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>2db8b77 -
research: Candidate 4 pre-build validation probe (LLM explanation/citation quality) -- YELLOW, conditional go
Before building the full Candidate 4 hybrid (ADR-026: classical model decides presence, LLM only explains/cites its positives), a small, self-contained probe (validation/candidate4_probe/) checked whether an LLM can actually do that narrowed job well. 24 real (contract, category) items, classical decision held fixed, BM25-localized excerpts (not full documents) via the existing src/retrieval.py retriever. Result: 100% citation verbatim accuracy (16/16, zero hallucinated spans) and 93.8% explanation accuracy (15/16) on items where a citation was produced; the other 8 were honest declines (5 correct classical-FP recognitions, 3 real retrieval misses), not fabrications. Not a clean GREEN: the probe's generation step used the frontier model running this session, not the project's actual $0/local production model (Qwen2.5-7B-Instruct) -- an explicit, stated constraint of this specific probe, and an upper-bound limitation on these numbers given this project's own prior finding that the sam…596c106
-
research: full-77-contract v5 result is RED vs classical baseline; pivot to classical-primary hybrid (ADR-026)
Full-scale measurement (scripts/run_v5_full_experiment.py, all 77 real test contracts) reverses the pilot's G-vs-H ranking: category batching alone nearly doubles v4's macro F1 (0.117 -> 0.2619), but per-batch BM25 retrieval on top of batching does not improve on that at full scale (0.2569) -- a smaller 9-contract pilot had suggested the opposite. Neither configuration beats the classical TF-IDF+LogReg baseline (0.4532; both land around 58% of it). Per the pre-registered stop condition in docs/RESEARCH_AND_ARCHITECTURE_DECISION.md: three independent LLM-side interventions (schema-constraining, batching, retrieval) have now been tried and none closes the gap. Final decision: RED for further standalone-LLM extraction iteration; pivot production architecture to Candidate 4 (classical model as primary presence decision-maker, LLM narrowed to source-grounded explanation/citation on classical positives). Recorded as ADR-026. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>0850b0e -
research: record Phase 3 pilot results (H beats G beats A); launch full v5 retrieval-batched experiment
Pilot (docs/RESEARCH_AND_ARCHITECTURE_DECISION.md Section 9) measured a monotonic A < G < H improvement on macro F1, micro F1, and recall across 9 real test contracts -- category batching and per-batch BM25 retrieval each add real, additive value to local-LLM extraction. Decision is YELLOW, not GREEN: H's best pilot macro F1 (0.2128) is still well below the classical baseline (0.4532) measured on the full 77-contract set, so the full-scale run (scripts/run_v5_full_experiment.py, already running) is the next and deciding step, per the pre-registered experimental plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>343aa03 -
research: BM25 retrieval-recall measurement + category-batched extraction (Phase 3 hypothesis test)
Before committing further GPU time to either a bigger local model or a full Phase 3 retrieval build, this measures two things directly: 1. src/retrieval.py: BM25 section retriever + compute_retrieval_recall(), scored against CUAD's own gold answer_start offsets -- no LegalBench-RAG download needed (still blocked in this environment). Real result on the full 77-contract test set: Recall@10=0.749, MRR=0.469, vs. the LLM's own effective recall of 0.158 -- retrieval is not the bottleneck the LLM itself is. Also measured: localizing by unioning top-k sections across all 41 categories barely compresses long documents (ratio 0.70-0.89), motivating per-batch-scoped retrieval instead (tested in the pilot below). 2. src/baseline.py: added extract_clauses_batched() (category batching, isolates whether one-shot 41-category enumeration is itself the bottleneck -- directly motivated by ContractEval 2025 showing similar-size open models score far higher on a per-que…4f47f31 -
fix: use explicit 127.0.0.1 for Ollama provider, not localhost (ADR-025)
localhost resolves to IPv6 [::1] first on this machine, but Ollama binds 127.0.0.1 only. socket.create_connection() applies the full 300s request timeout to each resolved address before falling back, so every call could silently hang for minutes on a dead IPv6 attempt before reaching the real listener. Diagnosed live via netstat (process stuck in SYN_SENT to [::1]:11434) while investigating an apparently-hung Phase 3 pilot run. Verified fixed: a smoke generate() call went from indefinite hang to 0.4s.c37f321
-
experiment: measure real v4 baseline (RED); root-cause the remaining gap
REAL MEASURED RESULT, v4 prompt (schema-constrained JSON), full 77-contract test set: Macro F1: 0.117 (up from v3's 0.077, but still far below the classical baseline's 0.4532) Micro F1: 0.259 | Micro precision/recall: 0.720 / 0.158 TP=152, FP=59, FN=811 76/77 contracts (98.7%) succeeded with zero errors -- JSON decode errors and hallucinated categories both dropped to ZERO (confirmed fix), only 1 edge case remains (no JSON array found) 229 total clauses extracted (mean 2.97/contract vs. real gold mean 27.1/contract) Mean confidence 0.965, Brier score 0.243, Spearman rho 0.304 Runtime 14,695.9s (~4h) for 77 contracts Per the pre-registered gate: STATUS = RED, unchanged from v3. Root-caused the remaining gap (not just re-measured and stopped): correlated real contract length against gold category count and clauses actually found across all 77 contracts. Pearson r(length, gold)=0.685 (longer contracts genuinely have more to find) vs. r(length, found)=0.093…a04c5e3 -
experiment: measure real v3 local-LLM baseline (RED); fix root cause (ADR-024)
REAL MEASURED RESULT, v3 prompt (unconstrained JSON generation), full 77-contract test set, no synthetic data, no reduced sample: Macro F1: 0.077 (over 40 scored categories) Micro F1: 0.192 TP=106, FP=36, FN=857 (142 predicted / 963 gold contract,category pairs) 24/77 contracts (31.2%) had extraction errors: 18 JSON decode errors, 1 no-JSON-array, 5 hallucinated/invalid categories (correctly rejected) 153 total clauses extracted across 77 contracts (mean 2.0/contract vs. real gold mean 27.1/contract) Runtime: 14,321.7s (~4h) for 77 contracts; latency mean 186.0s, median 30.9s, p95 643.2s GPU: 8,180 MiB VRAM, 98% utilization (RTX 2000 Ada, full offload) Per the pre-registered gate (docs/PHASE2_BASELINE.md, committed BEFORE this result existed): STATUS = RED. This is far below the classical TF-IDF baseline (0.4532) and is reported as measured -- not softened, not hidden, not worked around by changing the evaluation protocol. Root cause investigation: reproduc…0e99b4a -
docs: error analysis structure for Phase 2, populated with real run evidence so far
2a04959 -
docs: record real smoke test results (Stage 4) -- model actually loaded and ran
Real, measured evidence: Qwen2.5-7B-Instruct loaded fully onto the RTX 2000 Ada via Ollama (29/29 layers offloaded to GPU, confirmed via Ollama's own server log), 8,180 MiB VRAM used, 98% GPU utilization during generation. Found exactly the expected two categories on a hand-written sample contract with correct verbatim spans. Real finding: the extractor's timer showed 335.4s for the first call, but Ollama's own request log shows the actual generation took 35.4s -- the difference is a one-time model cold-load cost (reading the 4.68GB quantized weights from disk into VRAM) that only applies to the very first request in a run, not steady-state per-contract latency. Recorded both numbers rather than letting the cold-load cost silently inflate an average. Full 77-contract run launched; results PENDING, not fabricated.0fc1b01 -
docs: pre-register Phase 2 acceptance gate before the LLM result exists
Stage 14: do not invent a success threshold after seeing results. README's existing ">0.70 macro F1" target predates the current (contract, category) evaluation methodology and the switch to a local 7B model, so it isn't a valid gate for this measurement. Written and committed while the Ollama model pull is still in progress (51% at time of writing, verified via the background task's own progress output) and before any real LLM output exists: GREEN if local LLM macro F1 exceeds the classical baseline (0.4532) by >=0.05, YELLOW within +/-0.05, RED if it underperforms by >0.05 -- tied to a concrete, already-measured floor rather than an arbitrary absolute number.2e83436 -
docs: record Docker build network-interruption failure as a negative result
4bc68db -
docs: record actual Ollama installation verification evidence
5ee4f5a -
chore: regenerate EVAL_REPORT_PHASE2.md with the local-provider readiness check
The committed report was stale (from before the provider refactor), still showing claude-3-5-sonnet and ANTHROPIC_API_KEY. Re-ran scripts/run_baseline.py, which now correctly detects the local Ollama server isn't running yet (connection refused -- confirmed the real is_available() check works, not just a code review) and writes an honest NOT MEASURED report naming the actual local model and the actual reason.1f75ba6 -
docs: update ARCHITECTURE.md and ROADMAP.md to reflect the local model
Both docs still described the Phase 2 baseline as calling a paid frontier LLM (GPT-4/Claude). Updated to reflect the actual local, open-weight approach (Qwen2.5-7B-Instruct via Ollama) and pointed at docs/LOCAL_MODEL_SELECTION.md, docs/LOCAL_INFERENCE.md, and docs/PHASE2_BASELINE.md for the rationale and results.e92bf9d -
docs: record measured classical baseline; mark LLM baseline PENDING
docs/PHASE2_BASELINE.md establishes the experimental protocol shared by both baselines and records the classical TF-IDF+LogReg result verbatim from the actual run (Macro F1 0.4532, Micro F1 0.6742 -- see the commit that ran scripts/run_classical_baseline.py for the full output). The local LLM section is explicitly marked PENDING, not filled with placeholder numbers, while Ollama installation is still in progress -- this file will be updated with real measured values once scripts/smoke_test_local_model.py and scripts/run_baseline.py actually complete.7345dfa -
refactor: switch baseline pipeline to local provider; drop Claude requirement
src/baseline.py::BaselineExtractor now takes a ModelProvider instead of constructing an Anthropic client directly, and defaults to get_provider() (local Ollama). PROMPT_VERSION bumped to v3 (provider-agnostic prompt, chunking-aware system prompt noting the text may be a partial excerpt). extract_clauses() now: chunks contracts exceeding the provider's context via src/chunking.py, extracts each chunk independently, and merges results with exact-duplicate removal across chunk overlaps. _parse_response now rejects hallucinated/invalid category names outright (previously these could slip through) and handles empty responses explicitly. scripts/run_baseline.py: replaced the ANTHROPIC_API_KEY presence check with provider.is_available() (is Ollama actually running and is the model actually pulled) -- the equivalent readiness gate for a local setup. Report template now shows "$0 -- local inference, no paid API used" instead of a deferred cost calculation, records the actual GPU name (nvidia-s…ee02d8b -
feat: add local open-source inference backend (Ollama + Qwen2.5-7B-Instruct)
LocalOllamaProvider is now the default ModelProvider (get_provider() defaults to "local", overridable only via CLAUSE_MODEL_PROVIDER=anthropic for someone who separately has a paid key). Talks to a locally-running Ollama server via its REST API using only the standard library (urllib) -- no new pip dependency for this. Implements retry-with-exponential-backoff for transient connection errors (Ollama still loading, brief refusal) distinct from real failures (model not found, malformed request), a fixed generation seed (42) for reproducibility, and is_available() for a real readiness check before a run starts (replaces the old ANTHROPIC_API_KEY presence check with "is the local server actually reachable and is the model actually pulled"). docs/LOCAL_INFERENCE.md records why Ollama over Transformers+bitsandbytes (Windows CUDA setup risk) or vLLM (no native Windows support at all, ruled out outright by the platform constraint) or raw llama.cpp (Ollama already wraps it with less build com…dbe4c60 -
feat: add ModelProvider abstraction to remove hard Claude dependency
The user has an absolute no-paid-API constraint; the Phase 2 baseline was designed entirely around Claude 3.5 Sonnet (a paid API), which is now categorically unusable. Rather than block the project, introduced a provider abstraction so src/baseline.py talks to a ModelProvider interface instead of a specific SDK. ModelProvider (base.py) defines a narrow generate() contract returning a uniform ProviderResponse (text, token counts, latency, retry count) regardless of backend. AnthropicProvider is kept as an optional adapter (lazy-imported, never required) for anyone who separately has their own paid key -- but it is not part of the default path. docs/LOCAL_MODEL_SELECTION.md records the actual model comparison behind the next commit's choice: Qwen2.5-7B-Instruct (Apache 2.0, ungated, 32K context, GQA-efficient) vs Llama-3.1-8B-Instruct (gated, requires manual Meta approval, custom restrictive license) vs Mistral-7B-Instruct-v0.3 (Apache 2.0 but less proven structured-output reliability)…3088fd0 -
docs: Phase 3 data/evaluation readiness assessment
Consolidates this session's dataset verification findings into an explicit readiness check, per the standing instruction to produce this before any Phase 3 implementation and to stop at the Phase 3 implementation boundary. States plainly: CUAD and ContractNLI are ready (downloaded, schema-verified); MAUD is deferred (optional per docs/ROADMAP.md, Phase 6+, not worth installing datasets/pandas for yet); LegalBench-RAG's source and MIT license are verified but the actual download is blocked by dropbox.com being unreachable from this environment (a network restriction, not a licensing or dataset problem). No retrieval code has been written. Phase 2 remains the gating blocker for Phase 3 proper, per the project's own phase-gate policy.45539a4 -
data: download and verify ContractNLI against the real official archive
Phase 2 remains blocked on ANTHROPIC_API_KEY. Continuing legitimate non-API dataset verification work. Unlike CUAD/MAUD (ADR-015), ContractNLILoader's schema guess was correct on first real download -- no loader code change needed. Downloaded from the official Stanford source (65,362,913 bytes, SHA256 recorded in data/DATA_SOURCES.md) and verified: - 607 documents total (matches the long-standing doc claim, now genuinely measured rather than asserted) - 606 unique document texts; one duplicate-text pair (id 52, a Customs and Border Protection NDA, and id 97, a Dept. of Homeland Security NDA -- byte-identical, evidently the same template reused by two related agencies). Verified both fall within the official 'test' split, so this is NOT train/test leakage. - CC BY 4.0 confirmed directly from LICENSE and TERMS files shipped inside the archive itself (Hitachi America, Ltd. as the stated owner) -- stronger evidence than CUAD's case, where no LICENSE file exists in that so…3c80489 -
docs+fix: correct overclaim about LegalBench-RAG source; verify license (ADR-020)
Phase 2 remains blocked on ANTHROPIC_API_KEY (confirmed not set, no CLAUDE_API_KEY, no .env). Continuing legitimate non-API Phase 3 prep work per standing instructions. An earlier pass (2026-08-25) flagged the Dropbox URL in LegalBenchRAGLoader as "fragile/unofficial" without actually checking the official repo. Fetched the real README from zeroentropy-ai/legalbenchrag (note: lowercase org, default branch "master" not "main") and found its own Download link is byte-for-byte identical to the URL already in the codebase (same folder id, share id, rlkey, st token; dl=1 vs the README's dl=0 -- dl=1 is the correct parameter for non-interactive download). No code change to the URL was needed; it was already correct. Confirmed MIT license via GitHub API's structured license.key field (stronger evidence than a topic tag). Attempted to actually reach the download: dropbox.com returns SSLConnectError from this environment on a bare-domain request while github.com succeeds in the same session -…e2ec5b7 -
chore: re-stamp NOT MEASURED report provenance after readiness audit
Re-running scripts/run_baseline.py (still no ANTHROPIC_API_KEY) to verify the Phase 2 readiness checkpoint updates the report's timestamp and git commit hash fields to the current state -- no content change, just confirms the provenance-stamping mechanism reads live values correctly.de27efd
-
docs: record Phase 2 scientific readiness audit (ADR-018, ADR-019)
Documents the two methodologically-invalidating defects found and fixed by actually loading and measuring against the real CUAD test split before spending any API budget: F1 granularity mismatch (ADR-018) and recall-capping contract truncation (ADR-019). Updated EVAL_PLAN.md's F1 definition to state the (contract, category) presence unit explicitly, and EVAL_REPORT.md's Phase 2 section with the audit findings. EVAL_REPORT_PHASE2.md regenerated to reflect current real state (NOT MEASURED, ANTHROPIC_API_KEY not set) after a dry run temporarily populated it with synthetic numbers for verification purposes.8737edc -
docs: replace overclaimed dataset/environment status with measured facts
DATA_CARD.md, EVAL_REPORT.md, PHASE0_2_COMPLETION.md, and README.md all asserted datasets were "Verified" and Phase 1 was "Complete" before any file had been downloaded or any loader executed -- aspirational status presented as fact. Corrected all four to state exactly what has and hasn't been checked, with evidence: - CUAD: genuinely downloaded, parsed, and tested against real data (SHA256, contract/clause counts, category list all measured) - ContractNLI / LegalBench-RAG: explicitly marked not verified, not downloaded, with what specifically remains to be checked - MAUD: schema corrected via API inspection but not downloaded - CUAD's CC BY 4.0 license is stated on the project website but has no LICENSE file in the source GitHub repo -- now flagged as claimed, not independently confirmed, rather than asserted as fact Also corrected the stale claim that the Python/Docker environment was blocked (the `py` launcher and Docker both work; only the API key is actually missing), and…1ffa63b -
docs: add ADR-015/016/017 for data-source, injection-defense, calibration fixes
Documents three consequential decisions made while fixing the code in preceding commits: - ADR-015: CUAD data source correction (HF -> official GitHub archive), including the category-casing normalization and deduplication findings discovered by actually running the loader against real data. - ADR-016: prompt injection defense (system/user separation) in the zero-shot baseline, and the stray-comment bug it also fixed. - ADR-017: confidence calibration must use the same per-instance matching as F1 computation, not a separate (and looser) correctness definition.50d0a13 -
phase-0-audit: Environmental analysis and findings
c6f8c4c -
docs: Phase 0-2 completion report
- Phase 0 (Architecture): ✓ COMPLETE (9 documents) - Phase 1 (Data Pipeline): ✓ COMPLETE (loaders, split tests, schemas) - Phase 2 (Baseline): ✓ SCAFFOLDING COMPLETE (ready to measure) All scaffolding in place. Awaiting user input to proceed: 1. ANTHROPIC_API_KEY environment variable 2. Dataset download (CUAD, ContractNLI, MAUD) 3. Python environment setup (Docker or venv) Then: python scripts/run_baseline.py → measures baseline F1 Zero fabrication. All metrics marked NOT MEASURED until executed.44eb9df -
phase-2-baseline: Baseline extraction architecture and scripts
- src/baseline.py: Zero-shot extraction using Claude 3.5 Sonnet * No retrieval, no agents, no optimization * Structured output (Pydantic) with validation * Deterministic (temperature=0) * Per-contract extraction + confidence scores - src/evaluator.py: Baseline evaluation metrics * Macro F1 (primary): average across all categories * Micro F1 (secondary): global metric * Per-category F1: diagnostic breakdown * Confidence calibration (Brier score, Spearman rho) - scripts/run_baseline.py: End-to-end baseline execution * Loads test contracts from Phase 1 splits * Calls Claude API for extraction * Computes all metrics * Generates evaluation report - docs/EVAL_REPORT.md: Phase 0-2 progress tracker * Current status: Phase 2 ready but NOT YET MEASURED * All metrics explicitly marked as NOT MEASURED * No fabrication, no assumptions * Clear blocking factors Status: Phase 2 complete (scaffolding). Ready to execute once: 1. ANTHROPIC_API_KEY is set (requires user…31c05a4 -
phase-1-data-pipeline: Dataset loaders, splitting, and leakage tests
- data/loaders.py: Unified loaders for CUAD, ContractNLI, MAUD, LegalBench-RAG * Load from official sources only (verified licenses) * Support for all major legal datasets * Contract-level data access (critical for split integrity) - data/split_utils.py: Contract-level train/val/test splitting * Guarantees zero leakage (no contract in multiple splits) * Reproducible with fixed seeds * Dataset statistics generation - tests/test_data_leakage.py: Critical leakage prevention tests * test_no_contract_leakage: MUST PASS before evaluation * test_no_duplicate_text_across_splits: Catch identical contracts * test_split_fractions: Verify split distribution * test_reproducibility_with_seed: Ensure deterministic splits - src/schemas.py: Pydantic models for validation * ClauseExtraction: Clause with category, confidence, source span * RiskFlag: Risk with verification status and citation * ContractExtractionResult: Complete extraction for a contract * EvaluationMetrics:…9c26600 -
phase-0-architecture: Initial architecture, decisions, roadmap, and data card
- ARCHITECTURE.md: System design and minimal tech stack - DECISIONS.md: ADRs (architectural decision records) for key choices - ROADMAP.md: Phase timeline and success criteria - DATA_CARD.md: Dataset provenance, licensing, leakage prevention - EVAL_PLAN.md: Evaluation metrics and acceptance gates - requirements.txt: Phase 0-2 dependencies (minimal, non-premature) - .env.example: Configuration template - .gitignore: Exclude data, credentials, build artifacts - README.md: Project overview and getting started guide Status: Phase 0 complete. Ready for Python environment setup and Phase 1 (data pipeline).dd8a3f5