Spaces:
Running
Running
Commit History
-
research: BM25 retrieval-recall measurement + category-batched extraction (Phase 3 hypothesis test)
Before committing further GPU time to either a bigger local model or a full Phase 3 retrieval build, this measures two things directly: 1. src/retrieval.py: BM25 section retriever + compute_retrieval_recall(), scored against CUAD's own gold answer_start offsets -- no LegalBench-RAG download needed (still blocked in this environment). Real result on the full 77-contract test set: Recall@10=0.749, MRR=0.469, vs. the LLM's own effective recall of 0.158 -- retrieval is not the bottleneck the LLM itself is. Also measured: localizing by unioning top-k sections across all 41 categories barely compresses long documents (ratio 0.70-0.89), motivating per-batch-scoped retrieval instead (tested in the pilot below). 2. src/baseline.py: added extract_clauses_batched() (category batching, isolates whether one-shot 41-category enumeration is itself the bottleneck -- directly motivated by ContractEval 2025 showing similar-size open models score far higher on a per-que…4f47f31 -
fix: use explicit 127.0.0.1 for Ollama provider, not localhost (ADR-025)
localhost resolves to IPv6 [::1] first on this machine, but Ollama binds 127.0.0.1 only. socket.create_connection() applies the full 300s request timeout to each resolved address before falling back, so every call could silently hang for minutes on a dead IPv6 attempt before reaching the real listener. Diagnosed live via netstat (process stuck in SYN_SENT to [::1]:11434) while investigating an apparently-hung Phase 3 pilot run. Verified fixed: a smoke generate() call went from indefinite hang to 0.4s.c37f321
-
experiment: measure real v3 local-LLM baseline (RED); fix root cause (ADR-024)
REAL MEASURED RESULT, v3 prompt (unconstrained JSON generation), full 77-contract test set, no synthetic data, no reduced sample: Macro F1: 0.077 (over 40 scored categories) Micro F1: 0.192 TP=106, FP=36, FN=857 (142 predicted / 963 gold contract,category pairs) 24/77 contracts (31.2%) had extraction errors: 18 JSON decode errors, 1 no-JSON-array, 5 hallucinated/invalid categories (correctly rejected) 153 total clauses extracted across 77 contracts (mean 2.0/contract vs. real gold mean 27.1/contract) Runtime: 14,321.7s (~4h) for 77 contracts; latency mean 186.0s, median 30.9s, p95 643.2s GPU: 8,180 MiB VRAM, 98% utilization (RTX 2000 Ada, full offload) Per the pre-registered gate (docs/PHASE2_BASELINE.md, committed BEFORE this result existed): STATUS = RED. This is far below the classical TF-IDF baseline (0.4532) and is reported as measured -- not softened, not hidden, not worked around by changing the evaluation protocol. Root cause investigation: reproduc…0e99b4a -
refactor: switch baseline pipeline to local provider; drop Claude requirement
src/baseline.py::BaselineExtractor now takes a ModelProvider instead of constructing an Anthropic client directly, and defaults to get_provider() (local Ollama). PROMPT_VERSION bumped to v3 (provider-agnostic prompt, chunking-aware system prompt noting the text may be a partial excerpt). extract_clauses() now: chunks contracts exceeding the provider's context via src/chunking.py, extracts each chunk independently, and merges results with exact-duplicate removal across chunk overlaps. _parse_response now rejects hallucinated/invalid category names outright (previously these could slip through) and handles empty responses explicitly. scripts/run_baseline.py: replaced the ANTHROPIC_API_KEY presence check with provider.is_available() (is Ollama actually running and is the model actually pulled) -- the equivalent readiness gate for a local setup. Report template now shows "$0 -- local inference, no paid API used" instead of a deferred cost calculation, records the actual GPU name (nvidia-s…ee02d8b -
fix: implement long-document extraction for the local model's 32K context
Qwen2.5-7B-Instruct's native context (32,768 tokens, verified from its HF config.json) is smaller than Claude 3.5 Sonnet's 200K, so ADR-019's "just send the whole contract" approach cannot be assumed to still work. Measured the real 77-contract test set against a conservative token budget: 64/77 (83.1%) fit in a single pass; only 13/77 need chunking. Implemented the simplest strategy that preserves recall for those 13: overlapping whitespace-snapped chunks (src/chunking.py), each extracted independently, merged at the (contract, category) presence level already established by ADR-018. Chunk boundaries are determined purely by text length -- identically for train/val/test -- so this introduces no leakage risk and never uses gold labels to pick boundaries. tests/test_chunking.py (6 tests) verifies: short text passes through unchanged, long text splits correctly, chunks collectively cover the entire document with no gaps (the property that actually matters for recall), and boundaries do…837de66 -
feat: add local open-source inference backend (Ollama + Qwen2.5-7B-Instruct)
LocalOllamaProvider is now the default ModelProvider (get_provider() defaults to "local", overridable only via CLAUSE_MODEL_PROVIDER=anthropic for someone who separately has a paid key). Talks to a locally-running Ollama server via its REST API using only the standard library (urllib) -- no new pip dependency for this. Implements retry-with-exponential-backoff for transient connection errors (Ollama still loading, brief refusal) distinct from real failures (model not found, malformed request), a fixed generation seed (42) for reproducibility, and is_available() for a real readiness check before a run starts (replaces the old ANTHROPIC_API_KEY presence check with "is the local server actually reachable and is the model actually pulled"). docs/LOCAL_INFERENCE.md records why Ollama over Transformers+bitsandbytes (Windows CUDA setup risk) or vLLM (no native Windows support at all, ruled out outright by the platform constraint) or raw llama.cpp (Ollama already wraps it with less build com…dbe4c60 -
feat: add ModelProvider abstraction to remove hard Claude dependency
The user has an absolute no-paid-API constraint; the Phase 2 baseline was designed entirely around Claude 3.5 Sonnet (a paid API), which is now categorically unusable. Rather than block the project, introduced a provider abstraction so src/baseline.py talks to a ModelProvider interface instead of a specific SDK. ModelProvider (base.py) defines a narrow generate() contract returning a uniform ProviderResponse (text, token counts, latency, retry count) regardless of backend. AnthropicProvider is kept as an optional adapter (lazy-imported, never required) for anyone who separately has their own paid key -- but it is not part of the default path. docs/LOCAL_MODEL_SELECTION.md records the actual model comparison behind the next commit's choice: Qwen2.5-7B-Instruct (Apache 2.0, ungated, 32K context, GQA-efficient) vs Llama-3.1-8B-Instruct (gated, requires manual Meta approval, custom restrictive license) vs Mistral-7B-Instruct-v0.3 (Apache 2.0 but less proven structured-output reliability)…3088fd0
-
fix: remove recall-capping truncation; add retry/output-token headroom
Scientific readiness audit measured that the previous 8,000-character contract truncation would have made the baseline structurally incapable of finding most gold clauses: 90.9% of the real test contracts exceed 8,000 chars, and 61.5% of all gold clause spans (by answer_start offset) begin beyond that cutoff. This caps measurable recall near ~38% regardless of model quality -- a ceiling baked into the harness, not a finding about the model (see docs/DECISIONS.md ADR-019). Removed the truncation; MAX_CONTRACT_CHARS=400,000 remains only as a safety ceiling far above the largest real CUAD contract (338,211 chars, well inside Claude 3.5 Sonnet's 200K-token context window), with a logged warning if any document ever actually exceeds it. Also, in the same audit pass: - max_tokens (output) raised 4096->8192: a category-rich contract (up to 41 categories) can produce JSON that overflows 4096 tokens and gets truncated mid-object. - Anthropic(max_retries=5), up from the client default of 2:…adde6df -
fix: score F1 at (contract, category) presence, not per-span instance
Scientific readiness audit (see docs/DECISIONS.md ADR-018) found that scoring at per-clause-instance granularity was measuring the wrong task. CUAD frequently annotates the same category many times per contract (measured on the real 77-contract test split: 39.6% of (contract, category) pairs have >1 gold span; one pair has 23), while the baseline prompt asks the model to identify categories present and provide ONE entry per category found -- not enumerate every mention. Under the old per-instance scoring, a model that perfectly identified a category but cited one representative span (exactly what was asked) would score up to 22 false negatives on that single pair alone. Verified with a synthetic reproduction of the real worst case: F1 0.087 (old) vs 1.0 (fixed) for identical model behavior. Also found: "Price Restrictions" has zero gold examples in the real test split (three more categories have 2-3). The old 0/0->0.0 default meant a category with no evidence either way was scored as …9158238 -
fix: prompt injection defense, stray-comment bug, token tracking
_build_prompt used an f-string with a trailing "# Truncate for token limit" after the interpolated contract text -- f-strings don't support inline comments, so that text was not a comment, it was literally sent to the model as part of the prompt. More importantly, despite ADR-014 already deciding that contract text must be treated as untrusted input with an enforced instruction hierarchy, the actual prompt had zero separation: all instructions and the raw contract text were concatenated into one `user` message. Moved all instructions into the Anthropic `system` parameter; the `user` message now contains only the contract text wrapped in an explicit <contract_text> delimiter, with the system prompt stating that content inside those tags is data, not instructions, and must not be followed even if it looks like a command (ADR-016). Also added input_tokens/output_tokens capture from the API response. This was previously untracked entirely, which meant cost could never be reported withou…32ce9c3 -
fix: correct F1 matching, calibration, and cross-contract identity bugs
- compute_f1: was comparing predicted/gold clause sets by category alone via Python set intersection on dicts (which would actually raise TypeError -- dicts aren't hashable). Rewritten to use Counter-based per-instance matching on grounded clause text within each category. - compute_confidence_calibration: judged a prediction "correct" if its category appeared anywhere in gold labels, independent of compute_f1's per-instance matching -- inconsistent definitions of "correct" in the same module (ADR-017). Rewritten to consume gold the same way compute_f1 does, decrementing a per-category Counter so one gold clause can back at most one "correct" prediction. - _clause_identity now incorporates contract_id when present, so identical boilerplate text in two different contracts (e.g. a common "Governing Law: State of Delaware" clause) can't spuriously count as a match when predictions/gold from multiple contracts are pooled into one compute_f1() call, as scripts/run_ba…4be55f3 -
fix: correct CUAD category list; add token usage fields
- CUAD_CATEGORIES previously listed the wrong 41 categories entirely (looked like a generic legal-contract category list, not CUAD's actual taxonomy). Replaced with the official list from https://raw.githubusercontent.com/The-Atticus-Project/cuad/main/category_descriptions.csv, now verified against the real downloaded dataset (see data/DATA_SOURCES.md and docs/DECISIONS.md ADR-015): exact match to the 41 categories found in 13,807 real clause annotations. - ContractExtractionResult gains input_tokens/output_tokens fields so src/baseline.py can record real API usage per call. Without this, cost could never be reported without fabrication (see ADR-016).227ccaa -
phase-2-baseline: Baseline extraction architecture and scripts
- src/baseline.py: Zero-shot extraction using Claude 3.5 Sonnet * No retrieval, no agents, no optimization * Structured output (Pydantic) with validation * Deterministic (temperature=0) * Per-contract extraction + confidence scores - src/evaluator.py: Baseline evaluation metrics * Macro F1 (primary): average across all categories * Micro F1 (secondary): global metric * Per-category F1: diagnostic breakdown * Confidence calibration (Brier score, Spearman rho) - scripts/run_baseline.py: End-to-end baseline execution * Loads test contracts from Phase 1 splits * Calls Claude API for extraction * Computes all metrics * Generates evaluation report - docs/EVAL_REPORT.md: Phase 0-2 progress tracker * Current status: Phase 2 ready but NOT YET MEASURED * All metrics explicitly marked as NOT MEASURED * No fabrication, no assumptions * Clear blocking factors Status: Phase 2 complete (scaffolding). Ready to execute once: 1. ANTHROPIC_API_KEY is set (requires user…31c05a4 -
phase-1-data-pipeline: Dataset loaders, splitting, and leakage tests
- data/loaders.py: Unified loaders for CUAD, ContractNLI, MAUD, LegalBench-RAG * Load from official sources only (verified licenses) * Support for all major legal datasets * Contract-level data access (critical for split integrity) - data/split_utils.py: Contract-level train/val/test splitting * Guarantees zero leakage (no contract in multiple splits) * Reproducible with fixed seeds * Dataset statistics generation - tests/test_data_leakage.py: Critical leakage prevention tests * test_no_contract_leakage: MUST PASS before evaluation * test_no_duplicate_text_across_splits: Catch identical contracts * test_split_fractions: Verify split distribution * test_reproducibility_with_seed: Ensure deterministic splits - src/schemas.py: Pydantic models for validation * ClauseExtraction: Clause with category, confidence, source span * RiskFlag: Risk with verification status and citation * ContractExtractionResult: Complete extraction for a contract * EvaluationMetrics:…9c26600