Spaces:
Running
Running
Commit History
-
data: download and verify ContractNLI against the real official archive
Phase 2 remains blocked on ANTHROPIC_API_KEY. Continuing legitimate non-API dataset verification work. Unlike CUAD/MAUD (ADR-015), ContractNLILoader's schema guess was correct on first real download -- no loader code change needed. Downloaded from the official Stanford source (65,362,913 bytes, SHA256 recorded in data/DATA_SOURCES.md) and verified: - 607 documents total (matches the long-standing doc claim, now genuinely measured rather than asserted) - 606 unique document texts; one duplicate-text pair (id 52, a Customs and Border Protection NDA, and id 97, a Dept. of Homeland Security NDA -- byte-identical, evidently the same template reused by two related agencies). Verified both fall within the official 'test' split, so this is NOT train/test leakage. - CC BY 4.0 confirmed directly from LICENSE and TERMS files shipped inside the archive itself (Hitachi America, Ltd. as the stated owner) -- stronger evidence than CUAD's case, where no LICENSE file exists in that so…3c80489 -
docs+fix: correct overclaim about LegalBench-RAG source; verify license (ADR-020)
Phase 2 remains blocked on ANTHROPIC_API_KEY (confirmed not set, no CLAUDE_API_KEY, no .env). Continuing legitimate non-API Phase 3 prep work per standing instructions. An earlier pass (2026-08-25) flagged the Dropbox URL in LegalBenchRAGLoader as "fragile/unofficial" without actually checking the official repo. Fetched the real README from zeroentropy-ai/legalbenchrag (note: lowercase org, default branch "master" not "main") and found its own Download link is byte-for-byte identical to the URL already in the codebase (same folder id, share id, rlkey, st token; dl=1 vs the README's dl=0 -- dl=1 is the correct parameter for non-interactive download). No code change to the URL was needed; it was already correct. Confirmed MIT license via GitHub API's structured license.key field (stronger evidence than a topic tag). Attempted to actually reach the download: dropbox.com returns SSLConnectError from this environment on a bare-domain request while github.com succeeds in the same session -…e2ec5b7
-
fix: replace unverified CUAD/MAUD loaders with schemas verified against real data
CUADLoader called load_dataset("theatticusproject/cuad") on HuggingFace. Verified via datasets-server.huggingface.co/info that this dataset is {"text": string} only -- no categories/labels/spans fields -- so the loader's example["categories"]/example["labels"]/example["spans"] access would have raised KeyError on first real use. It was never run before now. Replaced with a loader that downloads the real official archive (https://raw.githubusercontent.com/The-Atticus-Project/cuad/main/data.zip, note: GitHub org renamed TheAtticusProject -> The-Atticus-Project) and parses its actual SQuAD-style schema. Verified end-to-end against a real download: 510 contracts, 13,823 raw clause annotations, categories exactly matching src.schemas.CUAD_CATEGORIES after normalizing a real casing mismatch (10 of 41 categories are title-cased differently in the raw question/id text than in the canonical category_descriptions.csv list -- uncaught, this would have silently zeroed precision/recall for those 1…14423d1 -
phase-1-data-pipeline: Dataset loaders, splitting, and leakage tests
- data/loaders.py: Unified loaders for CUAD, ContractNLI, MAUD, LegalBench-RAG * Load from official sources only (verified licenses) * Support for all major legal datasets * Contract-level data access (critical for split integrity) - data/split_utils.py: Contract-level train/val/test splitting * Guarantees zero leakage (no contract in multiple splits) * Reproducible with fixed seeds * Dataset statistics generation - tests/test_data_leakage.py: Critical leakage prevention tests * test_no_contract_leakage: MUST PASS before evaluation * test_no_duplicate_text_across_splits: Catch identical contracts * test_split_fractions: Verify split distribution * test_reproducibility_with_seed: Ensure deterministic splits - src/schemas.py: Pydantic models for validation * ClauseExtraction: Clause with category, confidence, source span * RiskFlag: Risk with verification status and citation * ContractExtractionResult: Complete extraction for a contract * EvaluationMetrics:…9c26600