Spaces:
Running
Running
Commit History
-
refactor: switch baseline pipeline to local provider; drop Claude requirement
src/baseline.py::BaselineExtractor now takes a ModelProvider instead of constructing an Anthropic client directly, and defaults to get_provider() (local Ollama). PROMPT_VERSION bumped to v3 (provider-agnostic prompt, chunking-aware system prompt noting the text may be a partial excerpt). extract_clauses() now: chunks contracts exceeding the provider's context via src/chunking.py, extracts each chunk independently, and merges results with exact-duplicate removal across chunk overlaps. _parse_response now rejects hallucinated/invalid category names outright (previously these could slip through) and handles empty responses explicitly. scripts/run_baseline.py: replaced the ANTHROPIC_API_KEY presence check with provider.is_available() (is Ollama actually running and is the model actually pulled) -- the equivalent readiness gate for a local setup. Report template now shows "$0 -- local inference, no paid API used" instead of a deferred cost calculation, records the actual GPU name (nvidia-s…ee02d8b
-
fix: render calibration metrics; stamp real provenance into reports
Two gaps found while verifying the eval report actually contains measured values rather than placeholders: 1. compute_confidence_calibration() was already being called and its result stashed in metrics["calibration"], but generate_eval_report() never rendered it into the markdown -- computed values that never reached the output. Now shown in a dedicated Confidence Calibration section. 2. Model name, seed, and split fractions were hardcoded literal strings in the report templates, independent of what actually ran. Added _git_commit_hash() and _split_provenance() helpers; reports now read the real values from extraction_results (model, prompt_version, temperature -- set by src/baseline.py's extract_baseline()) and from data/processed/cuad_splits.json's own recorded seed/fractions, so a report can't silently drift out of sync with the code/data that produced it. Also fixed a crash the evaluator's new (contract, category) presence granularity would have ca…f5ec49d -
fix: persist full contract data across splits; make run_baseline usable
prepare_datasets.py previously wrote only contract IDs to cuad_splits.json. That's enough for leakage checks but not enough to actually run extraction -- there was no way to reconstruct a test contract's text/clauses from an ID alone without re-querying the (broken) loader. Now also writes cuad_{train,val,test}_contracts.json with full text + gold clauses per split. run_baseline.py had two defects that made it non-functional even with data and credentials present: 1. load_test_contracts() was a stub that always returned [] regardless of whether data existed (a known issue flagged before this change). 2. generate_eval_report() wrote a hardcoded "NOT MEASURED" markdown template unconditionally -- it never read its `results` argument, so even a fully successful extraction run would still have produced a report claiming nothing was measured. Fixed both: load_test_contracts() now reads the persisted cuad_test_contracts.json; generate_eval_report() takes an optional `metrics` d…cec37e6 -
phase-2-baseline: Baseline extraction architecture and scripts
- src/baseline.py: Zero-shot extraction using Claude 3.5 Sonnet * No retrieval, no agents, no optimization * Structured output (Pydantic) with validation * Deterministic (temperature=0) * Per-contract extraction + confidence scores - src/evaluator.py: Baseline evaluation metrics * Macro F1 (primary): average across all categories * Micro F1 (secondary): global metric * Per-category F1: diagnostic breakdown * Confidence calibration (Brier score, Spearman rho) - scripts/run_baseline.py: End-to-end baseline execution * Loads test contracts from Phase 1 splits * Calls Claude API for extraction * Computes all metrics * Generates evaluation report - docs/EVAL_REPORT.md: Phase 0-2 progress tracker * Current status: Phase 2 ready but NOT YET MEASURED * All metrics explicitly marked as NOT MEASURED * No fabrication, no assumptions * Clear blocking factors Status: Phase 2 complete (scaffolding). Ready to execute once: 1. ANTHROPIC_API_KEY is set (requires user…31c05a4