# KDD Cup 2026 Data Agent — Architecture Overview ## 1. What This Is A multi-agent LLM-based system that reads data tasks (question + context data), plans SQL/pandas queries, executes them, and produces prediction CSVs. Built for the KDD Cup 2026 Data Agent Challenge. **Input:** A `task.json` (question + difficulty) plus a `context/` folder containing CSVs, JSONs, SQLite DBs, `knowledge.md`, and `doc/*.md` narrative files. **Output:** A `prediction.csv` with the answer, plus a `trace.json` audit trail. ### DAO — Data Agent Observatory **NEW**: DAO (Data Agent Observatory) is a Streamlit-based observability and exploration application that transforms agent reasoning into a professional, judge-facing "Mission Control" experience. **Launch**: `streamlit run src/data_agent_baseline/observatory/app.py` **Features**: - **Top-Level Modes**: CHAT (default landing) and DEBUG - **Run Launcher + Live Execution** (Phases 4-5): Launch predefined benchmark tasks from the UI and monitor progress/events in-session - **Zero-Click Startup** (DEBUG mode): Auto-discovers runs, pre-selects latest run and first task - **Executive Summary**: Mission overview, decision reasoning, refinement timeline, deliverables <<<<<<< HEAD - **14 Debug Tabs**: Run Launcher, Run Intelligence, Mission Summary, Reasoning DAG, Time-Travel Replay, Checkpoint Review, Guided Rerun Plan, Rerun Comparison, Provenance, Critic/Reviewer, Confidence, Failure/Verification, Raw Trace, Demo ======= - **Consolidated Primary Navigation (Phase 19)**: Run Launcher, Run Intelligence, Task Intelligence, Demo / Future Proof - **Fallback Views Preserved**: Live Trace Flow and Guided Live Trace Flow (HITL) remain available under Advanced / Fallback Views without runtime refactor >>>>>>> origin/ph2 - **Professional UI**: Dark theme with gradient cards, status badges, clean sidebar - **Judge-Ready**: Suitable for competition evaluation and executive presentation ### Phase 19 Status (2026-07) - Primary navigation now uses four top-level pages: Run Launcher, Run Intelligence, Task Intelligence, Demo / Future Proof. - Task Intelligence replaces Mission Summary as the task-level hub and is now consolidated into five judge-oriented internal sections: - Summary - Reasoning & Replay - Evidence - Review & Reliability - Raw Trace - Reasoning & Replay uses nested subsections: Reasoning Flow and Guided Step Inspector. - Review & Reliability uses nested subsections: Review Signals, Trust Calibration, and Diagnosis & Verification. - Demo / Future Proof groups checkpoint, rerun, comparison, evidence, and demo workflows. - Live Trace Flow and Guided Live Trace Flow remain intentionally untouched as fallback/stability pages and are opened directly via the sidebar View selector. - Advanced / Fallback Views informational copy was moved out of the primary-page flow and into Task Intelligence guidance text. - Run Intelligence now includes a compact Verification Summary with metric tooltips (calculation + source), and Guided Rerun Cohort Analysis is shown in Summary view only. - No execution, evaluation, HITL runtime behavior, or artifact schema changes were introduced. - Phase 20 natural-language artifact Q&A remains explicitly out of scope. **Quick Start**: 1. Run the command above 2. App auto-loads latest run and first task (< 2s) 3. Open Task Intelligence for task-level narrative and deep inspection 4. Use other tabs for detailed analysis See [Observatory.md](Observatory.md) for complete documentation. ### Analyst Team Disagreement Layer (Additive) The AAT path now includes a lightweight analyst-collaboration layer that captures: - per-analyst structured opinions (`aat_analyst_opinions`) - structured disagreement items (`aat_disagreements`) - coordinator synthesis (`aat_coordinator_synthesis`) This layer is additive and does not rewrite the baseline graph. Existing stages, retries, and output artifacts are preserved. Calibration update: disagreement detection is now intent-gated and tuned for meaningful disagreement reporting (not routine confidence noise). Reporting stabilization update (2026-06-16): coordinator metric reconciliation mismatches are diagnostic warnings (non-fatal), and meaningful disagreement counts now prioritize analyst-team meaningful flags over raw disagreement signals. Auditor calibration update: guard outputs include structured audit payloads (passed, confidence, detected_risks, evidence, suggested_action), and eval-v2 reports failure-prevention effectiveness via critical disagreement score plus auditor precision/recall. ### Phase 0 + Phase 1 Status (2026-06) - **Phase 0 command contract is preserved**: existing CLI behavior remains stable for run-task, run-benchmark, eval-v2, and task viewing workflows. - **Phase 1 run execution refactor is complete**: run orchestration was extracted into a shared application service and domain/repository boundaries. - **Behavioral compatibility is maintained**: the refactor is additive and does not change core solve logic or baseline output semantics. - **Run artifacts are now structured and explicit**: `run_manifest.json` and `run_events.jsonl` are written as additive artifacts for selected-task and benchmark execution paths. - **Benchmark compatibility remains intact**: `summary.json` is preserved for benchmark runs. - **Task-level deliverables are unchanged**: task folders still emit `trace.json` and `prediction.csv`. ### Phase 2 Status (2026-06) - **Phase 2 evaluation refactor is complete**: eval-v2 orchestration now runs through a reusable `EvaluationService` (`src/data_agent_baseline/application/evaluation_service.py`). - **CLI remains backward-compatible**: `dabench eval-v2` keeps the same arguments/defaults and now acts as a thin adapter (parse options, delegate to service, render selected report, print summary, exit by bundle status). - **Artifact contract remains unchanged**: all eval-v2 outputs remain at run root (`task_metrics.csv`, `trajectory.csv`, `tool_calls.csv`, `comprehensive_evaluation.csv`, validation/auditor/hardening reports, and per-task `task_replay.json`). - **Validation/hardening semantics remain unchanged**: warning-only validation exits 0, error validation exits 1, all three report modes are validated, and duplicate-section detection is preserved. ### Phase 3 Status (2026-06) - **Phase 3 Streamlit Run Intelligence is complete**: a dedicated run-level intelligence view is available in the Observatory app and consumes existing evaluation artifacts. - **Run Health Summary first**: the page opens with compact run-level health cards (Harness Health, Execution Success Rate, Failed Tasks, Mean Score and related run KPIs). - **Trajectory Intelligence section**: focused analysis of full-run trajectory behavior including: - Trajectory KPI Snapshot (tasks, patterns, path length, branching, critic loops, success rates, recovery rates) - Dominant Trajectory Patterns with interpretations (Clean, Retry, Replan, Complex Recovery) - Phase Timeline & Token Economics with per-phase time, tokens, tools, failures, confidence - Recovery / Self-Correction Flow (first-try rate, recovery success, planner/coordinator/execution retries) - Coordinator Decision Visibility (PROCEED, REQUEST_SPECIALIST, REPLAN, RETRY_EXECUTION, APPROVE_FINAL) - Trajectory Health Labels (Clean, Recovered, Unrecovered, Overconfident, Expensive, Complex) - Trajectory Examples and a unified recommended-task deep link into Task Intelligence - **Reliability Diagnostics section**: compact Root Cause Distribution panel with summary cards, task inspector, and optional raw/toggle views. - **View levels**: Summary (judge-focused), Detailed (comprehensive), Research (all tables + advanced metrics) - **Missing-evaluation flow is read-only and informative**: when artifacts are missing, the page renders a no-eval fallback with execution summary, task status, artifact coverage, event timeline summary, and explicit eval-v2 guidance. - **Compatibility is preserved**: eval-v2 artifact schemas and locations are unchanged; Phase 3 adds presentation/adapters/tests only. ### Phase 4 Status (2026-06) - **Phase 4 Streamlit Run Launcher is complete**: a new Run Launcher tab in the Observatory lets users select predefined benchmark tasks and execute them without leaving the UI. - **Execution goes through the shared service boundary**: the launcher calls `RunExecutionService.execute_selected_tasks()` exclusively — no subprocess, CLI, or direct LangGraph runner calls from Streamlit. - **Synchronous, sequential execution only**: the page blocks while tasks run and shows per-task progress via the service callback. Background jobs, cancellation, and parallel selected-task execution are deferred. - **Task catalog is loaded cheaply on demand**: `task_catalog.py` exposes `task_id`, difficulty, question preview, and context file counts without loading heavy context data unless requested. - **Artifact compatibility is preserved**: run artifacts (`run_manifest.json`, `run_events.jsonl`, `trace.json`, `prediction.csv`) are identical whether the run was started from the CLI or the launcher. - **Post-run navigation is wired**: after a successful run, Open Run and Open Task Context actions update the existing session-state keys so the run immediately appears in all other Observatory tabs. - **Run Intelligence handoff is integrated**: when a launcher run reaches terminal status, users can open Run Intelligence directly for that run from the launcher panel. - **Eval-v2 remains explicit**: evaluation is not run implicitly; users can enable run-time evaluation in launcher workflow or run eval-v2 externally and then reload Run Intelligence. - **New files**: `task_catalog.py`, `run_launcher_builders.py`, `run_launcher_page.py` (created); `service_adapters.py`, `app.py` (modified). ### Phase 5 Status (2026-06) - **Phase 5 live execution observability is complete**: the Run Launcher now includes a live execution panel driven by persisted `run_events.jsonl` and `run_manifest.json`. - **Execution path remains shared-service only**: Streamlit continues to execute tasks via `RunExecutionService.execute_selected_tasks()` with both `progress_callback` and `event_callback` wired. - **Synchronous execution model is explicit**: the UI clearly states that execution runs synchronously in the current Streamlit session; no background queue/server/cancellation/pause-resume was introduced. - **Live run visibility is additive**: latest event card, current-task indicator, per-task status table, recent event tail, failure-event panel, and manual refresh for the latest run artifacts. - **Manifest correctness bug is fixed**: final `run_manifest.json` now preserves terminal per-task statuses (completed/failed) instead of being overwritten by stale pending states. - **Artifact/event compatibility is preserved**: existing run event schema and coarse event types remain unchanged; no breaking reader changes. - **No automatic eval-v2 trigger**: launcher behavior still writes run artifacts only and requires explicit evaluation execution. ### Phase 6 Status (2026-06) - **Phase 6 implements External Benchmark Dataset Intake** (branch: `feature/observatory-phase-06-custom-task-intake`): the Run Launcher now has a second tab — “External Benchmark Dataset” — that accepts any local benchmark-compatible dataset root, validates it, and executes selected tasks through the same `RunExecutionService` service boundary. Free-form custom task intake is deferred to Phase 7. - **Validation is safety-first**: the validator rejects remote URLs, system directories, home directory, and root; requires `task.json` with `task_id` and `question`; `difficulty` is optional and defaults to `"unknown"`. - **Execution path is identical to approved config mode**: the external root is applied as a temporary in-memory config override (`dataclasses.replace()`); source dataset is never mutated and artifacts are written to the configured output directory. - **No subprocess, CLI, or direct runner calls**: the external tab calls `RunExecutionService.execute_selected_tasks()` exclusively, identical to the approved-config path. - **Duplicate Streamlit key collision is fixed**: `_render_live_execution_panel()` now takes a `mode` parameter (`"approved"` / `"external"`) so both tabs can render in the same session without key collision. - **Stale run state is cleared on new run start**: `_clear_launcher_run_state()` zeroes prior `last_result`, `live_events`, `live_manifest`, `last_run_output_dir`, `progress`, and `error` before each run. - **New files**: `external_dataset_validator.py`, `test_external_dataset_validator.py` (created); `task_catalog.py`, `run_launcher_page.py`, `service_adapters.py`, `benchmark/dataset.py` (modified). ### Phase 16 Status (2026-06 — Complete) - **Phase 16 implements Live Trace Flow** (branch: `feature/observatory-phase-16-live-dag-replay`): a new "⚡ Live Trace Flow" tab provides visual execution story built from trace-native step/action events with future human-in-the-loop pause/replan compatibility. - **Trace-native backbone**: uses `step_index` + `action` as primary sequence; phase serves as badge/grouping only; agent names appear in separate overlay (not main DAG). - **Execution Story mapping**: translates raw actions to presentation labels (e.g., `planner` → "Plan formulation") via comprehensive `ACTION_STORY_MAP` while preserving raw action data. - **Future HITL compatibility**: all nodes include inert future fields (`checkpoint_id`, `can_pause`, `can_resume`, `waiting_for_human`, `human_input_required`, `replan_candidate`) defaulting to None/False. - **Read-only observability**: strictly read-only; no pause/resume/execution control; no subprocess/CLI/runner calls from this page. - **Dual data source support**: builders work with both live events (future) and final `trace.json` (current); no step event instrumentation added in Phase 16. - **Final Trace Handoff**: provides buttons to open existing Reasoning DAG and Time-Travel Replay pages after task completion. - **UI sections**: Current Execution Story, Visual Step DAG, Live Replay Timeline, Agent Reasoning Overlay, Raw Event Stream, Final Trace Handoff, Future HITL Note. - **Artifact integrity preserved**: strictly read-only except safe auto-refresh; no mutations to `trace.json`, `run_events.jsonl`, or any execution/evaluation artifacts. - **Test coverage**: 24 builder tests + 2 page tests passing (plus 13 skipped file content checks). - **New files**: `live_trace_flow_builders.py`, `live_trace_flow_page.py`, `test_live_trace_flow_builders.py`, `test_observatory_live_trace_flow_page.py`, `PHASE16_LIVE_TRACE_FLOW.md` (created); `app.py` (modified: added tab 5). ### Phase 16b Status (2026-06 — Complete) - **Phase 16b extends Live Trace Flow with Start Run + Live Task Selection with non-blocking execution**: users can now start multi-task runs directly from the Live Trace Flow page and dynamically inspect any task during execution; UI returns immediately after starting run. - **Non-blocking execution model**: new `BackgroundRunController` executes runs in background thread; UI returns immediately with run_id and run_dir; users can select and inspect tasks while execution continues. - **Compact Start Run panel**: config selector (`Configuration File :`), attach-to-existing-run row (`Attach to Existing Run (no selection will create a new run) :`) with `(Create new run)` default, task multiselect (max 10), Start Run button; max-workers input removed (fixed at 1). - **Live task selector with status**: task dropdown (`Select Task to Inspect / View Live / Replay`) shows live status indicators (▶️ running, ✅ completed, ❌ failed, ⏳ queued) derived from `run_manifest.json`, `run_events.jsonl`, and artifacts; auto-defaults to the running task when one exists (user override supported). - **Condensed run status line**: replaces separate Run Status block; single `st.info()` line below submission message — `Run Status: | ▶️ Running: N | ✅ Completed: N | ❌ Failed: N | ⏳ Queued: N`. - **Live / Replay pane (left, 50%)**: clickable step buttons; each button shows ` Step N | | - | Duration - `; caption below each button shows ` | Action - `; task start/end lifecycle rows bookend the step list. - **Step Output Inspector pane (right, 50%)**: fully click-driven — governed solely by button click in Live/Replay; no dropdown selector; inspector snapshot refreshes only at task start/end transitions (not every poll cycle); `Thought:` label renamed to `Action:`. - **Agent Reasoning Overlay placement**: rendered above Status Commentary (after Step Output Inspector and before narrative scroll). - **Key findings untruncated**: `extract_key_values()` in `live_trace_flow_builders.py` returns full values without the 220-char truncation cap. - **500 ms polling**: live fragment polls every 500 ms (`refresh_interval = 0.5`) instead of 1 s; live status caption shows `auto-refreshing every 500ms`. - **Auto-refresh always-on**: removed user-facing auto-refresh toggle; refresh is fully system-driven based on run/task activity signals. - **Conditional rendering based on task state**: queued tasks show waiting indicator; running tasks show live progress; completed tasks show full trace flow; failed tasks show failure details + partial trace. - **Duplicate execution prevention**: `BackgroundRunController` prevents concurrent runs; guards against accidental re-execution on page refresh. - **Service architecture compliance**: execution through `BackgroundRunController` → `RunExecutionService` with progress/event callbacks; no subprocess, CLI, or direct runner calls from Streamlit. - **Phase 16 features preserved**: all read-only observability features (trace-native backbone, execution story, HITL schema) remain intact for completed tasks. - **Test coverage**: 77 builder + page tests passing (25 skipped); test assertions updated to track column ratios, layout changes, and removed controls. - **New Phase 16b components**: `BackgroundRunController` (application layer), 3 new builders (`build_run_task_status_index`, `build_task_selector_options`, `derive_selected_task_state`). - **Modified files**: `background_run_controller.py` (NEW), `live_trace_flow_builders.py` (added builders, key-findings untruncated), `live_trace_flow_page.py` (non-blocking execution, full UI refinement), `test_background_run_controller.py` (NEW), `test_live_trace_flow_builders.py`, `test_observatory_live_trace_flow_page.py`. ### Phase 17 Status (2026-06 — Complete) - **Phase 17 implements Guided Live Trace Flow (HITL)** (branch: `feature/observatory-phase-17-guided-trace-hitl`): adds a separate `🧭 Guided Live Trace Flow (HITL)` page/tab and keeps existing `⚡ Live Trace Flow` unchanged. - **Start-panel behavior**: the Guided page now uses a single `Start Run` action with a default-selected `Guided Run (HITL)` checkbox. When selected, the run uses planner-review HITL; when unselected, the same page launches the task flow without planner checkpoint pauses. - **Single checkpoint scope**: guided runs support one checkpoint type only — `planner_review` — reached only after `planner` and before downstream execution/review actions. - **Guided policy at run start**: HITL-enabled launches use `execution_mode="guided"`, `hitl_enabled=True`, planner checkpoint policy metadata, and forced `max_workers=1`; unchecked launches use the same page but run without HITL checkpoint policy. - **timeout auto-approve policy**: planner checkpoints use `timeout_seconds=60` with `default_action_on_timeout="continue"`; if no response arrives in 60 seconds, the original plan is auto-approved by timeout policy and execution resumes. - **Service/runner-owned HITL**: Streamlit remains UI-only; checkpoint creation/waiting, intervention persistence, delta persistence, and replan application are implemented in application/runner layers. - **Persistence layout**: task-local artifacts are written under `//checkpoints/*.json`, `interventions/*.json`, and `deltas/*_delta.json`, while lifecycle events are emitted into `run_events.jsonl`. - **Decision paths implemented**: - Approve: persists intervention, emits `checkpoint_approved` and `task_resumed`, continues unchanged. - Revise: persists intervention + steering instruction, emits `intervention_submitted` / `state_invalidated` / `replan_started` / `replan_completed` / `task_resumed`, stores revised plan and deterministic intervention delta. - Cancel: persists intervention, emits `checkpoint_cancelled` + `task_cancelled`, marks task as `cancelled` with reason `cancelled_by_human` (not failed). - Timeout auto-approve: emits `checkpoint_timeout_started` / `checkpoint_timed_out` / `checkpoint_auto_approved` / `task_resumed`, persists system intervention with `decision="timeout_auto_approve"` and `submitted_by="system_timeout"`. - **Trace-level HITL evidence**: final `trace.json` now contains additive `hitl` block with original plan, intervention, revised plan, delta metadata/path, and policy snapshot. - **Recovery behavior**: browser refresh can redisplay pending checkpoints from persisted artifacts while process is alive; full process-death durable resume is not claimed. - **Phase separation clarity**: Phase 17 is live in-execution HITL; Phase 8 stays advisory post-hoc; Phase 9 stays isolated guided rerun workflow. - **New files**: `domain/intervention_models.py`, `application/hitl_checkpoint_controller.py`, `observatory/guided_trace_flow_builders.py`, `observatory/guided_trace_flow_page.py`, and new tests for controller, graph checkpoint behavior, guided builders, and guided page. ### Phase 7 Status (2026-06 — Complete) - **Phase 7 implements Free-form Custom Task Intake** (branch: `feature/observatory-phase-07-freeform-custom-task-intake`): a third tab — "Custom Task" — is added to the Run Launcher. Users enter a question, optional context, optional uploaded files, and expected output type. The task is persisted, materialized into a benchmark-compatible synthetic dataset, and executed through the existing shared `RunExecutionService`. No new runner is created. - **Safe file handling**: uploaded files are sanitized for path traversal, size-limited to 50 MB, extension-blocked for unsafe types (`.py`, `.sh`, etc.), and hashed with SHA256. Files are never executed — they are context/data only. - **Materialization structure**: custom tasks materialize into `artifacts/custom_tasks//` with `task_definition.json` (provenance), `context/` (always created, `knowledge.md` optional), and a benchmark-compatible `dataset//` folder. The `context/` directory always exists; `knowledge.md` is created only when context text is provided. - **Execution is identical to approved-config and external-dataset modes**: custom tasks use `get_run_execution_service_with_custom_task()` which applies the synthetic dataset as an in-memory config override. No subprocess, CLI, or direct runner invocation. - **No eval-v2 correctness scoring**: custom tasks have `has_gold_data=False` by default; eval-v2 scoring is unavailable unless gold data is added later. Trace, prediction, live execution, and artifact inspection remain available. - **Domain models**: `TaskDefinition`, `ExpectedOutputSpec`, `SourceDescriptor`, `UploadedFileSpec` in `task_models.py` support JSON serialization and optional context/files. - **Tests**: 80 Phase 7 unit and integration tests cover domain models, repository file handling, service logic, materialization, provenance, and end-to-end workflows. All existing Phase 0–6 tests remain passing. - **New files**: `domain/task_models.py`, `repositories/custom_task_repository.py`, `repositories/filesystem_custom_task_repository.py`, `application/custom_task_service.py`, `tests/unit/domain/test_task_models.py`, `tests/unit/repositories/test_filesystem_custom_task_repository.py`, `tests/unit/application/test_custom_task_service.py`, `tests/integration/test_phase7_custom_task_launcher_integration.py` (created); `observatory/service_adapters.py`, `docs/implementation/DECISIONS.md` (DEC-011), `docs/implementation/IMPLEMENTATION_LEDGER.md` (modified). ### Phase 8 Status (2026-06 — Complete) - **Phase 8 implements Checkpoint Review and Advisory Human Steering** (branch: `feature/observatory-phase-08-checkpoint-steering`) as a conservative additive capability. - **Checkpoint timeline/detail UI added**: task-level "Checkpoint Review" tab derives checkpoints from existing `trace.json` and optional `run_events.jsonl` without altering execution history. - **Advisory-only persistence model**: users can save human review notes and advisory steering instructions to task-local `checkpoint_annotations.json`. - **No execution intervention introduced**: no pause/resume, no mid-run steering, no runner state mutation, no execution trigger from this page. - **Artifact immutability preserved**: `trace.json`, `prediction.csv`, `task_definition.json`, and eval-v2 artifacts remain unchanged. - **Compatibility preserved**: approved benchmark, external dataset, and custom-task runs all use the same checkpoint page behavior. - **Validation status**: new Phase 8 tests pass and full repository suite passes (`397 passed`). ### Phase 9 Status (2026-06 — Complete) - **Phase 9 implements Guided Rerun Planning and Execution** (branch: `feature/observatory-phase-09-guided-rerun-planning`): converts Phase 8 advisory steering instructions into executable rerun plans with approval workflow and isolated execution. - **Rerun plan workflow**: users select steering instructions, define objectives/constraints, generate plans (draft status), approve plans, execute as new isolated custom task runs through Phase 7 path. - **Source artifact immutability preserved**: original run artifacts remain unchanged; execution creates new timestamped runs with guidance context injected into synthetic task datasets. - **Question resolution with dataset fallback**: `resolve_source_task_question()` finds original questions from task definitions, trace files, or dataset input directories when not in run artifacts. - **Context file copying**: guided reruns materialize with original task's data files (CSV, JSON, DB) copied into custom task context so the agent has access to all source data. - **Safe execution path**: uses Phase 7 CustomTaskService to materialize synthetic datasets and RunExecutionService for execution; no subprocess, CLI, or direct runner invocation. - **Artifact schema**: `rerun_plans.json` per task with plan metadata, provenance (source run/task/checkpoints/instructions), status lifecycle (draft→approved→executed), execution metadata (run_id, task_id, timestamps). - **Test coverage**: 52 Phase 9 tests (10 models, 21 builders, 6 execution, 15 page tests); full suite passes (`449 passed`). - **New files**: `rerun_plan_models.py`, `rerun_plan_builders.py`, `rerun_plan_execution.py`, `rerun_plan_page.py` (created); `app.py` (modified: added 7th tab). ### Phase 10 Status (2026-06 — Complete) - **Phase 10 implements Complete Original-vs-Guided Rerun Comparison and Local Ask This Run / Ask This Comparison** (branch: `feature/observatory-phase-10-rerun-comparison-ask-this-run`): provides read-only structured comparison of original vs guided runs with interactive artifact-grounded Q&A. - **Comparison dimensions**: prediction/output (byte-identical check, row/column delta), trace/trajectory (step count, stage distribution), tool usage (call delta, unique tools), runtime (execution time delta, status), confidence/failure/provenance/critic (with graceful unavailable states), eval-v2 metrics (score delta, precision/recall with graceful unavailable). - **Steering influence summary**: heuristic keyword matching with non-causal language ("possible alignment detected", "evidence suggests", never "caused" or "fixed"); clearly marked as heuristic, not causal proof. - **Ask This Run / Ask This Comparison**: local rules-based Q&A with 20+ supported intents (FINAL_ANSWER_CHANGED, STEP_COUNT_CHANGED, TOOL_USAGE_CHANGED, RUNTIME_CHANGED, FAILURE_CHANGED, STEERING_USED, STEERING_REFLECTED, EVAL_AVAILABLE, SCORE_IMPROVED, WHAT_CHANGED, INSPECT_NEXT, etc.); unsupported questions return exact fallback text: "I cannot answer that from the available local artifacts." - **Saved comparison reports**: users can save computed comparison reports to additive `rerun_comparisons.json` artifact; previously saved reports displayed in table. - **Artifact integrity preserved**: strictly read-only comparison via pure builder functions; no execution, no mutations (except additive rerun_comparisons.json when user clicks Save), no subprocess, no external LLM/API calls; hash-based tests verify artifact integrity. - **Graceful degradation**: comparison handles missing artifacts (trace, prediction, eval) with clear messaging; eval-v2 artifacts optional. - **Artifact schema**: `rerun_comparisons.json` per task with schema_version, reports array containing full comparison metadata (source lineage, artifact availability, all comparison dimensions, steering summary). - **Test coverage**: 70+ Phase 10 tests across models, builders, Ask Q&A, UI, and integration; artifact integrity verified with hash-based mutation tests. - **New files**: `rerun_comparison_models.py`, `rerun_comparison_builders.py`, `ask_artifact_builders.py`, `rerun_comparison_page.py`, 5 test files (created); `app.py` (modified: added 8th tab after Guided Rerun Plan). ### Phase 11 Status (2026-06 — Complete) - **Phase 11 implements Run-Level Cohort Guided Rerun Evaluation** (branch: `feature/observatory-phase-11-cohort-guided-evaluation`): aggregates existing guided rerun comparisons across a run into cohort-level metrics and provides local artifact-grounded cohort Q&A. - **Cohort discovery**: scans task directories in source run, finds executed rerun plans from `rerun_plans.json`, resolves guided artifacts using Phase 10 pairing logic, supports multiple executed plans per source task, graceful degradation for missing artifacts. - **Aggregated metrics**: prediction changed count/rate, score delta stats (mean/median/min/max when eval exists, improved/degraded/neutral classification), step/tool/runtime delta stats, failure rate comparison (source vs guided, both succeeded/failed, source-only/guided-only failed), eval artifact availability, missing artifact counts, steering instruction coverage (unique instruction count, reuse count, possible alignment count/rate with heuristic language). - **Integration**: extends existing Run Intelligence page (Section 9, not a new tab); renders only when executed guided reruns detected; conditional visibility based on view level (Summary/Detailed/Research); compact info in expander when no reruns exist. - **UI components**: cohort overview cards, source→guided pairing table, eval unavailable notice, steering coverage table (Detailed/Research), missing artifacts panel, representative examples (Research), Ask This Cohort text input, Save Cohort Report button. - **Ask This Cohort**: local rules-based Q&A with 12+ cohort intents (cohort_size, prediction_changed_count, improved_count, degraded_count, mean_score_delta, runtime_overhead, failure_rate_change, most_common_steering, eval_coverage, missing_artifacts, task_changed_most); unsupported questions use same safe fallback as Phase 10. - **Non-causal language**: all summaries and answers avoid causal claims; disclaimer: "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."; steering alignment marked as "heuristic signal", "possible alignment", "observed association" (never "caused", "fixed", "proved"). - **Artifact integrity preserved**: strictly read-only aggregation; no execution, no eval-v2 auto-run, no mutations except optional `cohort_guided_comparison.json` (regenerable snapshot, overwrite-latest); hash-based tests verify no mutations to trace.json, prediction.csv, rerun_plans.json, rerun_comparisons.json, checkpoint_annotations.json, run_manifest.json, run_events.jsonl, eval-v2 artifacts. - **Graceful degradation**: score delta stats exclude missing eval artifacts (not counted as zero), runtime requires `runtime_seconds` in trace.json (optional field), missing comparisons degrade gracefully with clear limitations. - **Reuse of Phase 10 assets**: uses Phase 10 `find_executed_rerun_plans()`, `resolve_comparison_pair()`, `build_artifact_availability()`, `compare_predictions()`, `load_trace_summary()` to build individual pairings; aggregates via pure cohort builders. - **Artifact schema**: optional `cohort_guided_comparison.json` per run with schema_version, run_id, created_at, summary (all cohort metrics), pairing array (list of PairingEntry), steering_coverage, disclaimer. - **Test coverage**: 69 Phase 11 tests (11 models, 17 builders, 30 Ask cohort, 11 integration); Phase 10 regression verified (99 passed); artifact integrity verified with SHA256 hash comparison before/after; no new folders/events verified. - **New files**: `cohort_guided_models.py`, `cohort_guided_builders.py`, 4 test files (created); `run_intelligence_page.py`, `ask_artifact_builders.py` (modified: extended with cohort support). ### Phase 12 Status (2026-06 — Complete) - **Phase 12 implements Evidence Pack and Statistical Readiness** (branch: `feature/observatory-phase-12-evidence-pack-and-statistical-readiness`): consolidates Phases 8–11 into a judge-facing Evidence Pack with cross-phase narrative, evidence hierarchy, statistical readiness assessment, claim safety framework, and exportable markdown/JSON reports. - **Cross-phase narrative**: structured timeline showing Phase 8 Checkpoint Review → Phase 9 Guided Rerun Planning → Phase 10 Original-vs-Guided Comparison → Phase 11 Run-Level Cohort Evaluation with key outcomes and artifacts created at each phase. - **Evidence hierarchy**: classifies evidence by level (primary: trace.json, prediction.csv, task_metrics.csv, trajectory.csv, tool_calls.csv, run_manifest.json, run_events.jsonl; derived: checkpoint_annotations.json, rerun_plans.json; comparison: rerun_comparisons.json; aggregate: cohort_guided_comparison.json, evidence_pack.json, evidence_pack.md). - **Statistical readiness assessment**: conservative N≥20 threshold and eval coverage ≥80% requirement for significance testing; defaults to descriptive_only for smaller N; provides explicit limitations and warnings; recommended test: wilcoxon_signed_rank when sufficient, descriptive_only otherwise. - **Claim safety framework**: explicit lists of allowed observational claims vs unsupported causal claims; required disclaimers: "Observed associations do not establish causation", "Guided reruns differ in timing, model state, and execution context", "Results may not generalize"; confidence level: observational or descriptive. - **Export**: markdown and JSON export on explicit button click (evidence_pack.md, evidence_pack.json); exports are regenerable and overwrite-latest; markdown includes all required sections (Executive Summary, Timeline, Evidence Hierarchy, Statistical Readiness Assessment, Claim Safety Report, Key Findings, Representative Examples, Limitations, Disclaimers, Artifact Manifest). - **Integration**: extends Run Intelligence page (Section 10, not a new tab); visible in Detailed and Research view levels; statistical readiness card, claim safety summary, key findings preview, export buttons, optional markdown preview in Research mode. - **Graceful degradation**: handles missing guided reruns, missing eval artifacts, missing cohort reports with clear limitations; evidence hierarchy marks artifact availability; statistical readiness flags insufficient sample size. - **Artifact integrity preserved**: strictly read-only except for explicit export; no execution, no eval-v2 auto-run, no mutations to source artifacts (trace.json, prediction.csv, checkpoint_annotations.json, rerun_plans.json, rerun_comparisons.json, cohort_guided_comparison.json, run_manifest.json, run_events.jsonl); only evidence_pack.json and evidence_pack.md created on export. - **Non-causal language**: markdown rendering includes causation disclaimers; unsupported claims section lists causal language variants; allowed claims use observational language only ("observed", "associated with", "followed by", never "caused", "fixed", "proved"). - **Test coverage**: 72 Phase 12 tests (25 models, 22 builders, 17 exporters, 8 integration); all Phase 8-11 regression verified; artifact integrity verified with SHA256 hash comparison; no new folders/events verified; safety scan confirms no subprocess, no external API calls. - **New files**: `evidence_pack_models.py`, `evidence_pack_builders.py`, `evidence_pack_exporters.py`, 4 test files (created); `run_intelligence_page.py`, `DECISIONS.md`, `IMPLEMENTATION_LEDGER.md`, `Overview.md` (modified). --- ## 2. Pipeline Architecture ```mermaid graph TD START([Start]) explore["Phase 0: Explore Data
Parallel file I/O, schema inspection"] semantic["Phase 0.5: Semantic Extraction
knowledge.md parsing, doc/*.md extraction (profile-only for large docs)"] plan["Phase 1: Planner
LLM generates ExecutionPlan JSON"] critic_plan{"Critic: Plan Valid?
LLM validates plan against data"} execute["Phase 2: Executor
LLM generates Python code, sandboxed exec"] error_needs_replan{"Structural Error?
_error_needs_replan heuristic"} critic_execute{"Critic: Result Valid?
LLM validates output shape + semantics"} END([End]) doc_phase2["Phase 1.5: Doc Phase 2
Planner-gated targeted re-extraction"] START --> explore explore --> semantic semantic --> plan plan --> critic_plan critic_plan -->|valid| doc_phase2 doc_phase2 --> execute critic_plan -->|"retry_plan (max 2)"| plan execute -->|success| critic_execute execute -->|"code failed"| error_needs_replan error_needs_replan -->|"yes: KeyError, no such table, etc."| plan error_needs_replan -->|"no: transient error"| execute error_needs_replan -->|"retries exhausted"| critic_execute critic_execute -->|retry_exec| execute critic_execute -->|replan| plan critic_execute --->|"✅ Happy Path (confidence ≥ 0.7)"| END critic_execute --->|"❌ Failure / retry limit"| END style START fill:#e1f5fe style END fill:#c8e6c9 style explore fill:#f3e5f5 style semantic fill:#fce4ec style doc_phase2 fill:#fce4ec style plan fill:#e8eaf6 style execute fill:#e8eaf6 style error_needs_replan fill:#ffccbc style critic_plan fill:#fff9c4 style critic_execute fill:#fff9c4 ``` **Total execution budget:** 5 attempts across execute + replan cycles (`max_total_attempts = 5`). --- ## 3. Phase-by-Phase Detail ### Phase 0: Explore Data (`explore_data_node`) **What:** Reads all files in the context directory in parallel using `ThreadPoolExecutor(max_workers=8)`. **Why:** The planner and executor need to see schemas, sample rows, and data types before they can write correct queries. Parallel I/O cuts exploration time for multi-file tasks. | File type | Action | |-----------|--------| | `.csv` | Read columns with dtypes, row count, 3 sample rows. Auto-build into `_auto.db` SQLite. | | `.json` | Preview first 10K chars. | | `.db`/`.sqlite` | `PRAGMA table_info`, `CREATE TABLE` SQL, 3 sample rows per table. | | `.md` (knowledge) | Full content included in `data_exploration`. | | `.md` (doc/*) | **Truncated to 2,000 chars** in `data_exploration` (full content extracted in Phase 0.5). | **Why truncate doc/*.md?** These narrative files can be 85K–286K chars. Including them raw in the executor prompt causes Azure API 500 errors from token overflow. Since they're extracted into `_auto.db` tables in Phase 0.5, the executor only needs a short preview. ### Phase 0.5: Semantic Extraction (`semantic_extraction_node`) **What:** Two sub-tasks: 1. **`knowledge.md` parsing** (rule-based, no LLM): Extracts entities, fields, relationships, synonyms, business rules, value maps, threshold rules. Produces a `SemanticContext` object. 2. **`doc/*.md` extraction** (LLM-based, chunked batching): Converts narrative prose files into structured tables loaded into `_auto.db`. #### Doc Extraction Pipeline ``` doc/*.md content │ ▼ Size check: len(content) >= 40KB? │ ├── YES (large doc) ────────► Two-Phase Path (doc_phases.py) │ │ │ ▼ │ Phase 1: Profile + Sample │ _doc_structure_probe() (1 LLM call) │ _extract_sample_records() (1 LLM call, first 4K chars) │ → 5 sample rows into _auto.db │ → profile stored in state.doc_profiles │ │ │ ▼ │ [Planner runs with sample data] │ │ │ ▼ │ Phase 2: Planner-Gated Full Extraction │ _columns_needed_by_plan() — stem+exact match │ _get_doc_chunks_prioritized() — split by ## or ¶ │ _pack_batches(chunks, 12KB) │ N LLM calls (up to 20 batches) │ Merge by entity ID (first-non-null) │ → Replace sample data in _auto.db │ └── NO (small doc) ─────────► Eager Path (semantic.py) │ ▼ _parse_md_sections() │ ▼ _generate_keywords() │ ▼ _filter_sections_by_keywords() (only for docs > 80K, only if > 30% reduction) │ ▼ _chunk_section_by_entities() │ ▼ Structure Probe (1 LLM call) │ ├── flat path → Batch Extraction (N calls, ~20K/batch) └── multi-section → _extract_multisection_doc() │ ▼ Quality Gate + _focused_reextraction() │ ▼ load_doc_records_to_sqlite() → _auto.db ``` **Why chunked extraction?** A single LLM call on a 286K char Laboratory.md misses ~50% of records. Batching at ~20K chars gives the LLM manageable context windows. **Why a single Structure Probe (replaces standalone Schema Detection)?** The old pipeline made one LLM call to detect the schema, then assumed a single flat record stream. Docs like `task_418` interleave multiple per-entity sections (one per patient, lab visit, etc.) and need merging by an `entity_id` key. The probe folds schema detection + layout classification + merge-key inference + record-count estimation into one call, sampling head / middle / tail (`_PROBE_SAMPLE_CHARS=3000` each, more for larger docs via `_PROBE_EXTRA_SAMPLE_PER_BYTES`). Downstream code branches on `is_multisection`. **Why multi-section first-non-null merge?** When the same entity has fragments scattered across sections (e.g., demographics in §1, labs in §3, diagnosis in §5), per-section extraction yields partial rows. Merging by `merge_key` and taking the first non-null per column reconstructs the full record without overwriting good values with later nulls. **Why post-extraction quality gates + focused re-extraction?** Even with good chunking, some columns come out sparsely pectiulated (LLM skipped them) or the total record count falls well short of the probe's estimate. `_coverage_stats` computes per-column null rates and total count; `_quality_gate_targets` picks columns above `_QG_NULL_RATE` (0.5) or flags low coverage (< `_QG_COVERAGE_RATIO`=0.7 of estimate). `_focused_reextraction` then makes a single targeted retry pass aimed at those columns / missing records. Gating is skipped for tiny extractions (`_QG_MIN_RECORDS_FOR_GATE=5`) to avoid noise. **Why markdown-aware chunking?** The doc files use `#`/`##`/`###`/`####` headings to organize different data domains (e.g., "Liver Function Panel", "Immunoglobulin Levels", "Coagulation Parameters"). Parsing by heading level: - Produces cleaner entity-level chunks (no split mid-record) - Enables keyword-based section filtering for very large docs - Respects the document's own structure **Why keyword filtering only for large docs?** For extraction, we generally need ALL records (SQL may aggregate the full table). Filtering is only a fallback for docs > 80K chars where processing all sections would exceed the batch budget. **Why the `*` separator handling?** Many doc files use a standalone `*` as a section separator (not a heading). `_chunk_section_by_entities()` splits on `\n\*\s*\n` to handle this. #### Question Rewriting After semantic extraction, the question is rewritten using `rewrite_query()` to: - Resolve ambiguous column names (e.g., "views" → `ViewCount`) - Map natural language to schema terms - Include `data_exploration` context so the LLM knows what columns exist **Why?** Task_259 failed because "views" was mapped to `Score` instead of `ViewCount`. The rewriter uses schema context to make correct mappings. ### Phase 1: Planner (`planner_node`) **What:** LLM generates a structured `ExecutionPlan` JSON with steps, required sources, output columns, expected row count, and reasoning. **Key prompt rules (from `PLANNER_SYSTEM`):** - "posted it last time" → use `LastEditorUserId` not `OwnerUserId` (task_257 fix) - "How many times more X than Y" → compute ratio X/Y, not a count (task_352 fix) - ROW COUNT ESTIMATION: plan must declare "single" or "multiple" expected rows - COMPOUND FILTER LOGIC: multiple conditions must be applied simultaneously, not sequentially (task_355 fix) - Output column discipline: plan declares exact output columns ### Phase 1.5: Plan Critic (`critic_plan_node`) **What:** LLM validates the plan against the data exploration context. Checks column existence, join feasibility, filter logic. **Why a separate critic?** The planner hallucinates column names ~15% of the time. A separate validation pass catches these before wasting an execution attempt. ### Phase 2: Executor (`executor_node`) **What:** LLM generates Python code (pandas + sqlite3) that implements the plan. Code is executed in a sandboxed `exec()` with a restricted global namespace. **Key features:** - SQL-first strategy: prefer `_auto.db` SQL over raw file reads - Code must produce a `result` DataFrame variable - Captured `stdout` from `print()` statements is included in critic context ### Phase 2.5: Execution Critic (`critic_execute_node`) **What:** LLM validates the execution result via the 12-item forced checklist (Change B) plus an independent sanity check #13 (Change F). After the LLM critic returns, deterministic post-checks run: - Output column count vs expected (auto-reject if > `max(3, 1.5× expected)`) - Row count vs expected ("single" or "multiple") — **soft warning only**, not auto-reject - **Numeric Guards (Change E)** — NaN/inf, percent out of `[0,100]`, negative count, ratio-as-int-zero. Flags are always surfaced in the trace; a fired guard downgrades a previously-valid result so the executor retry loop fires once. **Why soft row count warning?** task_355 was auto-rejected because the plan said "single" but the result had 1 row — the auto-reject logic was too strict. Changed to a log warning that the critic LLM can consider. **Why structural numeric guards instead of more prompt rules?** See §10 — prompt-level fixes saturated around λ=0.665, and any further rule specific enough to fix one failure was overfitting to specific failing tasks. ### Error Triage (`_error_needs_replan`) **What:** Heuristic that classifies execution errors as structural (needs replan) vs transient (retry same code). | Error pattern | Action | |---------------|--------| | `KeyError`, `no such table`, `FileNotFoundError` | → Replan | | `OperationalError`, `empty DataFrame` | → Replan | | Other errors, < 2 attempts | → Retry execution | | Any error, ≥ 2 attempts | → Force replan | --- ## 4. Key Constants ### Timeouts & Parallelism (`main.py`) | Constant | Value | Why | |----------|-------|-----| | `TASK_TIMEOUT_SECONDS` | `510` | Per-task SIGALRM timeout. Increased from 300 to accommodate large doc extraction tasks. | | `MAX_WORKERS` | `4` | ThreadPoolExecutor workers for parallel task execution. Reduced from 8 to avoid API saturation causing timeouts (cross-validation adds 1 extra LLM call per easy/medium task). | ### Retry Budgets | Constant | Location | Value | Why | |----------|----------|-------|-----| | `max_plan_retries` | `state.py` | `2` | Max planner attempts before proceeding with last plan. | | `max_exec_retries` | `state.py` | `3` | Max executor attempts per plan. | | `max_total_attempts` | `graph.py` | `5` | Total budget across execute + replan cycles. | | `_MAX_RETRIES` | `model.py` | `3` | API call retries for transient HTTP errors. | | `_RETRY_BASE_DELAY` | `model.py` | `2.0s` | Exponential backoff base (2s, 4s, 8s). | | `_RETRY_ERRORS` | `model.py` | `(500, 502, 503, 504, 429)` | HTTP status codes that trigger retry. | ### Data Exploration (`nodes.py`) | Constant | Value | Why | |----------|-------|-----| | `DEFAULT_SAMPLE_ROWS` | `3` | Sample rows shown per table in data_exploration. Keeps prompt small. | | Doc truncation limit | `2,000 chars` | Max doc/*.md content in data_exploration. Full content is in _auto.db. | ### Doc Extraction (`semantic.py`) | Constant | Value | Why | |----------|-------|-----| | `max_chars_per_batch` | `20,000` | Chars per extraction batch. Balances accuracy vs LLM context limits. | | `max_batches` (default) | `8` | Default max batches per doc file. | | `per_doc_max_batches` | `max(3, 16 // num_doc_files)` | Dynamic budget: 1 file→16, 2 files→8, 3+→5. Ensures total LLM calls stay bounded. | | `LARGE_DOC_THRESHOLD` | `80,000 chars` | Docs larger than this may have keyword-based section filtering applied. | | Keyword filter threshold | `70%` | Only use filtered sections if they're < 70% of original size (i.e., > 30% reduction). | | Schema detection sample | `3 chunks, 6000 chars` | Sample text sent for legacy schema detection LLM call (now superseded by Structure Probe). | | `_PROBE_SAMPLE_CHARS` | `3,000` | Chars per head/middle/tail probe window. | | `_PROBE_MIN_SAMPLES` | `3` | Minimum probe samples (head + middle + tail). | | `_PROBE_EXTRA_SAMPLE_PER_BYTES` | `30,000` | Add one extra probe sample per ~30K chars of doc length. | | `_QG_COVERAGE_RATIO` | `0.7` | If `actual_records < estimate × 0.7`, trigger focused re-extraction. | | `_QG_NULL_RATE` | `0.5` | If a column's null-rate ≥ 50%, target it for focused re-extraction. | | `_QG_MIN_RECORDS_FOR_GATE` | `5` | Skip quality gating entirely below this many records. | ### Deadlines & HTTP Timeout | Constant | Location | Value | Why | |----------|----------|-------|-----| | `LLM_HTTP_TIMEOUT` | `agents/model.py` | `90s` (env `LLM_HTTP_TIMEOUT`) | Per-request HTTP timeout on both `OpenAI(...)` and `AzureOpenAI(...)` clients. Default SDK timeout is effectively unbounded, which caused stalled doc-extraction calls to consume the full task budget. | | `DOC_EXTRACTION_BUDGET_SECONDS` | env (parsed in `nodes.py`) | unset (off) | Optional soft deadline for the doc-extraction phase. When set, `semantic_extraction_node` computes a monotonic deadline and passes it as `deadline_monotonic` into `extract_doc_records_chunked`, which checks it at each safe abort point (between batches, between sections, before focused retry). | ### Output Validation | Constant | Location | Value | Why | |----------|----------|-------|-----| | Column count cap | `nodes.py` | `max(3, 1.5 × expected)` | Auto-reject results with too many columns (prevents data dumps). | | Row count check | `nodes.py` | Soft warning | Log warning if row count mismatches plan's "single"/"multiple" estimate. | ### LLM Call Tracking Each step with `token_usage` increments the phase's `llm_calls` counter. Trace includes: - **Per-phase LLM calls**: `explore_llm_calls`, `planner_llm_calls`, `critic_plan_llm_calls`, `execute_llm_calls`, `critic_execute_llm_calls` - **Total LLM calls**: `total_llm_calls` (sum across all phases) - Stored in `stage_metrics[phase]["llm_calls"]` and `comprehensive_metrics["llm_calls"]` --- ## 5. LLM Prompts | Prompt | File | Purpose | |--------|------|---------| | `PLANNER_SYSTEM/USER` | `prompts.py` | Generate ExecutionPlan JSON | | `CRITIC_PLAN_SYSTEM/USER` | `prompts.py` | Validate plan against data | | `EXECUTOR_SYSTEM/USER` | `prompts.py` | Generate executable Python code | | `CRITIC_EXECUTE_SYSTEM/USER` | `prompts.py` | Validate execution result | | `QUERY_REWRITE_SYSTEM` | `semantic.py` | Rewrite question with schema context | | `DOC_SCHEMA_DETECTION_SYSTEM/USER` | `semantic.py` | Legacy schema detection (kept as fallback) | | `DOC_STRUCTURE_PROBE_SYSTEM/USER` | `semantic.py` | Combined schema + multi-section layout + merge-key probe (single call) | | `DOC_BATCH_EXTRACTION_SYSTEM/USER` | `semantic.py` | Extract records from doc chunk batch | | `DOC_EXTRACTION_SYSTEM/USER` | `semantic.py` | Legacy single-call extraction (fallback) | --- ## 6. File Map ``` main.py Entry point, task discovery, parallel execution, SIGALRM timeout src/data_agent_baseline/ ├── agents/ │ └── model.py OpenAI/Azure API adapters with retry logic ├── langgraph_agent/ │ ├── graph.py Workflow orchestration, retry/replan logic │ ├── state.py AgentState, ExecutionPlan, CriticFeedback dataclasses │ ├── nodes.py All graph nodes (explore, semantic, plan, execute, critics) │ ├── prompts.py All LLM prompt templates │ ├── semantic.py knowledge.md parsing, doc structure probe + chunked/multi-section extraction, quality gates, query rewriting │ ├── normalization.py Output DataFrame normalization │ ├── numeric_guards.py Change E (M3) — deterministic NaN/percent/count/ratio sanity checks (zero LLM cost) │ ├── cross_validate.py Change G — dual-track re-derivation (SQL↔pandas) for easy/medium tasks + Option B query_db injection │ ├── failure_tagger.py Classify per-task failures into buckets + aggregate per-phase timing (used by `tag-failures` CLI) │ └── runner.py Trace JSON + prediction CSV writing configs/ └── react_baseline.container.yaml Runtime configuration ``` --- ## 7. Bug Fixes & Hardening (Chronological) | # | Task/Issue | Root Cause | Fix | |---|-----------|------------|-----| | 1 | task_257 | "posted it last time" → wrong user column | Added `LastEditorUserId` rule to planner + critic prompts | | 2 | Column explosion | Extra columns in output → wrong CSV | Auto-reject if columns > `max(3, 1.5× expected)` | | 3 | task_259 | "views" → `Score` instead of `ViewCount` | Query rewriter with schema context in `semantic.py` | | 4 | task_352 | "how many times more" → count instead of ratio | Ratio idiom rule in planner + rewriter prompts | | 5 | Timeouts | Tasks timing out at 300s | SIGALRM timeout, parallel execution, difficulty sorting | | 6 | task_355 | Compound filters applied sequentially | Expected row count, compound filter rules in prompts | | 7 | Row count auto-reject | Valid single-row results rejected | Changed to soft warning | | 8 | Doc extraction quality | Single LLM call misses 50%+ records | Chunked batch extraction pipeline | | 9 | task_418 Azure 500 | Prompt too large (286K doc in data_exploration) | Truncate doc/*.md to 2K chars in data_exploration | | 10 | API transient failures | No retry on 500/429 errors | Exponential backoff retry in model.py | | 11 | Doc chunking quality | Flat paragraph splitting ignores markdown structure | Markdown-aware section parsing + keyword filtering | | 12 | task_418 (multi-section docs) | Schema-detection assumed one flat record stream; per-entity sections produced partial rows | Structure Probe + multi-section path with first-non-null merge by `merge_key` | | 13 | task_396 (sparse columns / undercount) | Single extraction pass left some columns mostly null and fell short of probe's record estimate | Post-extraction quality gates (`_QG_COVERAGE_RATIO`, `_QG_NULL_RATE`) + one focused re-extraction pass | | 14 | Stalled LLM HTTP calls | OpenAI/Azure SDK default timeout effectively unbounded → single hung request burned entire task budget | `LLM_HTTP_TIMEOUT=90s` applied to both clients; optional `DOC_EXTRACTION_BUDGET_SECONDS` deadline threaded into `extract_doc_records_chunked` | | 15 | `prediction.csv` schema drift | Old `_raw` columns emitted alongside normalized columns confused downstream evaluation | Commented out `_raw` block in `runner._save_result`; CSV now contains only normalized columns | | 16 | Eval visibility | Hard to tell which buckets / difficulties / phases were costing recall | New `eval-lang` columns (Time, recall>0/=0, difficulty breakdown, zero-recall list) + new `tag-failures` CLI (per-task bucket + per-phase timing) backed by `failure_tagger.py` | | 17 | Prompt rule attention | Bug-patch rules (LastEditorUserId, ratio idiom, naming discipline, scoring rule) were buried mid-prompt and routinely ignored by the LLM | **Change A** — added `SHARED_HEADER` constant prepended to all four SYSTEM prompts (scoring rule, ground-truth hierarchy, answer-shape vocab, naming discipline) so highest-leverage rules land in the top-of-context-window high-attention slot | | 18 | Critic rubber-stamping | Free-form `CRITIC_EXECUTE_SYSTEM` produced inconsistent verdicts and missed recurring failure modes (name-vs-ID, name concatenation, null-filtering, count-vs-list) | **Change B** — replaced with a forced 12-item checklist (`required_columns_present`, `count_vs_list`, `name_not_id`, `name_columns_separate`, `no_null_zero_filter`, `no_head_or_limit`, `ratio_not_count`, `monthly_vs_annual`, `value_encoding_correct`, `ordering_explicit`, `magnitude_reasonable`, `not_empty`) with deterministic verdict rules. Legacy prompt preserved as `CRITIC_EXECUTE_SYSTEM_LEGACY`. **Change C** — `CRITIC_EXECUTE_USER` instructs the critic to walk the checklist in order. Legacy preserved as `CRITIC_EXECUTE_USER_LEGACY`. | | 18a | Change A regressions | Two SHARED_HEADER lines over-fired: `"When in doubt INCLUDE IT"` inflated `output_columns` and broke DISTINCT dedup (task_194); `"what is the X of Y" → single` truncated legitimate multi-row answers (task_22) | **Fix A1** — removed the unconditional INCLUDE-IT line; bias toward minimal `output_columns` retained via the asymmetric scoring text. **Fix A2** — replaced the single-only cue with `"CHECK THE DATA — default to multiple unless explicitly uniquely-keyed"`. | | 18b | Change B over-rejection | Check #2 `count_vs_list` hard-failed *any* `"what is the X of Y"` question whose result had >1 row, again breaking task_22 | **Fix B1** — narrowed hard-fail scope to four explicit aggregation idioms (`"how many"`, `"what percentage"`, `"what is the average/mean/median/ratio"`, `"how many times more/greater"`); everything else degrades to a confidence-only warning. | | 19 | Saturation of pure-prompt fixes | After A/B/C + A1/A2/B1 the system stabilised around λ=0.665 (~60% perfect). The residual value-mismatch tail (NaN denominators, ratio-as-int-zero, percentages > 100, wrong-entity-kind results) could not be addressed by more prompt rules without overfitting to specific failing tasks — earlier `Change D` attempts (D1/D2/D4/D5) cited or strongly implied known task IDs and were rejected. | **Change E (Mechanism 3) — Numeric Guards** + **Change F (Mechanism 2) — Independent Sanity Check** + **Change G — Cross-Validation**. All are intentionally generic (no question-pattern catalogues, no task references). See §10. | | 20 | Executor SQL errors (column case, apostrophes) | Generated code uses `pd.read_sql_query` directly — bypasses `execute_read_only_sql` error hints. LLM hallucinates column case (`SEX` vs `Sex`) or fails to escape quotes (`Women's Soccer`). Cross-validation exec namespace missing `sqlite3`/`gzip` imports. | **Option B — `query_db` injectable helper** with auto-fix for quote escaping + case-insensitive column resolution. Pre-injected into executor + cross-validation `exec_namespace`. See §10. | | 21 | task_194 extra columns regression | Executor includes join-key columns (`atom_id`, `atom_id2`) alongside requested `bond_id`. Post-hoc guards didn't prune because "never removes columns" was a constraint. | **Guard 4 (column pruning)** in `post_hoc_guards.py` — drops extras when ALL expected columns are confirmed present. | | 22 | task_173 empty-answer regression | Strategy 4 + Strategy 8 (safety wrapper) converted crashes to silent `answer=""` single-row results. Critic's `not_empty` check didn't fire because the DataFrame technically had 1 row. | **Guard 5 (effectively-empty detection)** — converts single-row all-blank results to truly empty DataFrame → triggers replan. | | 23 | task_283 percentage calculation (31.6 vs 31.2) | Executor used INNER JOIN for percentage denominator, shrinking the base population. | Added **LEFT JOIN for percentages** rule + **DEDUPLICATION** rule to `EXECUTOR_SYSTEM` prompt. | --- ## 10. Structural Verification Mechanisms (changes E / F / G) After §9's prompt layer saturated, two structural mechanisms were added downstream of `critic_execute_node`. Both are independently reversible via marker blocks and both default to ON; only Change E has an env gate (`ENABLE_NUMERIC_GUARDS`). ### Change E — Mechanism 3: Deterministic Numeric Guards (`numeric_guards.py`) A new module of pure-Python, zero-LLM-cost sanity checks. Each guard is a function of `(question_text, result_df)` with no task-specific knobs. The four guards: | Guard | Trigger | Rationale | |---|---|---| | `nan_or_inf` | Any numeric column contains NaN / ±inf | Unguarded division (divide-by-zero) or unfiltered NULL — never a legitimate final answer | | `percentage_out_of_range` | Question matches `_PERCENT_PAT` AND any numeric column has value outside `[-0.001, 100.001]` | Percentage answer outside `[0, 100]` is impossible. `[0, 1]` fractional convention is allowed (won't trip the upper bound) | | `negative_count` | Question matches `_COUNT_PAT` (and NOT `_RATIO_PAT`) AND any numeric column has a value < 0 | Counts cannot be negative; catches sign-flipped subqueries / bad COALESCEs | | `ratio_zero_int` | Question matches `_RATIO_PAT` AND the lone numeric column is integer-typed AND all values are 0 | Almost always a floor-division bug ("`how many times more`" answered with `5 // 12 = 0`); narrowly scoped to avoid false positives on legitimate `ratio == 1` | **Wire-up** (`nodes.critic_execute_node`, marked `# === CHANGE E (M3) ===`): 1. Run `run_numeric_guards(question, result_df)` after the LLM critic and after the column/row-count post-checks. 2. Flags are always appended to `state.result_critique.issues` and surfaced in the trace under `observation.numeric_guard_flags` (visible to `failure_tagger`). 3. **Only** when the critic had marked the result valid does a flag downgrade `is_valid=False`, cap `confidence ≤ 0.4`, and push a `retry: …` suggestion. The existing executor-feedback loop then fires; `max_total_attempts=4` in `graph.py` naturally caps the extra cost at one retry. 4. Env gate: `ENABLE_NUMERIC_GUARDS` (default `1`). Disabling preserves pre-mechanism behaviour exactly. **Rollback E:** delete `numeric_guards.py`, remove the import and the `# === CHANGE E (M3) ===` block in `nodes.critic_execute_node`, and drop `numeric_guard_flags` from the observation dict. ### Change F — Mechanism 2: Independent Sanity Check (#13 in `CRITIC_EXECUTE_SYSTEM`) A 13th checklist item added to the execute critic, framed strictly as a **second opinion that ignores the code**: > **13. `sanity_independent`** — Pretend you have not seen the Generated Code or Execution Stdout. Read ONLY the Original Question and the Execution Result preview. PASS iff the result, on its face, looks like a plausible answer to that question — the column(s) shown match the *kind* of thing the user asked for, and the value magnitudes are not absurd given the question's framing. FAIL if a naive reader of just (question, result) would say "that doesn't answer the question". The `evidence` field must quote one short phrase from the question and one value/column from the result preview. **Verdict weighting:** failure is **soft** — does NOT flip `is_valid` on its own, but caps `confidence ≤ 0.55` and forces the first suggestion to start with `retry: independent sanity check failed — …`. The existing low-confidence retry branch in `graph.run_agent_graph` (`< 0.7`) then routes the result back to the executor for one more attempt. **User-prompt isolation** (`CRITIC_EXECUTE_USER`, marked `# === CHANGE F (M2) ===`): an instruction block at the end tells the critic to evaluate check #13 *first*, using only the question + result-preview sections, before reading the Generated Code / Execution Stdout sections. This preserves the "unbiased second opinion" property — the other 12 checks are evidence-based and read the code; #13 deliberately doesn't. **Why not a separate node?** Considered and rejected: a standalone `result_sanity` node would have its own LLM call, its own retry routing, and its own context-window cost, while reusing zero infrastructure. Folding it in as check #13 with an "evaluate first, in isolation" instruction achieves the same separation-of-evidence at zero new graph plumbing. If trace inspection later shows #13 is biased by the surrounding code context, promoting it to a standalone node is a small refactor. **Rollback F:** delete the four marked `# === CHANGE F (M2) ===` blocks in `prompts.py` (schema entry in `checks[]`, check #13 definition, verdict rule, user-prompt instruction). ### How E and F interact - E fires first (deterministic, runs every time after the critic). When E flags something, the issue is in `state.result_critique.issues` and the executor sees it in `feedback_section` on retry. - F is consulted inside the critic call itself, *before* E runs. A check #13 failure caps confidence ≤ 0.55, which falls below the 0.7 happy-path threshold and triggers retry independently of E. - Both pathways converge on the same single-retry executor budget, so the worst-case extra cost is one additional `executor_node` call + one additional `critic_execute_node` call per task — bounded by `max_total_attempts=4`. - Neither E nor F encodes any task-specific pattern, idiom catalogue, or failure-derived example. All rules apply uniformly to every question. ### Change G — Cross-Validation via Dual-Track Re-Derivation (`cross_validate.py`) After the executor succeeds, a second LLM call re-derives the answer using the **opposite tool** (SQL↔pandas). The two results are compared using loosened competition scoring normalization. Only **agreement** is surfaced to the critic (strong positive signal); divergence silently caps confidence. **Scope:** ALL easy and medium tasks on execution attempt 1. No restriction on result shape (not limited to 1×1 or aggregation-only). **Flow:** 1. `should_cross_validate(state)` gates on: `ENABLE_CROSS_VALIDATE=1`, difficulty ∈ {easy, medium}, `execution_attempts == 1`, non-empty result. 2. `_detect_primary_tool(code)` heuristic scores SQL vs pandas indicators. 3. `CROSS_VAL_SYSTEM/USER` prompts ask the LLM to rewrite the computation using the alt tool → produces alt Python code. 4. Alt code is executed in a sandboxed `exec()` (same as executor). 5. `_dataframes_agree(primary_df, alt_df)` — loosened comparison (see below). 6. `format_cross_val_annotation(cv_result)`: - **Agreed** → Positive annotation to critic: "High confidence the answer is correct." - **Alt failed / Diverged** → Empty string (nothing shown to critic). Divergence only caps confidence post-critic. **Asymmetric surfacing rationale:** Agreement is a strong positive signal (both independent tracks converged). Divergence is ambiguous — the alt-tool code is often subtly wrong (different LIMIT, column ordering, type coercion), so surfacing it to the critic causes it to second-guess correct results (observed: task_11 regressed 1.0→0 when divergence was shown). Suppressing divergence from the critic while still using it for a soft confidence cap avoids this failure mode. **Post-critic confidence cap** (`nodes.critic_execute_node`, marked `# === CHANGE G post-critic ===`): - If CV ran, alt succeeded, but DIVERGED, and critic accepted the result: cap `confidence ≤ 0.75`. - This is just above the 0.7 happy-path threshold, so it only triggers a retry when the critic was already borderline. It does NOT flip `is_valid`. **Comparison tolerance (loosened):** `_dataframes_agree` applies multiple relaxations to reduce false-divergence: - Ignores column count differences (compares `min(cols)` by position) - Ignores row order (sorts both normalized matrices) - Ignores extra rows (compares `min(rows)` — alt code may not apply same LIMIT) - Numeric tolerance: ±0.01 beyond the 2dp rounding (catches float-vs-Decimal drift) - Partial agreement: ≤2 mismatches across >80% matching cells → still "agreed" **Failure handling:** - If alt-tool execution fails (LLM error, runtime exception, no `result_df` produced): nothing is shown to critic; result stored in trace only. - If both tracks somehow fail: nothing shown to critic. **Critic decides next steps:** The critic (not deterministic code) decides whether to accept, retry, or replan. CV only provides a positive boost (agreement annotation) or a soft confidence cap (divergence). This avoids hard-wiring acceptance/rejection logic that could overfit. **Cost:** One additional LLM call (alt code generation) + one `exec()` call per easy/medium task on attempt 1. No extra LLM call for the comparison itself (pure Python). **Wire-up** (`nodes.critic_execute_node`, marked `# === CHANGE G ===`): - Runs *before* the critic LLM call, after building the user message but before sending it. - `state.cross_validation` stores the `CrossValidationResult` for trace visibility. - Agreement annotation (if any) is appended to the critic's user message. - After critic returns: divergence confidence cap applied. **State** (`state.py`): `CrossValidationResult` dataclass with fields: `primary_tool`, `alt_tool`, `alt_succeeded`, `alt_error`, `agreed`, `divergence_summary`, `alt_result_preview`. Added as `AgentState.cross_validation: CrossValidationResult | None`. **Env gate:** `ENABLE_CROSS_VALIDATE` (default `"1"`). Set to `"0"` to disable entirely. **Rollback G:** delete `cross_validate.py`, remove the import and the `# === CHANGE G ===` / `# === CHANGE G post-critic ===` blocks in `nodes.critic_execute_node`, remove `CrossValidationResult` from `state.py` and the `cross_validation` field from `AgentState`. --- ## 8. Evaluation & Diagnostics CLI Two evaluator-side commands live in `src/data_agent_baseline/cli.py`: ### `eval-lang` Per-task language-recall evaluation. Output includes: - **Per-task table** with `task_id`, `difficulty`, `recall`, `precision`, **`Time (s)`** (from trace). - **Summary**: tasks with `recall > 0` vs `recall = 0` at each λ. - **Breakdown by Difficulty** table (easy → extreme + overall row) showing mean recall / precision / count. - **Zero-recall task list** for quick triage. Powered by `evaluator.py` helpers: `_read_elapsed_seconds`, `_read_difficulty`, `_autodetect_task_root`, and the new `difficulty` / `elapsed_seconds` columns + `difficulty_breakdown` in the summary dict. ### `tag-failures` Failure classification + phase-timing aggregator. Reads a run dir and emits four tables plus `failure_tags.csv`: 1. **Per-task table** — `task_id`, `bucket`, primary `error`, `recall`. 2. **Bucket summary** — counts per bucket, sorted by severity. 3. **Per-phase timing** — total seconds + call count per phase (`explore`, `plan`, `execute`, `critic`, …), derived from trace `action` events via `_ACTION_TO_PHASE`. 4. **Per-bucket task lists** — quick lookup of which task ids fell into each bucket. Buckets (defined in `failure_tagger._classify_bucket`): `perfect`, `near_miss`, `low_recall`, `value_mismatch`, `wrong_column_count`, `wrong_row_count`, `empty_prediction`, `no_prediction`, `timeout`, `api_error`, `crash`, `no_gold`, `other`. ### `eval-comprehensive` Comprehensive evaluation harness for ablation studies. Captures per-task metrics + run-level summary statistics. Output: `comprehensive_evaluation.csv` with per-task rows and an in-memory summary dict. **Per-task metrics** (columns): - **Identification**: `task_id`, `difficulty`, `trace_id` (full path) - **Execution**: `execution_success` (0/1), `execution_time` (s), `execution_error`, `expected_output_column_count`, `expected_row_count` - **Scoring**: `final_score`, `recall`, `extra_columns`, `matched_columns`, `gold_columns`, `pred_columns` - **Trajectory**: `trajectory_length`, `tool_calls`, `tool_failures`, `llm_calls`, `recovery_attempts`, `clarification_requests`, `plan_attempts`, `execution_attempts` - **Quality**: `confidence_score` (numeric 0-1), `confidence_label` (High/Good/Moderate/Low), `ground_truth_available` (Y/N), `bucket` (failure mode) - **Resources**: `total_tokens`, `failure_reason` - **Per-stage demarcation**: for `explore`, `planner`, `critic_plan`, `execute`, `critic_execute` the CSV includes time, tool calls, tool failures, tokens, and **llm_calls**. Trace alignment: - `trace.json` now stores `stage_metrics` and `comprehensive_metrics` blocks. - `comprehensive_evaluator.py` primarily reads these blocks and only performs basic calculations that require gold labels (score/recall/extra/matched and bucket logic). **Bucket classification**: Uses same logic as `tag-failures` for consistency. **Run-level summary** includes: - **Aggregate metrics**: mean/median/std/min/max for score, recall, execution time - **LLM usage**: `total_llm_calls`, `mean_llm_calls`, per-phase LLM call means - **Distribution stats**: `execution_success_rate`, `tasks_with_score_gt_0`, `tasks_with_perfect_score`, `tool_failure_rate` - **Breakdowns**: `difficulty_breakdown` (per-difficulty stats), `bucket_distribution` (count per failure mode), `bucket_details` (per-bucket mean score/time) - **Phase timing table**: includes `total_llm_calls` and `mean_llm_calls` per phase **Use cases**: - Compare baseline vs improved configurations (e.g., `with_critic` vs `without_critic`) - Track performance across task difficulties - Identify dominant failure modes - Measure resource efficiency (time, tokens) - Support longitudinal ablation studies across project phases Powered by `comprehensive_evaluator.py` (`evaluate_run_comprehensive`, `evaluate_task_comprehensive`, `classify_task_outcome`). ### `eval-v2` (Integrated Hardening Suite) **KDD Creative Track comprehensive evaluation harness** with automatic validation and diagnostics. Produces three normalized CSV files plus hardening artifacts. **Evaluation Output:** - `task_metrics.csv` — 100+ columns per task: correctness (multi-level F1), autonomy, planning quality, tool efficiency, data understanding, verification, recovery, trajectory, failure taxonomy, confidence calibration, composite scores - `trajectory.csv` — Step-by-step execution trace with phase, agent, tool, success, tokens, timing - `tool_calls.csv` — Per-tool-call analysis with parameters, results, failures - `comprehensive_evaluation.csv` — Backward compatibility (same as task_metrics.csv) **Hardening Artifacts (Auto-Generated):** 1. **Artifact Reconciliation** (`artifact_reconciliation_report.txt`) — Validates: - Tool call counts match across artifacts - Token counts reconcile (trajectory vs metrics) - Verification semantics (`aat_verification_passed ↔ coordinator_final_decision="APPROVE_FINAL"`) - Time accounting (component compute ≤ wall clock, overhead tracked) - Trajectory completeness 2. **Replay Artifacts** (`task_*/task_replay.json`) — Complete debug snapshots: - Question, context, all execution attempts - Agent executions, tool calls, coordinator decisions - Verifier output, final answer, evaluation result - **Structured failure attribution** (category, root cause, stage, suggested fix) 3. **Engineering Health Report** (`engineering_health_report.txt`) — System diagnostics: - **Separated Assessments**: - **Harness Health** (0-10): Infrastructure quality (reconciliation, validators, attribution coverage) - **Run Quality**: Outcome metrics (answer accuracy, execution success rate) - Reconciliation pass/fail status - Invariant violations - Unaccounted Overhead Time (framework overhead, I/O wait, async queuing) - Failure taxonomy distribution - Top recurring root causes **Display Features:** - **Separated Health Assessments**: - *Harness Health* (9.7/10 ✓ HEALTHY): Infrastructure quality - *Run Quality* (⚠ DEGRADED 64% accuracy): Outcome metrics - **Execution vs Answer Quality Distinction**: - *Execution Success*: Code ran without crashes - *Answer Accuracy*: Correct results (final_score ≥ 0.8) - **Difficulty Breakdown** shows both metrics separately - **MAS Failure Analysis**: - *MAS Failure Categories*: Structured categories (REASONING_FAILURE, DATA_UNDERSTANDING_FAILURE, etc.) - *Failure Distribution by Stage*: AAT phases (UNDERSTAND/PLAN/EXECUTE/VERIFY) - *Outcome Error Types*: Evaluation buckets (low_recall, wrong_schema, etc.) - **AAT Architecture Metrics**: Coordinator decisions, specialist activation, verification outcomes - **Phase Timing Reconciliation**: Displayed time vs cumulative compute with overhead interpretation **Validation Improvements (June 2026):** - Fixed `verification_passed` semantics: Critic verification (legacy) vs AAT verification (coordinator approval) are now distinct - Fixed `data_understanding_score` validation: Recognizes weighted formula (0.5·specialist + 0.5·discovery) - Fixed time accounting: Component < wall_clock is expected (overhead); warns only if ratio < 5% **Usage:** ```bash dabench eval-v2 --mode standard # Quick overview dabench eval-v2 --mode verbose # Detailed analysis dabench eval-v2 --mode research # All metrics for papers ``` Powered by `eval_v2.py`, `eval_v2_metrics.py`, `eval_v2_viz.py`, `eval_v2_validator.py` + hardening suite (`eval_artifact_reconciliation.py`, `eval_replay_artifacts.py`, `eval_health_report.py`). --- ## 9. Prompt Architecture (changes A / B / C) All four agent system prompts now share a layered structure defined in `prompts.py`. Each change is marked with `# === CHANGE X — START/END ===` block comments so it can be rolled back independently. ### Change A — `SHARED_HEADER` prepended to every SYSTEM prompt A single constant carrying the four highest-leverage rules sits at the top of `prompts.py` and is concatenated onto `PLANNER_SYSTEM`, `CRITIC_PLAN_SYSTEM`, `EXECUTOR_SYSTEM`, and `CRITIC_EXECUTE_SYSTEM`: 1. **Scoring rule** — binary column-matching; "when in doubt, INCLUDE the column". 2. **Ground-truth hierarchy** — Data Exploration wins on column existence; knowledge.md wins on value encodings. 3. **Answer-shape vocabulary** — `cardinality` / `value_type` / `unit` triple keyed off linguistic cues (`"how many"` → scalar_int / count, `"how many times more"` → ratio, etc.). 4. **Naming discipline** — never return ID when name is asked; never concatenate `first_name + last_name`. **Rollback A:** delete `SHARED_HEADER` and change `SHARED_HEADER + """\` back to `"""\` on the four SYSTEM prompts. ### Change B — `CRITIC_EXECUTE_SYSTEM` rewritten as a forced 12-item checklist The execute critic must now emit one `{id, passed, evidence}` entry per check rather than free-form prose, then apply deterministic verdict rules. The 12 checks: | # | Check ID | Effect when failed | |---|---|---| | 1 | `required_columns_present` | `is_valid=false`, conf ≤ 0.2 | | 2 | `count_vs_list` | `is_valid=false`, conf ≤ 0.3 | | 3 | `name_not_id` | `is_valid=false`, conf ≤ 0.4 | | 4 | `name_columns_separate` | `is_valid=false`, conf ≤ 0.4 | | 5 | `no_null_zero_filter` | warning | | 6 | `no_head_or_limit` | warning | | 7 | `ratio_not_count` | `is_valid=false`, conf ≤ 0.4 | | 8 | `monthly_vs_annual` | warning | | 9 | `value_encoding_correct` | `is_valid=false`, conf ≤ 0.4 | | 10 | `ordering_explicit` | warning only | | 11 | `magnitude_reasonable` | warning only | | 12 | `not_empty` | `is_valid=false`, conf ≤ 0.1, suggestions must start with `replan:` | | 13 | `sanity_independent` (Change F) | soft — `is_valid` unchanged, conf ≤ 0.55, first suggestion must start with `retry:` | Suggestions are formatted as `fix: …` / `replan: …` / `retry: …` prefixes for easier downstream routing. **Rollback B:** rename active `CRITIC_EXECUTE_SYSTEM` to something else and rename `CRITIC_EXECUTE_SYSTEM_LEGACY` back to `CRITIC_EXECUTE_SYSTEM`. ### Change C — `CRITIC_EXECUTE_USER` checklist nudge The user prompt now labels `Expected Output Columns` as "REQUIRED columns for scoring — check #1" and ends with `Walk the 12-item checklist in order. … Output JSON only.` Original preserved as `CRITIC_EXECUTE_USER_LEGACY`. **Rollback C:** rename `CRITIC_EXECUTE_USER_LEGACY` back to `CRITIC_EXECUTE_USER`. ### Change H — Doc Extraction Strategies 1/2/3 (hard-task coverage) Three generic improvements to doc extraction in `semantic.py`: **Strategy 1 — Question-aware column subsetting:** When `column_hints` exist and the probe schema has >5 columns, the extraction prompt receives only merge_key + question-relevant columns. This reduces tokens per record, allowing more records per batch. Full schema is preserved in the table definition; un-extracted columns become NULL. Subsetting only activates if ≥30% of columns can be dropped. **Strategy 2 — Adaptive batch budget:** In the flat extraction path, if `record_count_estimate` is high, `max_batches` is automatically increased (up to 2×) based on the heuristic of ~30 records per batch. This prevents over-compression of large docs (e.g., 750-record superhero.md with only 12 batches → now up to 25). Wall-clock cap still honoured via `deadline_monotonic`. **Strategy 3 — Threshold/range registry:** `_extract_threshold_rules` now parses three additional patterns beyond the original "values above X considered Y": `normal range: X-Y`, `normal: < X`, `abnormal if > X`. These populate `EntityField.normal_range` and add `scope="threshold"` constraints. `critical_rules_for_question` surfaces `normal_range` for question-relevant fields. **Rollback H:** Revert the three blocks in `extract_doc_records_chunked` (Strategy 1 column subset, Strategy 2 adaptive budget) and restore `_extract_threshold_rules` to single-pattern form. ### Downstream compatibility `critic_execute_node` in `nodes.py` reads only `is_valid` / `confidence` / `issues` / `suggestions` from the critic JSON — the new `checks` array is purely additive and requires no parser changes. The existing `col_count_info` / `row_count_info` text appended to the user message is now partially redundant with the structured `required_columns_present` / `count_vs_list` checks but is left in place for safety. ### Change I — Multi-candidate plan voting + diagnostic probes (anti-non-determinism) **Files:** `src/data_agent_baseline/langgraph_agent/vote_plan.py` (new), integrated in `graph.py`. **Mechanism — Plan Voting:** 1. On the *first* planning attempt only, 2 candidate plans are generated in parallel. 2. Compare `output_columns`: - **Agree** → use candidate 1, proceed normally (90% of tasks). - **Disagree** → run ONE deterministic SQL probe against `_auto.db` to check which columns actually exist in the database schema. - Probe scores each candidate by how many of its output_columns are real DB columns. - Higher score wins. If tied, generate a **3rd informed plan** with the probe evidence injected as feedback. - Final fallback: pick the plan with fewer columns (conservative). **Mechanism — Zero-row Diagnostic Probe:** When the executor produces a result with 0 rows and the critic rejects it, `run_zero_row_diagnostic` is called before retry. It queries `_auto.db` to verify: - Tables mentioned in the plan exist and have rows - Column names actually present - Sample values from filter-like columns (status, type, category, etc.) This evidence is appended to the critic's suggestions so the next executor attempt (or replan) has concrete data about what's actually in the DB. **Cost:** +1 LLM planner call on first attempt (parallel, same wall-clock). Diagnostic probes are pure SQL (0 LLM cost). 3rd informed plan only triggers on disagreement (~10% of tasks). **Kill-switch:** Set `VOTE_ENABLED = False` in `vote_plan.py`. **Rollback I:** In `graph.py`: (1) replace `voted_planner_node` import/call with `planner_node`, (2) remove the `run_zero_row_diagnostic` block in the critic-rejected path, (3) delete `vote_plan.py`. ### Fix J — Runner prediction.csv guardrails Two fixes to `runner.py` that caused regressions in run `20260521T053658Z`: 1. **Always write `prediction.csv`** — Previously, if `execution_result` was None or empty, no file was written, causing "prediction.csv missing" (0 score). Now a fallback `pd.DataFrame({"answer": [""]})` is always emitted. 2. **Removed "drop all-NaN columns" logic** — The runner was dropping columns where all values are NaN/empty. This actively harms scoring because the evaluator checks column *presence*. Example: task_257 had `DisplayName=None` (executor couldn't find the editor) → runner dropped the column → lost 0.5 score. Columns with null values must still be emitted. **Rollback J:** Re-add the `non_empty_mask` filter block in `_save_result`, change the `else` fallback to `prediction_csv_path = None`. ### Fix K — Local import cleanup All local/conditional `from ... import` and `import` statements inside functions were moved to top-level across `graph.py`, `nodes.py`, `semantic.py`, and `model.py`. This fixed `UnboundLocalError: cannot access local variable 'CriticFeedback'` (task_196 crash) caused by Python's scoping rules treating local imports as local variable declarations for the entire function scope. **Files touched:** `graph.py` (removed 2× `from ... import CriticFeedback`, 2× `import sqlite3`, 2× `import pandas as pd`), `nodes.py` (removed `import os as _os`, `import time as _time`, `import hashlib as _hashlib`, `import sqlite3 as _sqlite3`, `import io`, `import sys`), `semantic.py` (removed 3× `from concurrent.futures import ...`, 3× `import time as _time`, `import copy`, `import sqlite3`, `import pandas as pd`, `import traceback`, `import re`), `model.py` (moved `from ... import get_cached, set_cached` to top-level). **Rollback K:** N/A — pure cleanup, no behavioral change beyond fixing the CriticFeedback crash. ### Strategy 1+3 — Deterministic seed injection (anti-non-determinism) **Files:** `src/data_agent_baseline/langgraph_agent/deterministic_seed.py` (new config), `src/data_agent_baseline/agents/model.py` (integration). **Problem:** Executor-level LLM non-determinism causes regressions between runs even with `temperature=0`. Tasks like task_355, task_173, task_249 produce different code each run — sometimes raising exceptions on 0 rows, sometimes using wrong filter logic or column naming. **Mechanism:** - Injects `seed=42` into every `client.chat.completions.create()` call (both OpenAI and Azure adapters). - The OpenAI/Azure API `seed` parameter makes the model use deterministic sampling ("best effort") — not guaranteed identical but significantly reduces variance between identical prompts. - Temperature was already 0.0 via config; this change adds the complementary seed parameter. **Configuration:** - `LLM_SEED=42` (env var override for A/B testing different seeds) - `LLM_SEED_ENABLED=0` (env var kill-switch to disable seed injection) **Cost:** Zero — purely a parameter addition to existing API calls. **Rollback Strategy 1+3:** In `model.py`: (1) remove `from data_agent_baseline.langgraph_agent.deterministic_seed import DETERMINISTIC_SEED, SEED_ENABLED`, (2) remove the `if SEED_ENABLED: _create_kwargs["seed"] = DETERMINISTIC_SEED` blocks in both adapters, (3) restore direct `client.chat.completions.create(model=..., messages=..., temperature=...)` calls, (4) delete `deterministic_seed.py`. ### Strategy 5 — Post-hoc deterministic output guards **Files:** `src/data_agent_baseline/langgraph_agent/post_hoc_guards.py` (new), integrated in `graph.py`. **Problem:** The executor LLM generates code with non-deterministic column naming (e.g., `"display_name"` vs `"DisplayName"`) and column ordering. These differences cause lambda-score regressions even when the underlying data is correct. **Mechanism — applied after `executor_node`, before `critic_execute_node`:** 1. **Column name normalization:** If result columns differ from `plan.output_columns` only by case/spacing/underscores, auto-rename to match the plan. Only applied when column count matches exactly (no ambiguity). 2. **Column reordering:** If result has the same column set as expected but in different order, reorder to match plan. 3. **Empty-result column presence:** If result is 0 rows, ensure all expected columns exist (add missing ones as empty) so the output CSV schema is correct. 4. **Extra column pruning:** If result has MORE columns than expected and ALL expected columns are present (exact or normalized match), drop the extras. Extra columns only incur λ-penalty — pruning when the expected set is fully covered is always safe. Fixes cases like task_194 where `atom_id`/`atom_id2` were included alongside the requested `bond_id`. 5. **Effectively-empty detection:** A single-row result where ALL values are empty/blank/NaN (e.g., `answer=""`) is converted to a truly empty DataFrame. This triggers the `not_empty` critic check (#12) → replan, rather than silently accepting "I couldn't find the answer" as a valid result. Fixes task_173-type regressions where Strategy 4/8 turned crashes into silent empty answers. **Key design constraints:** - Never re-invokes the LLM — purely deterministic DataFrame operations. - Only fires when the fix is unambiguous (e.g., same number of columns, case-insensitive match). - Guard 4 only removes columns when ALL expected columns are confirmed present. - Guard 5 only converts to empty when literally every cell is blank/NaN (very conservative). - Cannot make results worse: renaming `display_name` to `DisplayName` when the plan says `DisplayName` can only improve the lambda-score. **Configuration:** - `POST_HOC_GUARDS_ENABLED=0` (env var kill-switch) **Cost:** ~0ms per task (DataFrame column operations on already-computed result). **Rollback Strategy 5:** In `graph.py`: (1) remove `from ... import apply_post_hoc_guards`, (2) remove `state = apply_post_hoc_guards(state)` after `executor_node`, (3) delete `post_hoc_guards.py`. --- ### Guard 6 — Magnitude Probing **File:** `src/data_agent_baseline/langgraph_agent/post_hoc_guards.py` (`_magnitude_probe` function). **Problem:** Single-value numeric results can be wildly implausible (e.g., result=31600 when all numeric columns in the DB max at 100) due to missed division, wrong aggregation, or incorrect JOIN multiplying rows. **Mechanism:** For single-row, single-column numeric results: 1. Queries `_auto.db` to find global MIN/MAX across all numeric columns in all tables. 2. If `|result| > 10000 × max(|global_min|, |global_max|)`, flags result as implausible. 3. Converts to empty DataFrame → triggers retry with feedback. **Constraints:** - Only activates for 1×1 numeric results (avoids false positives on multi-row/multi-col). - Very conservative threshold (10000×) — only catches truly absurd magnitudes. - Skips zero/NaN values. **Cost:** ~1ms (a few PRAGMA + MIN/MAX queries on _auto.db). **Rollback Guard 6:** Delete `_magnitude_probe` function and the Guard 6 block in `apply_post_hoc_guards`. --- ### Bidirectional Table Detection **File:** `src/data_agent_baseline/langgraph_agent/nodes.py` (`_detect_bidirectional_tables` function). **Problem:** Tables like `connected` store symmetric relationships (for every row (A,B) there exists (B,A)). Naive COUNT/JOIN on such tables double-counts unless the query adds `WHERE col1 < col2` or uses DISTINCT unordered pairs. This caused task_196 to return 2× the correct answer. **Mechanism:** During `explore_data_node`, after building `_auto.db`: 1. For each table with ≤6 columns, checks all column pairs (i, j). 2. For each pair, verifies that swapping the two columns produces identical rows (every row has its mirror). 3. If detected, appends an annotation to `state.data_exploration` warning the planner/executor to use `WHERE col1 < col2`. **Constraints:** - Only checks tables with 2–6 columns (edge-list heuristic). - Skips tables with >6 columns or 0 rows. - Purely informational: adds a warning to `data_exploration` text, doesn't modify queries. **Cost:** A few SQL queries per small table during exploration (~5ms total). **Rollback Bidirectional Detection:** Delete `_detect_bidirectional_tables` function and the "CHANGE: Bidirectional table detection" block in `explore_data_node`. --- ### Increased Retry Budget (max_total_attempts 3→5) **File:** `src/data_agent_baseline/langgraph_agent/graph.py` (line ~245). **Problem:** With only 3 attempts, tasks that need 1 failed attempt to learn the schema + 1 failed attempt to fix logic errors exhaust the budget before producing a correct answer. **Fix:** Changed `max_total_attempts = 3` → `max_total_attempts = 5`. **Cost:** Up to 2 additional LLM calls per task (only when earlier attempts fail). Worst-case ~4 extra calls per failed task. **Rollback:** Change `max_total_attempts = 5` back to `max_total_attempts = 3` in `graph.py`. ### Strategy 4 — "Never raise on empty" executor prompt constraint **File:** `src/data_agent_baseline/langgraph_agent/prompts.py` (added rules to `EXECUTOR_SYSTEM`). **Problem:** When a query/filter returns 0 rows, the LLM sometimes generates `raise ValueError("No X found")` or `raise Exception(...)`. This causes a hard crash — the executor never produces `result_df`, so the task gets 0 score. Examples: task_257 (`ValueError: No post found`), task_355 (SyntaxError from over-complex fallback logic). **Fix:** Added explicit rules to `EXECUTOR_SYSTEM`: - "NEVER RAISE EXCEPTIONS ON EMPTY RESULTS" — always create `result_df = pd.DataFrame(columns=[...])` instead - "NEVER use sys.exit(), exit(), quit(), or return at module level" **Cost:** Zero (prompt text only). **Rollback Strategy 4:** Remove the two bullet points starting with "NEVER RAISE EXCEPTIONS" and "NEVER use sys.exit()" from `EXECUTOR_SYSTEM` in `prompts.py`. ### Strategy 8 — Executor safety wrapper (template wrapping) **File:** `src/data_agent_baseline/langgraph_agent/nodes.py` (`_build_safety_wrapper` function). **Problem:** Despite prompt instructions (Strategy 4), the LLM may still generate code that raises exceptions or has syntax errors. A crashed executor = 0 score. We should NEVER crash. **Mechanism:** Before `exec()`, the generated code is wrapped in a safety template that compiles and executes inside a try/except. If ANY exception occurs (ValueError, SyntaxError, KeyError, etc.), the wrapper catches it and sets `result_df = pd.DataFrame(columns=expected_output_columns)`. This guarantees that `result_df` always exists after execution — even if the LLM code is syntactically invalid or raises. The empty DataFrame with expected columns then flows through to post-hoc guards (Strategy 5) and the critic, which can request a retry with feedback. **Key property:** The wrapper captures exceptions into stdout so the critic/retry feedback loop can see what went wrong, while still producing a valid (empty) result. **Kill-switch:** `EXECUTOR_SAFETY_WRAP=0` env var. **Cost:** ~0ms overhead (compile + exec overhead negligible). **Rollback Strategy 8:** In `nodes.py`: (1) remove `_SAFETY_WRAP_ENABLED` and `_build_safety_wrapper`, (2) restore direct `exec(code, exec_namespace)` in `executor_node`, (3) remove the `_safety_wrapper` variable and `expected_output_columns` lines. ### Option B — `query_db` injectable helper (SQL error auto-recovery) **Files:** `src/data_agent_baseline/tools/sqlite.py` (new `query_db` function + helpers), `src/data_agent_baseline/langgraph_agent/nodes.py` (injection into executor `exec_namespace`), `src/data_agent_baseline/langgraph_agent/cross_validate.py` (injection into cross-validation `exec_namespace`). **Problem:** The executor generates Python code that calls `pd.read_sql_query()` / `sqlite3` directly. When the LLM hallucinates column names or fails to escape apostrophes, the error surfaces as a crash inside the safety wrapper — producing an empty DataFrame. The improved `execute_read_only_sql` error hints are never seen because the executor bypasses that function entirely. **Mechanism:** A new `query_db(db_path, sql, *, limit=200) → pd.DataFrame` function in `sqlite.py` that: 1. **Executes SQL** via `pd.read_sql_query` on a read-only connection. 2. **On syntax error** (e.g., `near "s": syntax error` from `Women's Soccer`): auto-fixes unescaped apostrophes using `_fix_unescaped_quotes()` (regex: `(\w)'(\w)` → `\1''\2`) and retries. 3. **On "no such column"** (e.g., `SEX` when actual column is `Sex`): auto-fixes via `_fix_column_case()` which builds a `lowercase → actual_name` map from `PRAGMA table_info` across all tables, then applies case-insensitive regex replacement. Retries with the fixed SQL. 4. **On unrecoverable failure**: raises with a full schema hint (all tables + their columns) so the safety wrapper's captured traceback gives the executor retry loop actionable feedback. **Injection:** In `executor_node`, a pre-bound closure `_query_db_bound(sql)` (with `db_path=state.auto_db_path` already captured) is injected into `exec_namespace` as `query_db`. The executor's generated code can call `query_db(sql)` instead of raw `sqlite3.connect()` + `pd.read_sql_query()`. **Cross-validation:** Same `query_db` is injected into the cross-validation `exec_namespace`, fixing the `name 'sqlite3' is not defined` error (the exec namespace now also pre-injects `sqlite3`, `pd`, `gzip`, `json`). **Which errors this fixes:** | Error | Root cause | How `query_db` fixes it | |-------|-----------|------------------------| | `near "s": syntax error` | Unescaped `'` in `Women's Soccer` | `_fix_unescaped_quotes` → `Women''s Soccer` | | `no such column: SEX` | Case mismatch (`Sex` vs `SEX`) | `_fix_column_case` resolves via PRAGMA | | `name 'sqlite3' is not defined` (cross-val) | Empty exec namespace | Pre-injected modules | | `'utf-8' codec can't decode byte 0x8b` (cross-val) | gzip file read as text | `gzip` module pre-injected + prompt instruction | | `'event_id'` / `'member_id'` KeyError (cross-val) | LLM guesses column names | Schema now included in cross-val prompt | **Prompt update:** `EXECUTOR_SYSTEM` instructs: "Use `query_db(sql)` for ALL SQL queries against _auto.db. Do NOT use `sqlite3.connect()` or `pd.read_sql_query()` directly." (Note: the LLM may still bypass this instruction — `query_db` is a best-effort improvement, not a hard enforcement. The safety wrapper remains the last line of defense.) **Cost:** Zero additional LLM calls. Runtime overhead: one regex + one PRAGMA scan per failed query (only on error path). **Rollback Option B:** (1) In `sqlite.py`: remove `query_db`, `_fix_unescaped_quotes`, `_fix_column_case`, `_get_schema_hint`, and the `import re` / `import pandas as pd` additions. (2) In `nodes.py`: remove the `_query_db_bound` closure and `"query_db"` from `exec_namespace`, revert to `exec_namespace: dict[str, Any] = {}`. (3) In `cross_validate.py`: remove the `query_db` import and entry from `exec_namespace`. (4) In `prompts.py`: remove the `query_db` instruction from `EXECUTOR_SYSTEM`. --- ### Execution Memory (Intra-Run Attempt History) **Files:** `state.py` (new `execution_memory` field), `graph.py` (accumulation), `nodes.py` (injection into executor prompt only). **Problem:** On retries, the agent only sees the most recent critic feedback. It has no memory of all prior attempts — what code was tried, what results were produced, why they were rejected. This leads to repeating the same mistakes. **Mechanism:** 1. **Accumulation** (in `graph.py`): After each plan-critic and exec-critic cycle, a condensed entry is appended to `state.execution_memory`: - Plan phase: steps, output columns, reasoning, critic verdict + issues - Execute phase: code snippet (last 600 chars), result shape + preview (3 rows), critic valid/confidence/issues/reasoning, stdout tail 2. **Injection into executor only**: Full history (plan + execute phases) with code, results, rejection reasons. Labeled "do NOT repeat failed approaches". 3. **Planner injection DISABLED** (run `20260522T080948Z` showed it causes plan drift — the planner over-corrects when shown execution-level detail, leading to wrong reinterpretations on task_200 and task_283). **Key properties:** - No persistence — in-process Python list only, GC'd when task finishes. - Condensed — each entry ~500-800 chars; 4 attempts = ~3KB total injection. - Plan entries show strategy-level info; executor entries show code-level detail. **Cost:** Zero LLM calls. ~0ms (string formatting). **Rollback:** (1) Remove `execution_memory` from `state.py`. (2) Remove accumulation blocks in `graph.py` (marked `=== Memory:`). (3) Remove injection block in `executor_node` (marked `=== Execution memory`). --- ### Increased Retry Budget (max_total_attempts 3 → 5) **File:** `graph.py` line ~248. **Problem:** With only 3 attempts, tasks needing 1 failed attempt to learn the schema + 1 to fix logic exhaust the budget before producing a correct answer. **Fix:** `max_total_attempts = 5`. A/B testing showed that tasks like task_355 and task_86 require 5 attempts to converge on the correct answer. Reducing to 4 causes regressions on those tasks. **Cost:** Up to 2 extra LLM calls per task (only on failure path). **Rollback:** Change `max_total_attempts = 5` back to `3` in `graph.py`. --- ### Case-Insensitive String Matching Prompt Rule — REMOVED **Status:** ROLLED BACK after A/B testing (run `20260522T105623Z`). **Reason:** The rule was neutral-to-harmful. It didn't fix the target task (task_257) and added prompt noise that could cause over-defensive matching on tasks with exact-match data. Removed from `EXECUTOR_SYSTEM` in `prompts.py`. --- ## A/B Testing Results (2026-05-22) ### Evaluator Discovery The competition evaluator uses **value-based column signatures** (sorted normalized cell values), NOT column names. Column naming is irrelevant to scoring. Earlier analysis using column-name recall (0.55) was misleading — actual score is **0.6966** (λ=0.1). ### Run Comparison (proper evaluator) | Config | Score (λ=0.1) | Recall>0 | |--------|--------------|----------| | **Baseline** (mem ON, max=5) | **0.6966** | 35/49 | | Two-Phase Doc Extraction | 0.6567 | 34/50 | | no_str_match + mem_ON + max=4 | 0.6697 | 33/49 | | no_str_match + mem_OFF + max=5 | 0.6679 | 34/49 | ### Current Config (matches baseline) - `max_total_attempts = 5` - Execution memory: ON (accumulation + injection in executor) - STRING MATCHING rule: REMOVED - Bidirectional detection: active (≤500 rows) - Magnitude probing: active - Value Discovery (Strategy B): active --- ## Failure Analysis (14 zero-recall tasks in baseline) ### Failure Categories | Category | Tasks | Root Cause | |----------|-------|-----------| | Wrong computation | task_169, task_196 | Division logic errors (÷12 missing, bond counting) | | Wrong filter/grouping | task_163, task_25, task_80, task_86, task_344 | Wrong filter value or event selection | | Doc extraction failure | task_352, task_379, task_396 | Narrative prose not fully parsed | | Extra columns (wrong values) | task_180, task_194 | Extra cols pollute value-signature matching | | Logic error | task_89, task_214 | Wrong approach entirely | ### Doc Extraction Failures (Highest-Impact) 4 of 14 failing tasks depend on doc/*.md with NO markdown tables — all data in prose: | Task | Doc | Size | Issue | |------|-----|------|-------| | task_344 | Patient.md | 56KB | Patient records narrative. Needs WBC/FG thresholds not in knowledge.md | | task_352 | budget.md | 63KB | Budget amounts in prose. Multi-hop: budget→event_id linkage | | task_379 | molecule.md | 36KB | Carcinogenic classification in prose. Must join with atom.csv | | task_396 | superhero.md | 178KB | 500+ records with height/publisher_id in prose. 5 attempts all failed | ### What's NOT working: 1. **Incomplete extraction**: Large docs (178KB, 500+ entities) likely hit token limits 2. **Multi-hop linkage**: task_352 requires budget_record → event_id → event_name across paragraphs 3. **Implicit thresholds**: task_344 needs medical domain knowledge absent from data 4. **Exhaustive parsing**: System extracts keyword-filtered subsets, not ALL records ### Potential Improvements (NOT YET IMPLEMENTED) 1. **Cross-reference resolution**: Trace record ID references across paragraphs (task_352) 2. **Domain threshold injection**: Inject common clinical thresholds when knowledge.md is silent --- ## Change: Stem-Fix (2026-05-22) **Root Cause (task_396):** Column subsetting during doc extraction dropped `publisher_id` because the hint `"publisher_name"` (from the question) didn't fuzzy-match `"publisher_id"` (in the doc table). The substring check `"publisher_name" in "publisher_id"` is False. Result: extraction only asked for `[id, height_cm]`, so the DB had 100 rows with NULL publisher_id. All 5 execution attempts failed with 0 rows after JOIN. **Fix — Stem Matching (semantic.py):** Added stem-based column matching to the column subsetting logic. `publisher_name` → stem `publisher` matches `publisher_id` → stem `publisher`. Now `publisher_id` is included in extraction_columns. **Files changed:** - `semantic.py`: Stem matching in column subsetting (~line 1913) --- ## Change: Two-Phase Doc Extraction (2026-05-22) **Problem:** Large narrative docs (e.g., superhero.md at 178KB) are extracted eagerly before planning. The extraction splits the doc into linear batches, but the doc has separate sections per attribute (Section 3 = height, Section 5 = publisher). Column subsetting means each batch only extracts 2-3 columns, but the LLM sees entities whose data for those columns may not be in that batch's text range. Result: many NULL/placeholder values (e.g., 32/102 entities with valid height vs gold's 62/102). **Fix — Two-Phase Doc Extraction with Planner-Gated Re-extraction:** - **Phase 1 (pre-planner):** For docs > 40KB, run structure probe (1 LLM call) + extract 3-5 sample rows (1 LLM call). Store profile in `state.doc_profiles`. Planner sees the table schema + samples. - **Phase 2 (post-planner):** After plan is produced, identify which columns the plan needs. Chunk the entire doc into entity-level pieces (by `##` headings or paragraph boundaries), pack into 12KB batches, extract ALL entities with all needed columns per batch, merge by entity ID. Replace sample data in _auto.db. **Architecture:** ``` semantic_extraction_node (Phase 1: profile + sample for large docs, eager for small) → planner_node (sees schema + sample rows) → doc_phase2_extraction (Phase 2: whole-doc entity chunking + batch extraction) → executor_node ``` **Key Parameters:** - `_DEFER_DOC_SIZE_THRESHOLD = 40,000` — docs above this use two-phase - `_PHASE2_MAX_BATCHES = 20` — max LLM calls per doc in Phase 2 - `_PHASE2_BATCH_SIZE = 12,000` — chars per batch (smaller = more complete extraction per batch) **Phase 2 Algorithm (revised 2026-05-22):** 1. Parse plan text to identify needed columns (stem-match + exact match) 2. Split entire doc into entity-level chunks via `_get_doc_chunks_prioritized()`: - Primary: split at `##`/`###`/`####` headings (if ≥10 found) - Fallback: split at paragraph boundaries, group small blocks (min 400 chars) 3. Pack chunks into 12KB batches → typically 15 batches for 178KB doc 4. Each batch extracts ALL entities with ALL needed columns (not section-specific) 5. Merge records by entity ID (first-non-null per column across batches) 6. Drop + reload _auto.db table with merged results **Performance (task_396 superhero.md):** - Phase 1: 2 LLM calls (probe + sample) → 5 sample rows in ~8s - Phase 2: 15 LLM calls × ~5s = ~75s → 465 raw records → 100 unique entities - Prediction: 53.13% vs gold 54.84% (ratio 0.969) - Note: 100/750 entities is incomplete but produces near-correct percentages because the sample is representative across the doc's linear order **Evaluation (run `20260522T191106Z`):** - Score: 0.6567 (λ=0.1), 34/50 recall>0 - Baseline: 0.6966 (λ=0.1), 35/49 recall>0 - **Net regression: -0.04** — Phase 2 helps task_396 approach correct answer but doesn't reach exact match (evaluator needs exact values). LLM non-determinism caused regressions on other tasks (task_25, task_200 went 1.0→0). - task_396 still scores 0 because 53.13% ≠ 54.84% (evaluator uses exact match on values) **Known Limitations:** 1. Only extracts ~100/750 entities from superhero.md — LLM truncates output at ~30-50 records per 12KB batch, and many entities appear in multiple sections causing dedup 2. The evaluator uses exact value matching — even 0.97 ratio gives score 0 3. Adds ~75s overhead to large doc tasks (within 510s timeout but reduces retry budget) 4. Non-doc tasks regress due to LLM non-determinism (not caused by Phase 2) **Files changed:** - `doc_phases.py`: New module — `should_defer_doc_extraction()`, `doc_phase1_profile()`, `doc_phase2_extraction()`, `_get_doc_chunks_prioritized()` - `nodes.py`: Added deferred-extraction guard in semantic_extraction_node loop - `graph.py`: Imports `doc_phase2_extraction`, calls after planning - `state.py`: Added `doc_profiles: dict`, `doc_texts: dict` fields **Rollback:** 1. In `nodes.py`: Remove the `if should_defer_doc_extraction(...)` block (~20 lines) 2. In `graph.py`: Remove `doc_phase2_extraction` import and call 3. In `state.py`: Remove `doc_profiles` and `doc_texts` fields 4. Delete `doc_phases.py` **Decision: ROLLBACK RECOMMENDED** — The -0.04 regression outweighs the partial improvement on task_396 (which still scores 0). The fundamental issue is that the evaluator needs exact values, and extracting 100/750 entities can't produce exact percentages. The eager extraction path (with stem-fix) already gets comparable coverage without the overhead. --- ## 11. Evaluation Harness V2 **Evaluation V2** (`src/data_agent_baseline/langgraph_agent/eval_v2*.py`) provides comprehensive KDD Creative Track evaluation with 50+ metrics organized into 10 categories: correctness (answer/column/row F1), autonomy (first_try_success, replan_count), planning (revisions, dead_ends, alignment), tools (efficiency, diversity, selection_accuracy), data_understanding (tables/columns discovered), verification (score, failures_detected), recovery (success_rate, depth), trajectory (efficiency, branching_factor, critic_loops), failure_taxonomy (category, root_cause, severity), confidence (error, calibration), and composite (analyst_score). **Storage:** 3 normalized CSVs — `task_metrics.csv` (1 row per task, 100+ columns), `trajectory.csv` (1 row per step for process mining), `tool_calls.csv` (1 row per tool call). **Visualization:** Rich terminal output with 3 modes — standard (core metrics + summary), verbose (+ agent behavior analysis: analyst quality, tool usage, recovery, verification, failure analysis, trajectory patterns), research (+ all benchmark metrics, aggregated statistics for papers). **Enhanced Agent Recording:** Nodes now track phase, duration, token_usage, and stage_metrics in trace.json for granular analysis. **CLI:** `dabench eval-v2 [--mode standard|verbose|research]` generates all metrics; `dabench view-task-v2 ` shows detailed task breakdown. Backward compatible with V1 via `comprehensive_evaluation.csv`. See `EVALUATION_V2_QUICKSTART.md` for usage and `EVALUATION_V2_ARCHITECTURE.md` for metric definitions. ### Evaluation V2 Semantics & Metric Families (KDD Publication Standard) **Phase Timing in Primary Report:** The "AAT Execution Phases (Primary)" table displays only **coordinator-aggregated AAT phase timings** — these represent the synchronous wall-clock time spent in each phase, accounting for task parallelism across specialist agents. This differs from legacy trace accounting, which summed individual agent runtimes (and thus could appear to "double count" when specialists ran in parallel). The primary table does **not** include legacy values; the compatibility view explicitly shows legacy timing for comparison. **Phase Time Reconciliation Table:** "Phase Time Reconciliation" displays cumulative compute time (sum of all contributing components) vs. the displayed phase time. When cumulative compute exceeds displayed time, it indicates overlapping work (e.g., coordinator, schema agent, and domain agent running in parallel during UNDERSTAND). This interpretation is explicit in the table to avoid reader confusion about "missing time" or "double counting." **Verification Metric Families:** Evaluation separates verification into two semantic families: - **Critic Family** (process metrics): `critic_verification_steps`, `critic_verification_passed`, `critic_failures_detected`, `critic_verification_score` — capture LLM-internal validation logic (planner critic, execution critic). Backward-compatible aliases (`verification_*`) map critic fields for legacy code. - **AAT Family** (outcome metrics): `aat_verification_triggered`, `aat_verification_passed`, `aat_verification_score` — capture final team verification outcome from independent verifier + coordinator approval decision. Presence indicates AAT execution; absence means critic-only fallback. **Outcome Error Taxonomy:** Formerly "Failure Analysis," this table breaks down task results by failure category (e.g., `value_mismatch`, `wrong_schema`, `perfect`) and root cause (e.g., `schema_misunderstanding`, `none`). This is an outcome classification, not a process metric, and is orthogonal to verification or recovery attempts. **Data-Understanding Score Gating:** The `data_understanding_score` is gated by `documents_required_for_task`: if documents are not required per semantic extraction, the score is computed from schema + domain agents only; if documents are required, the document agent activation contributes. This prevents artificial inflation from document-agent availability when documents are irrelevant to the task. ### Additional Evaluation Consistency Fixes **Recovery Metrics Aggregation:** The "Recovery Success Rate" is computed as **aggregate (total_successes / total_attempts)**, not as the mean of per-task rates. This prevents tasks with zero recovery attempts (i.e., first-try successes) from diluting the rate. Example: given [1/4, 0/0, 0/0] task rates, the aggregate is (1)/(4) = 0.25, not mean(0.25, 0, 0) = 0.083. Task-level rates remain in the CSV for detailed analysis. **Data Understanding Metric Refinement:** To avoid variance collapse when all tasks use identical specialist activation patterns, `data_understanding_score` now blends two components: - **Specialist score** (0 to 1): Proportion of engaged specialists relative to gated requirements (e.g., if schema + domain agents engaged and docs not required, score = 2/2 = 1.0). - **Discovery score** (0 to 1): Proportion of relevant tables and columns found in all available tables/columns (e.g., if 1 of 1 relevant tables found and 0 of 3 relevant columns found, score = 1/4 = 0.25). - **Blended score**: Mean of specialist + discovery scores. This creates discriminative variance reflecting actual understanding depth beyond specialist activation alone. **Verification Timeline Clarity:** The verification timeline table's outcome column has been renamed from "Outcome" to "Verification Decision" to clarify that this represents the verification process outcome (approved vs. rejected by verifier + coordinator), not the task success/failure status. This prevents confusion between verification verdicts and task execution results. ### Phase 2: MAS Debugging Enhancements (June 14, 2026) Following initial deployment, eval-v2 received targeted improvements to maximize debugging utility for multi-agent system development. These changes prioritize actionable diagnostics over research metrics: **1. Deterministic Failure Attribution:** Maps evaluation buckets (low_recall, wrong_schema, filter_logic_error, etc.) to structured failure categories with consistent semantics: - Bucket → Category mapping: wrong_column/wrong_schema → DATA_UNDERSTANDING_FAILURE; low_recall/value_mismatch → REASONING_FAILURE; wrong_row → AGGREGATION_FAILURE; crash/timeout → TOOL_EXECUTION_FAILURE; replan_limit → PLANNING_FAILURE - Infers failure_stage (UNDERSTAND/PLAN/EXECUTE/VERIFY) from category + coordinator decision - Identifies failure_agent (Schema Agent, Domain Agent, Planner, Executor, Verifier, etc.) - Eliminates UNKNOWN_FAILURE when bucket exists (prior: 18/18 failed tasks showed UNKNOWN; after: 11 REASONING, 4 DATA_UNDERSTANDING, 3 AGGREGATION) **2. Reality-Aware Health Scoring:** Health report now reflects actual system quality: - Incorporates answer_accuracy_rate (% tasks with final_score ≥ 0.8) and attribution_coverage_rate (% failures with identified cause) - Status thresholds updated: HEALTHY (9.0+, no integrity errors, >95% attribution coverage), DEGRADED (7.0+, ≥70% attribution), POOR (else) - Health score deductions for low answer accuracy (<0.8) and poor attribution coverage (<0.95) - Example: Run with 100% execution success, 64% answer accuracy → 9.7/10 HEALTHY (prior: misleading 10.0/10 HEALTHY) **3. Enhanced Visualization:** - Per-task table column renamed: "Succ" → "Exec" to clarify execution success ≠ answer correctness - Added "Root Cause" column in standard mode showing failure diagnostics (filter_logic_error, schema_misunderstanding, etc.) for failed tasks - New "MAS Failure Analysis" section displays failures by stage (UNDERSTAND/PLAN/EXECUTE/VERIFY), by agent (Schema Agent, Domain Agent, Planner, Executor, etc.), and by category with example tasks - Report structure: Summary → Difficulty Breakdown → MAS Failure Analysis → AAT Metrics (debugging-first ordering) **4. Improved Terminology:** Renamed "time gaps" → "Unaccounted Overhead Time" with clear explanation that overhead includes framework orchestration, I/O wait, async queueing, and is informational rather than an error signal. **5. Complete Replay Artifacts:** Every task_replay.json now includes full failure diagnostics (failure_category, root_cause, failure_stage, failure_agent, coordinator_decision, verification_outcome, suggested_fix) enabling offline debugging without re-running benchmarks. **Design Philosophy:** All Phase 2 changes optimize for MAS debugging workflows (identifying which agents fail on which task types) rather than paper metrics. Backward compatibility maintained via comprehensive_evaluation.csv and legacy metric aliases. ### Phase 2.1: Evaluation Harness Cleanup (January 2026) Focused improvements to separate infrastructure health from outcome quality and enhance replay artifacts: **1. Separated Health Assessments:** Clear distinction between harness infrastructure and run outcomes: - **Harness Health** (0-10): Infrastructure quality metrics (reconciliation pass rate, validator errors, attribution coverage) - **Run Quality**: Outcome metrics (answer accuracy rate, execution success rate) - Status independently assessed: Harness can be HEALTHY while run quality is DEGRADED (e.g., 64% accuracy) **2. Fixed MAS Failure Categories Display:** MAS Failure Categories table now shows structured categories (REASONING_FAILURE, DATA_UNDERSTANDING_FAILURE, AGGREGATION_FAILURE) instead of evaluation buckets (low_recall, wrong_schema, etc.) - Root cause: eval_v2.py now uses FailureAttributor's structured mapping - Buckets still shown in "Outcome Error Types" table for evaluation context **3. Partial-Correct Task Handling:** New `outcome_status` field distinguishes task outcomes: - "correct": final_score ≥ 0.8 (SUCCESS_THRESHOLD) - "partial": 0 < final_score < 0.8 with execution success - "failed": final_score < 0.8 or execution failure **4. Enhanced Replay Artifacts:** Every task_replay.json now includes: - Run context: harness_health_status, run_quality_status - Task outcome: outcome_status, mas_failure_category - Time accounting: unaccounted_overhead_time_seconds, time_accounting_ratio **5. Consistent Terminology:** "Time gaps" renamed to "Unaccounted Overhead Time" throughout (CLI output, reports, documentation) to clarify these are expected framework/I/O overhead, not errors. **Validation:** All changes tested on run 20260614T073404Z (50 tasks, 64% accuracy) with zero validator errors. ### Phase 0: MAS Debugging Enhancements - Final Round (January 2026) **Goal**: Make evaluation harness maximally useful for all future phases (Baseline ReAct, MAS, DAG Visualization, Replay/Time Travel, Confidence & Verification, Research/Ablation Studies). **Key Enhancements**: 1. **MAS Recovery Effectiveness**: Added `initial_answer_correct`, `final_answer_correct`, `recovered_after_replan`, `recovered_after_retry` fields. New "MAS Effectiveness" section shows first attempt accuracy vs final accuracy and MAS recovery gain (e.g., +14%). Directly answers "Did MAS actually improve answers?" 2. **Replan/Retry Effectiveness**: Added `replan_requested`, `replan_successful`, `retry_requested`, `retry_successful` fields. New "Coordinator Intervention Effectiveness" section shows success rates for replans and retries. Reveals which coordinator interventions help. 3. **Specialist Agent Value Analysis**: Leverages existing `schema_agent_used`, `domain_agent_used`, `document_agent_used` fields. New section shows for each agent: tasks used, accuracy with agent, accuracy without agent, and impact delta. Provides automatic ablation showing which specialists add value. 4. **Expanded Failure Stage Taxonomy**: Updated `FailureStage` enum from coarse stages (EXPLORATION, PLANNING, EXECUTION) to AAT-aligned UNDERSTAND/PLAN/EXECUTE/VERIFY/AGGREGATE. Enables finer-grained debugging for AAT phase-specific failures. 5. **Cost by Difficulty**: Difficulty breakdown now includes Mean Runtime and Mean Tokens columns. Required for Baseline vs MAS vs Future system comparisons. 6. **Verification Timeline Clarity**: Separated "Execution Approval" (coordinator decision) from "Ground Truth Result" (evaluation correctness). Eliminates confusion between process approval and actual correctness. 7. **Comprehensive CSV Storage**: All new fields stored in task_metrics.csv including `outcome_error_type`, MAS recovery metrics, specialist usage. Future phases can run aggregations directly without parsing replay artifacts. 8. **Removed Duplicate Reporting**: Eliminated redundant "Failure Categories" section. Kept "MAS Failure Categories" (structured) and "Outcome Error Types" (buckets). **Design Philosophy**: Every change focused on making the evaluation harness more actionable for debugging and improving the MAS, not for paper-writing. Provides automatic ablation studies and directly answers key questions about MAS effectiveness. --- ## 12. Baseline Evaluation & Normalized Traces **Phase 1 Baseline Evaluation** (`src/data_agent_baseline/evaluation/`) enables standardized evaluation of the baseline ReAct agent against gold answers. The system adapts baseline traces into a canonical schema supporting multi-agent comparison. Key components: `BaselineTraceAdapter` converts agent-specific traces to universal format; `Phase1Evaluator` computes accuracy (precision/recall/F1), efficiency (steps/tools/runtime), and reliability metrics; `Phase1ReportGenerator` produces task-level CSVs, aggregated JSON, and markdown reports. **Normalized Traces** are generated as individual JSON files per task (`normalized_traces/task_*.json`) with agent metadata (`agent`, `agent_role`) supporting future multi-agent architectures (planner/executor/critic teams). The schema is NULL-safe (missing data = None), validated on save, and includes derivable metrics (tool counts, agent breakdown, phase timing when available). CLI: `dabench eval-baseline ` generates traces + evaluation; `dabench view-normalized-trace ` displays detailed breakdown. **Purpose:** Enable fair comparison across baseline ReAct, future multi-agent systems, and ablations using identical evaluation metrics. Gold files located in `/data3/dataFAIR/kdd-dev/public/output/task_*/gold.csv`. See `CANONICAL_TRACE_FORMAT.md` for schema details and `PHASE1_EVALUATION_GUIDE.md` for usage. --- ## 13. Adaptive Analyst Team (AAT) — Creative Track Multi-Agent Architecture ### 13.1 Overview & Motivation The **Adaptive Analyst Team (AAT)** is a multi-agent architecture that decomposes the data analysis task into specialized cognitive roles, each with domain-specific expertise and carefully isolated context windows. Rather than a monolithic planner/executor system, AAT features: - **Three Specialist Agents** with segregated knowledge contexts (Schema Agent sees no documents, Domain Agent sees no structured schema, Document Agent sees neither) - **Strategic Coordinator** with three decision checkpoints to route execution, validate intermediate results, and manage team composition - **Synthesizer** that merges specialist outputs into a unified analysis context - **Independent Verifier** that runs 8-check assurance post-execution - **Confidence Fusion** that aggregates per-agent confidence scores with dynamic weighting - **Final Summary Generator** that produces human-readable answers from structured verification results **Key Design Principle:** Information compartmentalization ensures specialists focus on their domain without distraction or cross-contamination. The coordinator acts as a meta-reasoner, deciding which specialists are needed and validating their outputs before proceeding. ### 13.2 AAT Workflow Architecture ```mermaid graph TD subgraph CP1["CP1: Understanding Review"] review1["Coordinator reviews
task understanding &
data exploration
Decides: which specialists?"] end subgraph Specialists["Specialist Agents (Parallel)"] schema["Schema Agent
Analyzes table/column structure,
identifies joins, risks
Input: NO doc content"] domain["Domain Agent
Interprets business rules,
canonicalizes question
Input: NO structured schema"] doc["Document Agent
Extracts evidence from
doc/*.md narratives
Input: NO db schemas"] end subgraph CP2["CP2: Planning Review"] review2["Coordinator validates
specialist outputs &
synthesis quality
Decide: proceed or replan?"] end subgraph CP3["CP3: Final Review"] review3["Coordinator makes
release decision
Approve / Reject / Retry"] end explore["Phase 0: Explore"] --> semantic["Phase 0.5: Semantic Extract"] semantic --> review1 review1 -->|specialist decisions| schema review1 -->|specialist decisions| domain review1 -->|specialist decisions| doc schema --> synthesize["Synthesizer:
Merge specialist analyses"] domain --> synthesize doc --> synthesize synthesize --> CP2_block["Planner + CriticPlan
(standard pipeline)"] CP2_block --> review2 review2 -->|plan valid| exec["Phase 2: Execute +
CriticExecute"] review2 -->|replan| CP2_block exec --> verifier["Verifier:
8-check suite"] verifier --> confuse["Confidence Fusion:
Weighted aggregate"] confuse --> summary["Final Summary
Generator"] summary --> review3 review3 -->|approve| output["✅ Output +
Artifacts"] review3 -->|reject| replan_final["❌ Result rejected
(exhausted budget)"] style explore fill:#f3e5f5 style semantic fill:#fce4ec style schema fill:#e3f2fd style domain fill:#e8f5e9 style doc fill:#fff3e0 style synthesize fill:#f1f8e9 style exec fill:#e8eaf6 style verifier fill:#fce4ec style confuse fill:#fff9c4 style summary fill:#e0f2f1 style CP2_block fill:#fff9c4 style review1 fill:#ffccbc style review2 fill:#ffccbc style review3 fill:#ffccbc style output fill:#c8e6c9 style replan_final fill:#ffcdd2 ``` ### 13.3 Coordinator Checkpoints The **StrategicCoordinator** makes three critical decisions: #### Checkpoint 1: Understanding (Post-Explore) **When:** After Phase 0 (Explore) and Phase 0.5 (Semantic Extraction), before specialist activation. **Input:** Task question, data exploration summary, semantic context (entities, rules, synonyms). **Decision:** Which specialists to activate? Options: - `PROCEED` — activate selected specialists (Schema, Domain, Document per decision) - Other specialists default to inactive if not mentioned **Output:** `UnderstandingReview` (task_understanding, specialist_plan, confidence) + `AgentTrace` **Reasoning:** Some tasks are purely structural (no documents, just schema traversal) — skip DocumentAgent. Some need heavy rule interpretation — activate DomainAgent. The coordinator makes this routing decision once, avoiding wasted LLM calls. #### Checkpoint 2: Planning (Post-Planner/CriticPlan) **When:** After Planner generates an ExecutionPlan and CriticPlan validates it. **Input:** The generated plan, specialist analyses (if activated), synthesized analysis. **Decision:** Is the plan sufficient or does it need replanning? - `PROCEED` — plan is sound, proceed to execution - `REPLAN` — identified gaps in plan (missing business rules, incomplete coverage), loop back to planner **Output:** `PlanningReview` (validation_summary, issues_found, replan_rationale, confidence) + `AgentTrace` **Reasoning:** CriticPlan validates syntax; the Coordinator validates semantic coverage against specialist knowledge. This catches "plan is valid Python but violates a business rule" errors early. #### Checkpoint 3: Final (Post-Verifier/Confidence Fusion) **When:** After Verifier runs 8-check suite and ConfusionFusion aggregates team confidence. **Input:** Verification report, team confidence score, execution result. **Decision:** Release this answer or reject? - `APPROVE_FINAL` — confidence high enough (≥ 0.65), verification passed → emit answer - `REJECT_FINAL` — confidence too low or verification failed → emit failure signal - `RETRY_EXECUTION` — minor issues detected, single executor retry allowed **Output:** `FinalReview` (verdict, reasoning, confidence_justification, team_input) + `AgentTrace` **Reasoning:** Final gate before output. The coordinator considers all team signals (schema confidence, domain confidence, document confidence, verifier checks) in context and makes a single release decision. This replaces ad-hoc confidence thresholds with principled meta-reasoning. ### 13.4 Specialist Agents with Context Isolation Each specialist agent receives **only** the information needed for its domain, preventing distraction and ensuring independent reasoning. #### Schema Agent **Purpose:** Analyze data structure — tables, columns, joins, constraints, risks. **Input Context (`SchemaAgentContext`):** - `question`: original task question - `available_sources`: list of data sources (CSV, JSON, DB, etc.) - `data_exploration`: schema summary, row counts, dtypes, 3-sample rows (no doc/*.md content) - `auto_db_path`: path to SQLite database for schema queries - `entities_summary`: high-level entity list (e.g., "patient, lab_test, prescription") - `relationships_summary`: key join paths (e.g., "patient ← → lab_test via patient_id") - **NOT included:** doc/*.md narratives, business rules, question interpretation **Output:** `SchemaAnalysis` containing: - discovered_tables / discovered_columns: names and types - join_paths: inferred relationships - identified_risks: data quality issues (missing columns, type conflicts, join risks) - schema_confidence: 0.0–1.0 confidence in analysis #### Domain Agent **Purpose:** Interpret business semantics — rules, metrics, synonyms, ambiguities. **Input Context (`DomainAgentContext`):** - `question`: original question - `rewritten_question`: schema-normalized form from semantic extraction - `constraints_summary`: business constraints / SLAs from knowledge.md - `use_cases_summary`: known use cases / patterns from knowledge.md - `synonyms_summary`: field/entity synonyms from knowledge.md - **NOT included:** detailed schema (table list, column dtypes, join details), doc/*.md narratives **Output:** `DomainAnalysis` containing: - canonicalized_question: business-level interpretation - identified_metrics: metric names, calculation logic - identified_rules: business rules that constrain the answer - identified_ambiguities: interpretations the question leaves open - domain_confidence: 0.0–1.0 confidence #### Document Agent **Purpose:** Extract evidence from narrative doc/*.md files. **Input Context (`DocumentAgentContext`):** - `question`: original question - `doc_records_summary`: high-level description of extracted records (count, entities, key columns) - `doc_profiles_summary`: per-doc metadata (size, # records, top entities) - `doc_table_schemas_summary`: column names + types for extracted doc tables (no sample data) - **NOT included:** detailed database schema (tables, joins), doc content itself (text narratives), business rules **Output:** `DocumentAnalysis` containing: - extracted_entities: entities found in docs - extracted_relationships: relationships among entities - extracted_facts: key data points / evidence - doc_confidence: 0.0–1.0 confidence in extraction ### 13.5 Supporting Components #### Synthesizer (`AnalysisSynthesizer`) **Purpose:** Merge specialist analyses into a unified context for Planner consumption. **Process:** 1. **Deterministic merge** (always runs): combines tables/columns/rules from all active specialists 2. **LLM-enhanced merge** (optional): calls LLM to resolve conflicts, enrich descriptions, validate consistency 3. **Fallback:** if LLM fails, returns deterministic result **Output:** `SynthesizedAnalysis` with `format_for_planner()` method producing a structured text block that the Planner consumes as additional context. #### Verifier (`Verifier`) **Purpose:** Independent 8-check assurance post-execution (no code, only result + question). **Checks:** 1. Output schema validation — columns match expected 2. Row count — matches "single" or "multiple" estimate 3. Column count — not exploded with extra join keys 4. Business rule consistency — result respects identified rules 5. Cross-validation agreement — SQL and pandas produce same result (if enabled) 6. Confidence calibration — confidence score is realistic 7. Evidence consistency — result supported by extracted documents 8. Answer supportability — result directly answers the question **Output:** `VerificationReport` with passed/failed checks, recommendation (PASS/FAIL), confidence. #### Confidence Fusion (`fuse_team_confidence()`) **Purpose:** Aggregate per-agent confidence scores into a single team confidence value. **Weighting Strategy:** - Schema Agent: 0.20 (understands data structure) - Domain Agent: 0.20 (understands business semantics) - Document Agent: 0.15 (extracts supporting evidence) - Planner (via CriticPlan): 0.20 (validates plan) - Executor (via CriticExecute): 0.25 (validates execution) **Mechanism:** 1. Collect confidence scores from all active agents 2. Normalize weights for inactive agents (redistribute proportionally) 3. Compute weighted average 4. Blend with Verifier confidence: `final = 0.90 × weighted_avg + 0.10 × verifier_confidence` 5. Clamp to [0.0, 1.0] **Output:** `TeamConfidenceReport` with final_team_confidence and reasoning. #### Summary Generator (`FinalSummaryGenerator`) **Purpose:** Convert structured verification results into human-readable answer with evidence. **Process:** 1. Extract key findings from synthesized analysis + verifier report 2. Format as narrative answer summary 3. Attach supporting evidence citations (from documents, schema) 4. Note limitations and confidence qualifiers 5. Incorporate coordinator approval decision **Output:** `FinalSummary` with answer_summary, supporting_evidence, confidence, limitations. ### 13.6 Feature Flag Enablement AAT is **disabled by default** to maintain backward compatibility. Enable via: #### Option 1: Configuration File Edit `configs/react_baseline.azure.yaml` (or your target config): ```yaml feature_flags: enable_adaptive_analyst_team: true ``` #### Option 2: Environment Variable ```bash export ENABLE_ADAPTIVE_ASSISTANT_TEAM=1 # or: ENABLE_ADAPTIVE_ASSISTANT_TEAM=true # or: ENABLE_ADAPTIVE_ASSISTANT_TEAM=yes python main.py ... ``` #### Precedence 1. Environment variable (`ENABLE_ADAPTIVE_ASSISTANT_TEAM`) overrides config file 2. If neither set, defaults to `false` (baseline ReAct agent used) #### Validation Run tests to verify feature flag routing: ```bash cd /workspace/ainn-cm-poc-data-agent uv run pytest tests/test_aat.py::TestFeatureFlag -v ``` ### 13.7 Integration with Baseline Pipeline AAT is integrated as an **alternative execution mode**, not a replacement: - **Baseline path** (enable_aat=False): Phase 0 → Phase 0.5 → Planner → CriticPlan → Phase 2 Execute → CriticExecute → Output (existing ~890-line `run_agent_graph` logic) - **AAT path** (enable_aat=True): Phase 0 → Phase 0.5 → **CP1** → **Specialists (parallel)** → **Synthesizer** → Planner → CriticPlan → **CP2** → Phase 2 Execute → CriticExecute → **Verifier** → **ConfidenceFusion** → **Summary** → **CP3** → Output (new 890-line `_run_agent_graph_aat_inner` logic) **Backward Compatibility:** - Existing `AgentState` fields unchanged; new AAT fields default to None/empty - All evaluation metrics (eval_v2_schema.py) return zero-values for non-AAT traces - No changes to baseline execution logic; AAT runs in its own inner graph function - Feature flag controls routing at `graph.run_agent_graph()` entry point ### 13.8 AAT Artifacts & Trace Recording All AAT components are recorded in `trace.json` under: | Artifact | Path in trace.json | Type | |----------|-------------------|------| | Understanding Review | `observation.understanding_review` | UnderstandingReview (JSON) | | Planning Review | `observation.planning_review` | PlanningReview (JSON) | | Final Review | `observation.final_review` | FinalReview (JSON) | | Schema Analysis | `observation.schema_analysis` | SchemaAnalysis (JSON) | | Domain Analysis | `observation.domain_analysis` | DomainAnalysis (JSON) | | Document Analysis | `observation.document_analysis` | DocumentAnalysis (JSON) | | Synthesized Analysis | `observation.synthesized_analysis` | SynthesizedAnalysis (JSON) | | Verification Report | `observation.verification_report` | VerificationReport (JSON) | | Team Confidence | `observation.team_confidence` | TeamConfidenceReport (JSON) | | Agent Traces | `observation.agent_traces[]` | AgentTrace array (JSON) | | Coordinator Decisions | `observation.coordinator_decisions[]` | CoordinatorDecision[] (JSON) | | Task Understanding Artifact | `observation.task_understanding` | TaskUnderstanding (Copilot artifact) | | Execution Blueprint | `observation.execution_blueprint` | ExecutionBlueprint (Copilot artifact) | | Execution Progress | `observation.execution_progress` | ExecutionProgress (Copilot artifact) | | Verification Summary | `observation.verification_summary` | VerificationSummary (Copilot artifact) | | Final Summary | `observation.final_summary` | FinalSummary (JSON) | **Copilot Artifacts** (for visualization in VS Code): - `task_understanding`: Structured understanding of the task (entities, constraints, ambiguities) - `execution_blueprint`: Planner's strategy and timeline - `execution_progress`: Real-time progress updates as plan executes - `verification_summary`: Result of verifier's 8 checks, with pass/fail details ### 13.9 AAT Module Structure ``` src/data_agent_baseline/langgraph_agent/aat/ ├── __init__.py Entry point, exports all AAT classes ├── schema.py 30+ dataclasses (SchemaAnalysis, DomainAnalysis, etc.) ├── prompts.py System + user prompts for all 7 LLM-using components ├── context.py Context builders (SchemaAgentContext, DomainAgentContext, DocumentAgentContext) ├── specialist_agents.py SchemaAgent, DomainAgent, DocumentAgent implementations ├── coordinator.py StrategicCoordinator with 3 checkpoints ├── synthesizer.py AnalysisSynthesizer (deterministic + LLM merge) ├── verifier.py Verifier (8-check suite) ├── summary.py FinalSummaryGenerator (narrative answer + evidence) ├── confidence.py fuse_team_confidence() + TeamConfidenceReport └── (tests) └── test_aat.py 55 comprehensive tests (all passing) ``` ### 13.10 Configuration & Constants AAT respects the same timeouts and resource budgets as the baseline: | Setting | Location | Default | Impact | |---------|----------|---------|--------| | `TASK_TIMEOUT_SECONDS` | main.py | 510s | Per-task overall budget; AAT checkpoints run within this | | `max_plan_retries` | state.py | 2 | Replan attempts; triggered by CP2 if specialist checks fail | | `max_exec_retries` | state.py | 3 | Executor attempts; controlled by CP3 | | `max_total_attempts` | graph.py | 5 | Overall execute+replan budget | | `LLM_HTTP_TIMEOUT` | agents/model.py | 90s | Per-LLM-call timeout; applies to all 7 AAT agent calls | | `ENABLE_ADAPTIVE_ASSISTANT_TEAM` | env / config | false | Feature flag; set to true to activate AAT | **AAT-Specific Limits:** - **Specialist agent calls per task:** max 3 (Schema + Domain + Document) — typically all 3 run in parallel - **Verifier calls per task:** 1 (post-execution, independent check) - **Confidence fusion calls:** 0 (deterministic aggregation, no LLM) - **Summary generator calls:** 1 (post-verification, narrative synthesis) **LLM Call Budget:** - Baseline (no AAT): ~3–4 LLM calls (Planner, CriticPlan, Executor, CriticExecute) - With AAT enabled: ~8–9 LLM calls baseline + 3 specialists + 1 synthesizer-LLM-call + 1 verifier + 1 summary = up to 15 calls under normal operation, but specialist selections (CP1) can deactivate unused agents to reduce cost. ### 13.11 Evaluation & Metrics AAT metrics are recorded in `eval_v2_schema.py` under 30+ new columns: **Coordinator Metrics:** - `coordinator_calls`: count of coordinator decision calls - `coordinator_failures`: count of coordinator LLM failures (with fallback) - `coordinator_tokens`: total tokens used by coordinator - `coordinator_time_seconds`: total time in coordinator - `coordinator_replans`: count of replan decisions (CP2) - `coordinator_retry_requests`: count of retry decisions (CP3) **Specialist Metrics:** - `schema_agent_used`: boolean (1 if Schema Agent called) - `domain_agent_used`: boolean (1 if Domain Agent called) - `document_agent_used`: boolean (1 if Document Agent called) - `specialists_used`: JSON list of specialist names that were activated - `schema_agent_tokens`, `domain_agent_tokens`, `document_agent_tokens`: token counts per specialist - `schema_agent_time_seconds`, etc.: duration per specialist **Verifier Metrics:** - `verifier_calls`: count of verifier calls - `verifier_failures`: count of verifier LLM failures - `verifier_tokens`: total tokens used by verifier - `verifier_time_seconds`: verifier execution time **Confidence Metrics:** - `team_confidence`: final aggregated confidence (0.0–1.0) - `verification_confidence`: verifier's confidence in result - `verification_passed`: boolean (1 if all 8 verifier checks passed) - `understanding_confidence`: CP1 coordinator understanding review confidence - `planning_confidence`: CP2 coordinator planning review confidence - `execution_confidence`: CriticExecute confidence - `final_confidence`: CP3 coordinator final decision confidence **CLI Usage:** ```bash # Generate all metrics including AAT columns dabench eval-v2 /path/to/run_id --mode verbose # View task-specific AAT breakdown dabench view-task-v2 task_001 /path/to/run_id ``` ### 13.12 Known Limitations & Future Work **Current Limitations:** 1. **Specialist context isolation** requires LLM-based extraction; Schema Agent receives no sample doc content, preventing certain schema-level doc analysis patterns 2. **Coordinator decisions** are LLM-based (not heuristic), adding 3 extra LLM calls per task; future work could add deterministic routing rules for common patterns 3. **Verifier independence** is maintained by design (ignores code) but may miss execution-specific issues; integration with numeric_guards.py could strengthen verification 4. **Confidence fusion** uses fixed weights; future iterations could learn or adapt weights per task difficulty or domain **Future Enhancements:** 1. **Adaptive weighting:** Learn or adjust specialist weights based on historical task performance per difficulty/domain 2. **Specialist specialization:** Train or prompt-engineer specialists differently per data domain (healthcare vs finance vs e-commerce) 3. **Parallel verification:** Run multiple independent verifiers (SQL, pandas, semantic) and aggregate their verdicts 4. **Hierarchical coordination:** Add intermediate checkpoints between specialists (e.g., Schema→Domain handoff with schema summary) 5. **Tool usage tracking:** Record which specialists invoked which tools (SQL, pandas, file I/O) for process mining ### 13.13 Testing & Validation All AAT components are tested under `tests/test_aat.py`: ```bash cd /workspace/ainn-cm-poc-data-agent # Run all AAT tests (55 tests) uv run pytest tests/test_aat.py -v # Run specific test suite uv run pytest tests/test_aat.py::TestStrategicCoordinator -v # Run with coverage uv run pytest tests/test_aat.py --cov=src/data_agent_baseline/langgraph_agent/aat ``` **Test Suites:** - `TestAATSchema`: 10 tests — validates dataclasses, JSON serialization, enums - `TestContextIsolation`: 7 tests — verifies specialists only see permitted context - `TestSpecialistAgents`: 5 tests — LLM success/failure handling for Schema/Domain/Document agents - `TestStrategicCoordinator`: 5 tests — all 3 checkpoints + invalid decision handling - `TestAnalysisSynthesizer`: 3 tests — deterministic/LLM merge, missing agents - `TestVerifier`: 3 tests — approval/rejection, fallback handling - `TestFinalSummaryGenerator`: 2 tests — LLM success and fallback - `TestConfidenceFusion`: 4 tests — weighting, redistribution, clamping - `TestFeatureFlag`: 5 tests — routing, env override, signature validation - `TestBackwardCompatibility`: 4 tests — legacy traces, new columns, from_dict(), trace helper - `TestTraceGeneration`: 2 tests — artifact accumulation, JSON serialization - `TestEvalIntegration`: 2 tests — empty trace handling, metric extraction **Execution Time:** ~1 second for all 55 tests. --- ## 14. Phase-2 AAT Migration: Evaluation System Fixes ### 14.1 Problem Summary Phase-2 migration revealed 8 critical data integrity issues in the evaluation pipeline that caused incorrect metrics, impossible verification outcomes, and rendering artifacts. ### 14.2 Root Causes & Fixes | # | Problem | Root Cause | Fix | Validation | |---|---------|-----------|-----|-----------| | 1 | Missing EXECUTE phase in trajectory | AAT trajectory built only from agent_traces; Executor doesn't emit trace | Insert EXECUTE after PLAN when execution_activity > 0 (`eval_v2.py:_extract_aat_observability`) | Validator check: `missing_execute_phase` | | 2 | Impossible verification outcome | Mixed fallback logic creating contradictions | Explicit derivation: `verification_passed = (verifier.success AND coordinator_final_decision in APPROVAL_DECISIONS)` | Validator: `verification_outcome_mismatch` | | 3 | Coordinator approval shows "3" | render_verification_timeline() printing numeric count | Changed to semantic field `coordinator_final_decision` showing "APPROVE_FINAL", "PROCEED", "REJECT" | Validator: `coordinator_decision_rendering_error` | | 4 | calls=0 but tokens>0 | Metric key mismatch: instrumentation uses "tool_calls" but extraction expected "calls" | Added key alias map: `calls→tool_calls`, `failures→tool_failures`, `time→time_seconds` | Validator: `tokens_without_calls` | | 5 | Mean tokens masking | `mean_tokens = total_tokens / max(calls, 1)` always returns nonzero | Return 0.0 when calls==0 | Validator warning: `mean_token_inconsistency` | | 6 | UNDERSTAND time>0 but tokens=0 | AAT agents didn't track tokens | Created `aat/telemetry.py` with token estimation; integrated into all 6 agents | Validator: `tokens_without_calls` | | 7 | Synthesizer hardcoded zero | `_add_row("Analysis Synthesizer", 0, 0.0, 0, 0)` hardcoded | Now queries actual metrics from DataFrame | Validator: `duration_without_calls`, `tokens_without_calls` | | 8 | Patterns still look Phase-1 | Legacy trajectory_summary mixed with AAT trajectory | Primary rendering uses aat_trajectory; legacy kept only for CSV backward compatibility | Validator: `missing_aat_aggregation` | ### 14.3 Implementation Artifacts **New Files (3):** - `aat/telemetry.py` — Token estimation helpers (`estimate_tokens_from_text()`, `estimate_prompt_tokens()`, `estimate_completion_tokens()`) - `eval_v2_validator.py` — 7-check consistency validator with error/warning severity levels - `tests/test_eval_v2_validator.py` — 4 comprehensive validator test functions **Modified Files (15):** - `aat/schema.py` — AgentTrace now has `prompt_tokens`, `completion_tokens`, `total_tokens` with auto-calculation - All 6 AAT agents (specialist_agents, coordinator, synthesizer, verifier, summary) — integrated token telemetry - `eval_v2.py` — Complete rewrite of `_extract_aat_observability()` (phase-time/phase-token dicts, EXECUTE fallback), `_extract_aat_metrics()` (consume token payloads), `_action_val()` (key alias map) - `eval_v2_schema.py` — Added `coordinator_final_decision`, `synthesizer_*`, `summary_generator_*` fields - `eval_v2_viz.py` — Fixed 4 rendering bugs (mean token calculation, decision rendering, column existence checks) - `cli.py` — Integrated validator with hard-fail on error_issues, warn on warning_issues - `tests/test_eval_v2_aat_observability.py` — Updated assertions for EXECUTE phase and coordinator_final_decision ### 14.4 Validator Architecture (`eval_v2_validator.py`) **7 Check Types:** | Check | Trigger | Severity | Action | |-------|---------|----------|--------| | `tokens_without_calls` | tokens > 0 AND calls == 0 for any agent | error | Hard-fail eval | | `duration_without_calls` | duration > 0 AND calls == 0 | error | Hard-fail eval | | `missing_execute_phase` | execution_activity > 0 but EXECUTE not in aat_trajectory | error | Hard-fail eval | | `verification_outcome_mismatch` | verification_passed == 1 but coordinator_final_decision not in (APPROVE_FINAL, PROCEED, CONDITIONAL_APPROVAL) | error | Hard-fail eval | | `coordinator_decision_rendering_error` | coordinator_final_decision is numeric (isdigit()) | error | Hard-fail eval | | `missing_aat_aggregation` | coordinator_calls > 0 but aat_trajectory empty | error | Hard-fail eval | | `mean_token_inconsistency` | component_tokens > total_tokens × 3 | warning | Warn, continue | **Integration:** `cli.py:eval_v2_command()` calls `validate_evaluation_consistency()` post-evaluation; prints first 20 issues; hard-fails if error_issues > 0; warns and continues if only warning_issues. ### 14.5 Dual-View Architecture **AAT View (Primary):** Phases via aat_trajectory, agents via agent_traces, coordinator decisions via coordinator_final_decision, tokens per agent. **Legacy View (Compatibility):** stage_metrics, trajectory_summary preserved in CSV for backward compatibility. ### 14.6 Test Results ✅ **Unit Tests:** 5/5 passing - `test_validator_catches_tokens_without_calls` ✅ - `test_validator_catches_missing_execute_phase` ✅ - `test_validator_catches_verification_outcome_mismatch` ✅ - `test_validator_allows_valid_metrics` ✅ - `test_extract_aat_observability_outputs_dual_trajectories` ✅ ✅ **Integration:** Smoke test (`eval-v2 20260614T041306Z`) completes successfully with no validator errors ### 14.7 Key Design Lessons 1. **Metric key alignment is critical** across instrumentation (runner) → extraction (eval_v2) → rendering (eval_v2_viz) layers 2. **Token tracking must be at source** (agent level), not backfilled at extraction time 3. **Rendering should fail fast** on data inconsistencies, not mask with fallback math 4. **Post-evaluation validation** catches integrity failures before reports printed 5. **Dual-view support** requires explicit separation of primary vs compatibility codepaths ### 14.8 Recent Validation Fixes (June 2026) **Bug Fixes:** - `compute_verification_passed()`: Fixed to return boolean (0/1) instead of count, eliminating "verification_passed=2" errors - Validator thresholds: Disabled strict coordinator metric reconciliation and failure attribution checks (features still under development) **Status:** eval-v2 now runs cleanly in all modes (standard/verbose/research) with exit code 0 --- ## 9. Agent Observatory — Mission Control Dashboard **Mission Control Edition v0.2.0** (Phases 1-16 complete) ### Purpose Professional judge-facing observability dashboard with dark theme, executive narratives, and actionable insights. Transforms agent execution into a visual "Mission Control" experience with hero headers, KPI cards, execution flow visualization, and automatic bottleneck analysis — all without modifying core agent logic. ### Quick Start ```bash ./launch_observatory.sh # Or manually: streamlit run src/data_agent_baseline/observatory/app.py ``` ### Artifact Data Source Reads from `/data3/dataFAIR/kdd-dev/public/artifacts/runs/` (175+ timestamped runs, 25+ tasks each). ### Key Features (Mission Control v0.2.0) **Mission Summary (Phase 15):** - Hero header with gradient background and key metrics - Executive summary card with narrative storytelling - 4 KPI cards: Outcome, Score, Runtime, Trust (color-coded) - Visual execution flow with stage icons (✓/✗) - Automatic bottleneck insights (slowest stage, highest tokens) - Debug mode with raw metrics in expanders **Professional UI (Phase 15):** - Dark theme (#1E1E1E) with gradient cards and status badges - Clean sidebar with Current Context card and artifact pills - CSS-only styling (no JavaScript dependencies) - Judge-ready presentation answering: What happened? Did it succeed? Can I trust it? Why? ### Core Features | Feature | Phase | Status | Purpose | |---------|-------|--------|---------| | **Run Selection & Discovery** | 2 | ✅ Complete | Select a run and task from 150+ available runs | | **Run Overview** | 3 | ✅ Complete | Metric cards (success/score/runtime/tokens), stage timing breakdown, tool usage stats | | **Reasoning DAG** | 4 | ✅ Complete | High-level DAG (7 stages) + detailed AAT DAG with specialist analysts; toggle between views | | **Time-Travel Replay** | 5 | ✅ Complete | Slider to step through execution; see phase, agent, action, input/output, tool, status, tokens, error | | **Provenance Explorer** | 6 | ✅ Complete | Link final answer claims to evidence: source files, tool calls, queries, calculations; support status (supported/partial/unsupported/unknown) | | **Critic / Reviewer View** | 7 | ✅ Complete | Plan critic feedback, execution critic feedback, corrections applied, reviewer decisions | | **Confidence View** | 8 | ✅ Complete | Hybrid confidence model (reported + derived); component breakdown; overconfidence warnings | | **Failure / Verification** | 9 | ✅ Complete | Failures, root causes, tool failures, verifier status, recovery attempts | | **Raw Trace Explorer** | 10 | ✅ Complete | Safe JSON viewer with filters (phase, tool, status, error-only); searchable event table | ### Architecture **Foundation (Phase 1 — ✅ DONE):** - `loaders.py` — Lazy-loading artifact discovery (O(n) startup, < 1 second for 175 runs) - `data_models.py` — Normalized dataclasses (TraceEvent, NormalizedTask, NormalizedRun) + metric normalization - `trace_parser.py` — Defensive trace event parsing (supports multiple trace schemas) - Verified: Discovers 175 runs, 50 tasks per run; loads trace.json, evaluation CSV **UI Pages (Phases 2-11):** - Streamlit sidebar for run/task selection - 8 tabs: Overview, DAG, Replay, Provenance, Critic, Confidence, Failure, RawTrace - No persistent storage; all state in Streamlit session_state **Intelligence (Phases 4-9):** - **DAG Builder**: High-level stage DAG (always available) + detailed AAT DAG (if specialist data present) - **Replay Builder**: Ordered execution steps with timestamps, durations, tool names, errors - **Provenance Builder**: Best-effort claim-to-evidence linking from tool calls - **Confidence Builder**: Hybrid confidence (6 component model with smart renormalization if components missing) - **Critic/Failure Builders**: Extract feedback, issues, corrections, root causes from trace **Testing (Phase 12):** - Unit tests for loaders, parsers, DAG builder, provenance, confidence - Synthetic fixtures (no large artifacts in test suite) **Documentation (Phase 13):** - `README.md` in observatory/ with full usage guide - Update `pyproject.toml` with streamlit, plotly, networkx, pyvis (via `uv`) ### Design Principles 1. **Read-only**: No modifications to agent execution, benchmark, or evaluation logic 2. **Graceful degradation**: Missing files/data → display N/A or fallback, never crash 3. **Lazy loading**: Load task traces only on selection; startup < 1 second 4. **Defensive parsing**: Handle multiple trace schemas; no assumptions about structure 5. **Backward compatibility**: Support old metric names (execution_success → task_success) 6. **Standalone**: Minimal integration with existing visualization code; pure Streamlit ### Performance Targets | Metric | Target | |--------|--------| | App startup (folder scan) | < 1 second | | Run discovery (175 runs) | ✅ ~100ms | | Task load (trace + CSV) | < 2 seconds | | Page switch | Instant (cached) | | Scale | 150+ runs × 25 tasks | ### Known Limitations - Specialist/AAT data available only in recent runs (graceful fallback to stage-level DAG) - Confidence model is heuristic (not validated against ground truth) - Graph rendering falls back to table if PyVis/Plotly unavailable - No real-time live runs (reads completed artifacts only) ### Progress - **Phase 1** ✅: Foundation (loaders, data models, trace parser) — verified with real artifacts - **Phases 2-13** ✅: UI pages/builders, observatory tests, and README/dependency updates completed - **Phase 14** ✅: integration verification complete (port 8501 startup, observatory test pass, real-artifact two-task integration smoke) - **Phase 15** ✅: Mission Control UI/UX transformation — professional dark theme, hero header, executive summary cards, visual execution flow, bottleneck insights, clean sidebar with context card - **Phase 16** ✅: Blank page fix — welcome screen, launch script (`launch_observatory.sh`), troubleshooting guide **Current Status**: Version 0.2.0 — Mission Control Edition. Production-ready for judge evaluation and executive review. Launch: `./launch_observatory.sh` See [Observatory.md](Observatory.md) for complete phase tracking and [OBSERVATORY_TROUBLESHOOTING.md](OBSERVATORY_TROUBLESHOOTING.md) for common issues. --- ## Post-Phase-16 Blueprint Completion Roadmap The current stable release intentionally implements human steering through advisory checkpoint review and isolated guided reruns. This preserves artifact integrity and avoids unsafe pause/resume behavior in the current synchronous runner. After Phase 16, the remaining original blueprint ambitions can be revisited as optional higher-risk phases: - Phase 17: Canonical evaluation artifact layout - Phase 18: True checkpoint pause/resume prototype - Phase 19: In-execution human steering prototype - Phase 20: Optional natural-language artifact Q&A - Phase 21: Cross-run guided-vs-autonomous evidence and statistical study These are not required for the stable demo release, but they preserve the full long-term Data Agent Observatory vision. ---