docs/overview/Overview.md

KDD Cup 2026 Data Agent — Architecture Overview

1. What This Is

A multi-agent LLM-based system that reads data tasks (question + context data), plans SQL/pandas queries, executes them, and produces prediction CSVs. Built for the KDD Cup 2026 Data Agent Challenge.

Input: A task.json (question + difficulty) plus a context/ folder containing CSVs, JSONs, SQLite DBs, knowledge.md, and doc/*.md narrative files.

Output: A prediction.csv with the answer, plus a trace.json audit trail.

DAO — Data Agent Observatory

NEW: DAO (Data Agent Observatory) is a Streamlit-based observability and exploration application that transforms agent reasoning into a professional, judge-facing "Mission Control" experience.

Launch: streamlit run src/data_agent_baseline/observatory/app.py

Features: - Top-Level Modes: CHAT (default landing) and DEBUG - Run Launcher + Live Execution (Phases 4-5): Launch predefined benchmark tasks from the UI and monitor progress/events in-session - Zero-Click Startup (DEBUG mode): Auto-discovers runs, pre-selects latest run and first task - Executive Summary: Mission overview, decision reasoning, refinement timeline, deliverables <<<<<<< HEAD - 14 Debug Tabs: Run Launcher, Run Intelligence, Mission Summary, Reasoning DAG, Time-Travel Replay, Checkpoint Review, Guided Rerun Plan, Rerun Comparison, Provenance, Critic/Reviewer, Confidence, Failure/Verification, Raw Trace, Demo

• Consolidated Primary Navigation (Phase 19): Run Launcher, Run Intelligence, Task Intelligence, Demo / Future Proof
• Fallback Views Preserved: Live Trace Flow and Guided Live Trace Flow (HITL) remain available under Advanced / Fallback Views without runtime refactor

  origin/ph2

• Professional UI: Dark theme with gradient cards, status badges, clean sidebar
• Judge-Ready: Suitable for competition evaluation and executive presentation

Phase 19 Status (2026-07)

• Primary navigation now uses four top-level pages: Run Launcher, Run Intelligence, Task Intelligence, Demo / Future Proof.
• Task Intelligence replaces Mission Summary as the task-level hub and is now consolidated into five judge-oriented internal sections:
  • Summary
  • Reasoning & Replay
  • Evidence
  • Review & Reliability
  • Raw Trace
• Reasoning & Replay uses nested subsections: Reasoning Flow and Guided Step Inspector.
• Review & Reliability uses nested subsections: Review Signals, Trust Calibration, and Diagnosis & Verification.
• Demo / Future Proof groups checkpoint, rerun, comparison, evidence, and demo workflows.
• Live Trace Flow and Guided Live Trace Flow remain intentionally untouched as fallback/stability pages and are opened directly via the sidebar View selector.
• Advanced / Fallback Views informational copy was moved out of the primary-page flow and into Task Intelligence guidance text.
• Run Intelligence now includes a compact Verification Summary with metric tooltips (calculation + source), and Guided Rerun Cohort Analysis is shown in Summary view only.
• No execution, evaluation, HITL runtime behavior, or artifact schema changes were introduced.
• Phase 20 natural-language artifact Q&A remains explicitly out of scope.

Quick Start:

1. Run the command above
2. App auto-loads latest run and first task (< 2s)
3. Open Task Intelligence for task-level narrative and deep inspection
4. Use other tabs for detailed analysis

See Observatory.md for complete documentation.

Analyst Team Disagreement Layer (Additive)

The AAT path now includes a lightweight analyst-collaboration layer that captures:

• per-analyst structured opinions (aat_analyst_opinions)
• structured disagreement items (aat_disagreements)
• coordinator synthesis (aat_coordinator_synthesis)

This layer is additive and does not rewrite the baseline graph. Existing stages, retries, and output artifacts are preserved.

Calibration update: disagreement detection is now intent-gated and tuned for meaningful disagreement reporting (not routine confidence noise).

Reporting stabilization update (2026-06-16): coordinator metric reconciliation mismatches are diagnostic warnings (non-fatal), and meaningful disagreement counts now prioritize analyst-team meaningful flags over raw disagreement signals.

Auditor calibration update: guard outputs include structured audit payloads (passed, confidence, detected_risks, evidence, suggested_action), and eval-v2 reports failure-prevention effectiveness via critical disagreement score plus auditor precision/recall.

Phase 0 + Phase 1 Status (2026-06)

• Phase 0 command contract is preserved: existing CLI behavior remains stable for run-task, run-benchmark, eval-v2, and task viewing workflows.
• Phase 1 run execution refactor is complete: run orchestration was extracted into a shared application service and domain/repository boundaries.
• Behavioral compatibility is maintained: the refactor is additive and does not change core solve logic or baseline output semantics.
• Run artifacts are now structured and explicit: run_manifest.json and run_events.jsonl are written as additive artifacts for selected-task and benchmark execution paths.
• Benchmark compatibility remains intact: summary.json is preserved for benchmark runs.
• Task-level deliverables are unchanged: task folders still emit trace.json and prediction.csv.

Phase 2 Status (2026-06)

• Phase 2 evaluation refactor is complete: eval-v2 orchestration now runs through a reusable EvaluationService (src/data_agent_baseline/application/evaluation_service.py).
• CLI remains backward-compatible: dabench eval-v2 keeps the same arguments/defaults and now acts as a thin adapter (parse options, delegate to service, render selected report, print summary, exit by bundle status).
• Artifact contract remains unchanged: all eval-v2 outputs remain at run root (task_metrics.csv, trajectory.csv, tool_calls.csv, comprehensive_evaluation.csv, validation/auditor/hardening reports, and per-task task_replay.json).
• Validation/hardening semantics remain unchanged: warning-only validation exits 0, error validation exits 1, all three report modes are validated, and duplicate-section detection is preserved.

Phase 3 Status (2026-06)

• Phase 3 Streamlit Run Intelligence is complete: a dedicated run-level intelligence view is available in the Observatory app and consumes existing evaluation artifacts.
• Run Health Summary first: the page opens with compact run-level health cards (Harness Health, Execution Success Rate, Failed Tasks, Mean Score and related run KPIs).
• Trajectory Intelligence section: focused analysis of full-run trajectory behavior including:
  • Trajectory KPI Snapshot (tasks, patterns, path length, branching, critic loops, success rates, recovery rates)
  • Dominant Trajectory Patterns with interpretations (Clean, Retry, Replan, Complex Recovery)
  • Phase Timeline & Token Economics with per-phase time, tokens, tools, failures, confidence
  • Recovery / Self-Correction Flow (first-try rate, recovery success, planner/coordinator/execution retries)
  • Coordinator Decision Visibility (PROCEED, REQUEST_SPECIALIST, REPLAN, RETRY_EXECUTION, APPROVE_FINAL)
  • Trajectory Health Labels (Clean, Recovered, Unrecovered, Overconfident, Expensive, Complex)
  • Trajectory Examples and a unified recommended-task deep link into Task Intelligence
• Reliability Diagnostics section: compact Root Cause Distribution panel with summary cards, task inspector, and optional raw/toggle views.
• View levels: Summary (judge-focused), Detailed (comprehensive), Research (all tables + advanced metrics)
• Missing-evaluation flow is read-only and informative: when artifacts are missing, the page renders a no-eval fallback with execution summary, task status, artifact coverage, event timeline summary, and explicit eval-v2 guidance.
• Compatibility is preserved: eval-v2 artifact schemas and locations are unchanged; Phase 3 adds presentation/adapters/tests only.

Phase 4 Status (2026-06)

• Phase 4 Streamlit Run Launcher is complete: a new Run Launcher tab in the Observatory lets users select predefined benchmark tasks and execute them without leaving the UI.
• Execution goes through the shared service boundary: the launcher calls RunExecutionService.execute_selected_tasks() exclusively — no subprocess, CLI, or direct LangGraph runner calls from Streamlit.
• Synchronous, sequential execution only: the page blocks while tasks run and shows per-task progress via the service callback. Background jobs, cancellation, and parallel selected-task execution are deferred.
• Task catalog is loaded cheaply on demand: task_catalog.py exposes task_id, difficulty, question preview, and context file counts without loading heavy context data unless requested.
• Artifact compatibility is preserved: run artifacts (run_manifest.json, run_events.jsonl, trace.json, prediction.csv) are identical whether the run was started from the CLI or the launcher.
• Post-run navigation is wired: after a successful run, Open Run and Open Task Context actions update the existing session-state keys so the run immediately appears in all other Observatory tabs.
• Run Intelligence handoff is integrated: when a launcher run reaches terminal status, users can open Run Intelligence directly for that run from the launcher panel.
• Eval-v2 remains explicit: evaluation is not run implicitly; users can enable run-time evaluation in launcher workflow or run eval-v2 externally and then reload Run Intelligence.
• New files: task_catalog.py, run_launcher_builders.py, run_launcher_page.py (created); service_adapters.py, app.py (modified).

Phase 5 Status (2026-06)

• Phase 5 live execution observability is complete: the Run Launcher now includes a live execution panel driven by persisted run_events.jsonl and run_manifest.json.
• Execution path remains shared-service only: Streamlit continues to execute tasks via RunExecutionService.execute_selected_tasks() with both progress_callback and event_callback wired.
• Synchronous execution model is explicit: the UI clearly states that execution runs synchronously in the current Streamlit session; no background queue/server/cancellation/pause-resume was introduced.
• Live run visibility is additive: latest event card, current-task indicator, per-task status table, recent event tail, failure-event panel, and manual refresh for the latest run artifacts.
• Manifest correctness bug is fixed: final run_manifest.json now preserves terminal per-task statuses (completed/failed) instead of being overwritten by stale pending states.
• Artifact/event compatibility is preserved: existing run event schema and coarse event types remain unchanged; no breaking reader changes.
• No automatic eval-v2 trigger: launcher behavior still writes run artifacts only and requires explicit evaluation execution.

Phase 6 Status (2026-06)

• Phase 6 implements External Benchmark Dataset Intake (branch: feature/observatory-phase-06-custom-task-intake): the Run Launcher now has a second tab — “External Benchmark Dataset” — that accepts any local benchmark-compatible dataset root, validates it, and executes selected tasks through the same RunExecutionService service boundary. Free-form custom task intake is deferred to Phase 7.
• Validation is safety-first: the validator rejects remote URLs, system directories, home directory, and root; requires task.json with task_id and question; difficulty is optional and defaults to "unknown".
• Execution path is identical to approved config mode: the external root is applied as a temporary in-memory config override (dataclasses.replace()); source dataset is never mutated and artifacts are written to the configured output directory.
• No subprocess, CLI, or direct runner calls: the external tab calls RunExecutionService.execute_selected_tasks() exclusively, identical to the approved-config path.
• Duplicate Streamlit key collision is fixed: _render_live_execution_panel() now takes a mode parameter ("approved" / "external") so both tabs can render in the same session without key collision.
• Stale run state is cleared on new run start: _clear_launcher_run_state() zeroes prior last_result, live_events, live_manifest, last_run_output_dir, progress, and error before each run.
• New files: external_dataset_validator.py, test_external_dataset_validator.py (created); task_catalog.py, run_launcher_page.py, service_adapters.py, benchmark/dataset.py (modified).

Phase 16 Status (2026-06 — Complete)

• Phase 16 implements Live Trace Flow (branch: feature/observatory-phase-16-live-dag-replay): a new "⚡ Live Trace Flow" tab provides visual execution story built from trace-native step/action events with future human-in-the-loop pause/replan compatibility.
• Trace-native backbone: uses step_index + action as primary sequence; phase serves as badge/grouping only; agent names appear in separate overlay (not main DAG).
• Execution Story mapping: translates raw actions to presentation labels (e.g., planner → "Plan formulation") via comprehensive ACTION_STORY_MAP while preserving raw action data.
• Future HITL compatibility: all nodes include inert future fields (checkpoint_id, can_pause, can_resume, waiting_for_human, human_input_required, replan_candidate) defaulting to None/False.
• Read-only observability: strictly read-only; no pause/resume/execution control; no subprocess/CLI/runner calls from this page.
• Dual data source support: builders work with both live events (future) and final trace.json (current); no step event instrumentation added in Phase 16.
• Final Trace Handoff: provides buttons to open existing Reasoning DAG and Time-Travel Replay pages after task completion.
• UI sections: Current Execution Story, Visual Step DAG, Live Replay Timeline, Agent Reasoning Overlay, Raw Event Stream, Final Trace Handoff, Future HITL Note.
• Artifact integrity preserved: strictly read-only except safe auto-refresh; no mutations to trace.json, run_events.jsonl, or any execution/evaluation artifacts.
• Test coverage: 24 builder tests + 2 page tests passing (plus 13 skipped file content checks).
• New files: live_trace_flow_builders.py, live_trace_flow_page.py, test_live_trace_flow_builders.py, test_observatory_live_trace_flow_page.py, PHASE16_LIVE_TRACE_FLOW.md (created); app.py (modified: added tab 5).

Phase 16b Status (2026-06 — Complete)

• Phase 16b extends Live Trace Flow with Start Run + Live Task Selection with non-blocking execution: users can now start multi-task runs directly from the Live Trace Flow page and dynamically inspect any task during execution; UI returns immediately after starting run.
• Non-blocking execution model: new BackgroundRunController executes runs in background thread; UI returns immediately with run_id and run_dir; users can select and inspect tasks while execution continues.
• Compact Start Run panel: config selector (Configuration File :), attach-to-existing-run row (Attach to Existing Run (no selection will create a new run) :) with (Create new run) default, task multiselect (max 10), Start Run button; max-workers input removed (fixed at 1).
• Live task selector with status: task dropdown (Select Task to Inspect / View Live / Replay) shows live status indicators (▶️ running, ✅ completed, ❌ failed, ⏳ queued) derived from run_manifest.json, run_events.jsonl, and artifacts; auto-defaults to the running task when one exists (user override supported).
• Condensed run status line: replaces separate Run Status block; single st.info() line below submission message — Run Status: <status> | ▶️ Running: N | ✅ Completed: N | ❌ Failed: N | ⏳ Queued: N.
• Live / Replay pane (left, 50%): clickable step buttons; each button shows <Icon> <Status> Step N | <Story Label> | <Stage> - <Operation> | Duration - <Duration>; caption below each button shows <story_group> | Action - <Thought>; task start/end lifecycle rows bookend the step list.
• Step Output Inspector pane (right, 50%): fully click-driven — governed solely by button click in Live/Replay; no dropdown selector; inspector snapshot refreshes only at task start/end transitions (not every poll cycle); Thought: label renamed to Action:.
• Agent Reasoning Overlay placement: rendered above Status Commentary (after Step Output Inspector and before narrative scroll).
• Key findings untruncated: extract_key_values() in live_trace_flow_builders.py returns full values without the 220-char truncation cap.
• 500 ms polling: live fragment polls every 500 ms (refresh_interval = 0.5) instead of 1 s; live status caption shows auto-refreshing every 500ms.
• Auto-refresh always-on: removed user-facing auto-refresh toggle; refresh is fully system-driven based on run/task activity signals.
• Conditional rendering based on task state: queued tasks show waiting indicator; running tasks show live progress; completed tasks show full trace flow; failed tasks show failure details + partial trace.
• Duplicate execution prevention: BackgroundRunController prevents concurrent runs; guards against accidental re-execution on page refresh.
• Service architecture compliance: execution through BackgroundRunController → RunExecutionService with progress/event callbacks; no subprocess, CLI, or direct runner calls from Streamlit.
• Phase 16 features preserved: all read-only observability features (trace-native backbone, execution story, HITL schema) remain intact for completed tasks.
• Test coverage: 77 builder + page tests passing (25 skipped); test assertions updated to track column ratios, layout changes, and removed controls.
• New Phase 16b components: BackgroundRunController (application layer), 3 new builders (build_run_task_status_index, build_task_selector_options, derive_selected_task_state).
• Modified files: background_run_controller.py (NEW), live_trace_flow_builders.py (added builders, key-findings untruncated), live_trace_flow_page.py (non-blocking execution, full UI refinement), test_background_run_controller.py (NEW), test_live_trace_flow_builders.py, test_observatory_live_trace_flow_page.py.

Phase 17 Status (2026-06 — Complete)

• Phase 17 implements Guided Live Trace Flow (HITL) (branch: feature/observatory-phase-17-guided-trace-hitl): adds a separate 🧭 Guided Live Trace Flow (HITL) page/tab and keeps existing ⚡ Live Trace Flow unchanged.
• Start-panel behavior: the Guided page now uses a single Start Run action with a default-selected Guided Run (HITL) checkbox. When selected, the run uses planner-review HITL; when unselected, the same page launches the task flow without planner checkpoint pauses.
• Single checkpoint scope: guided runs support one checkpoint type only — planner_review — reached only after planner and before downstream execution/review actions.
• Guided policy at run start: HITL-enabled launches use execution_mode="guided", hitl_enabled=True, planner checkpoint policy metadata, and forced max_workers=1; unchecked launches use the same page but run without HITL checkpoint policy.
• timeout auto-approve policy: planner checkpoints use timeout_seconds=60 with default_action_on_timeout="continue"; if no response arrives in 60 seconds, the original plan is auto-approved by timeout policy and execution resumes.
• Service/runner-owned HITL: Streamlit remains UI-only; checkpoint creation/waiting, intervention persistence, delta persistence, and replan application are implemented in application/runner layers.
• Persistence layout: task-local artifacts are written under <run_dir>/<task_id>/checkpoints/*.json, interventions/*.json, and deltas/*_delta.json, while lifecycle events are emitted into run_events.jsonl.
• Decision paths implemented:
  • Approve: persists intervention, emits checkpoint_approved and task_resumed, continues unchanged.
  • Revise: persists intervention + steering instruction, emits intervention_submitted / state_invalidated / replan_started / replan_completed / task_resumed, stores revised plan and deterministic intervention delta.
  • Cancel: persists intervention, emits checkpoint_cancelled + task_cancelled, marks task as cancelled with reason cancelled_by_human (not failed).
  • Timeout auto-approve: emits checkpoint_timeout_started / checkpoint_timed_out / checkpoint_auto_approved / task_resumed, persists system intervention with decision="timeout_auto_approve" and submitted_by="system_timeout".
• Trace-level HITL evidence: final trace.json now contains additive hitl block with original plan, intervention, revised plan, delta metadata/path, and policy snapshot.
• Recovery behavior: browser refresh can redisplay pending checkpoints from persisted artifacts while process is alive; full process-death durable resume is not claimed.
• Phase separation clarity: Phase 17 is live in-execution HITL; Phase 8 stays advisory post-hoc; Phase 9 stays isolated guided rerun workflow.
• New files: domain/intervention_models.py, application/hitl_checkpoint_controller.py, observatory/guided_trace_flow_builders.py, observatory/guided_trace_flow_page.py, and new tests for controller, graph checkpoint behavior, guided builders, and guided page.

Phase 7 Status (2026-06 — Complete)

• Phase 7 implements Free-form Custom Task Intake (branch: feature/observatory-phase-07-freeform-custom-task-intake): a third tab — "Custom Task" — is added to the Run Launcher. Users enter a question, optional context, optional uploaded files, and expected output type. The task is persisted, materialized into a benchmark-compatible synthetic dataset, and executed through the existing shared RunExecutionService. No new runner is created.
• Safe file handling: uploaded files are sanitized for path traversal, size-limited to 50 MB, extension-blocked for unsafe types (.py, .sh, etc.), and hashed with SHA256. Files are never executed — they are context/data only.
• Materialization structure: custom tasks materialize into artifacts/custom_tasks/<task_id>/ with task_definition.json (provenance), context/ (always created, knowledge.md optional), and a benchmark-compatible dataset/<task_id>/ folder. The context/ directory always exists; knowledge.md is created only when context text is provided.
• Execution is identical to approved-config and external-dataset modes: custom tasks use get_run_execution_service_with_custom_task() which applies the synthetic dataset as an in-memory config override. No subprocess, CLI, or direct runner invocation.
• No eval-v2 correctness scoring: custom tasks have has_gold_data=False by default; eval-v2 scoring is unavailable unless gold data is added later. Trace, prediction, live execution, and artifact inspection remain available.
• Domain models: TaskDefinition, ExpectedOutputSpec, SourceDescriptor, UploadedFileSpec in task_models.py support JSON serialization and optional context/files.
• Tests: 80 Phase 7 unit and integration tests cover domain models, repository file handling, service logic, materialization, provenance, and end-to-end workflows. All existing Phase 0–6 tests remain passing.
• New files: domain/task_models.py, repositories/custom_task_repository.py, repositories/filesystem_custom_task_repository.py, application/custom_task_service.py, tests/unit/domain/test_task_models.py, tests/unit/repositories/test_filesystem_custom_task_repository.py, tests/unit/application/test_custom_task_service.py, tests/integration/test_phase7_custom_task_launcher_integration.py (created); observatory/service_adapters.py, docs/implementation/DECISIONS.md (DEC-011), docs/implementation/IMPLEMENTATION_LEDGER.md (modified).

Phase 8 Status (2026-06 — Complete)

• Phase 8 implements Checkpoint Review and Advisory Human Steering (branch: feature/observatory-phase-08-checkpoint-steering) as a conservative additive capability.
• Checkpoint timeline/detail UI added: task-level "Checkpoint Review" tab derives checkpoints from existing trace.json and optional run_events.jsonl without altering execution history.
• Advisory-only persistence model: users can save human review notes and advisory steering instructions to task-local checkpoint_annotations.json.
• No execution intervention introduced: no pause/resume, no mid-run steering, no runner state mutation, no execution trigger from this page.
• Artifact immutability preserved: trace.json, prediction.csv, task_definition.json, and eval-v2 artifacts remain unchanged.
• Compatibility preserved: approved benchmark, external dataset, and custom-task runs all use the same checkpoint page behavior.
• Validation status: new Phase 8 tests pass and full repository suite passes (397 passed).

Phase 9 Status (2026-06 — Complete)

• Phase 9 implements Guided Rerun Planning and Execution (branch: feature/observatory-phase-09-guided-rerun-planning): converts Phase 8 advisory steering instructions into executable rerun plans with approval workflow and isolated execution.
• Rerun plan workflow: users select steering instructions, define objectives/constraints, generate plans (draft status), approve plans, execute as new isolated custom task runs through Phase 7 path.
• Source artifact immutability preserved: original run artifacts remain unchanged; execution creates new timestamped runs with guidance context injected into synthetic task datasets.
• Question resolution with dataset fallback: resolve_source_task_question() finds original questions from task definitions, trace files, or dataset input directories when not in run artifacts.
• Context file copying: guided reruns materialize with original task's data files (CSV, JSON, DB) copied into custom task context so the agent has access to all source data.
• Safe execution path: uses Phase 7 CustomTaskService to materialize synthetic datasets and RunExecutionService for execution; no subprocess, CLI, or direct runner invocation.
• Artifact schema: rerun_plans.json per task with plan metadata, provenance (source run/task/checkpoints/instructions), status lifecycle (draft→approved→executed), execution metadata (run_id, task_id, timestamps).
• Test coverage: 52 Phase 9 tests (10 models, 21 builders, 6 execution, 15 page tests); full suite passes (449 passed).
• New files: rerun_plan_models.py, rerun_plan_builders.py, rerun_plan_execution.py, rerun_plan_page.py (created); app.py (modified: added 7th tab).

Phase 10 Status (2026-06 — Complete)

• Phase 10 implements Complete Original-vs-Guided Rerun Comparison and Local Ask This Run / Ask This Comparison (branch: feature/observatory-phase-10-rerun-comparison-ask-this-run): provides read-only structured comparison of original vs guided runs with interactive artifact-grounded Q&A.
• Comparison dimensions: prediction/output (byte-identical check, row/column delta), trace/trajectory (step count, stage distribution), tool usage (call delta, unique tools), runtime (execution time delta, status), confidence/failure/provenance/critic (with graceful unavailable states), eval-v2 metrics (score delta, precision/recall with graceful unavailable).
• Steering influence summary: heuristic keyword matching with non-causal language ("possible alignment detected", "evidence suggests", never "caused" or "fixed"); clearly marked as heuristic, not causal proof.
• Ask This Run / Ask This Comparison: local rules-based Q&A with 20+ supported intents (FINAL_ANSWER_CHANGED, STEP_COUNT_CHANGED, TOOL_USAGE_CHANGED, RUNTIME_CHANGED, FAILURE_CHANGED, STEERING_USED, STEERING_REFLECTED, EVAL_AVAILABLE, SCORE_IMPROVED, WHAT_CHANGED, INSPECT_NEXT, etc.); unsupported questions return exact fallback text: "I cannot answer that from the available local artifacts."
• Saved comparison reports: users can save computed comparison reports to additive rerun_comparisons.json artifact; previously saved reports displayed in table.
• Artifact integrity preserved: strictly read-only comparison via pure builder functions; no execution, no mutations (except additive rerun_comparisons.json when user clicks Save), no subprocess, no external LLM/API calls; hash-based tests verify artifact integrity.
• Graceful degradation: comparison handles missing artifacts (trace, prediction, eval) with clear messaging; eval-v2 artifacts optional.
• Artifact schema: rerun_comparisons.json per task with schema_version, reports array containing full comparison metadata (source lineage, artifact availability, all comparison dimensions, steering summary).
• Test coverage: 70+ Phase 10 tests across models, builders, Ask Q&A, UI, and integration; artifact integrity verified with hash-based mutation tests.
• New files: rerun_comparison_models.py, rerun_comparison_builders.py, ask_artifact_builders.py, rerun_comparison_page.py, 5 test files (created); app.py (modified: added 8th tab after Guided Rerun Plan).

Phase 11 Status (2026-06 — Complete)

• Phase 11 implements Run-Level Cohort Guided Rerun Evaluation (branch: feature/observatory-phase-11-cohort-guided-evaluation): aggregates existing guided rerun comparisons across a run into cohort-level metrics and provides local artifact-grounded cohort Q&A.
• Cohort discovery: scans task directories in source run, finds executed rerun plans from rerun_plans.json, resolves guided artifacts using Phase 10 pairing logic, supports multiple executed plans per source task, graceful degradation for missing artifacts.
• Aggregated metrics: prediction changed count/rate, score delta stats (mean/median/min/max when eval exists, improved/degraded/neutral classification), step/tool/runtime delta stats, failure rate comparison (source vs guided, both succeeded/failed, source-only/guided-only failed), eval artifact availability, missing artifact counts, steering instruction coverage (unique instruction count, reuse count, possible alignment count/rate with heuristic language).
• Integration: extends existing Run Intelligence page (Section 9, not a new tab); renders only when executed guided reruns detected; conditional visibility based on view level (Summary/Detailed/Research); compact info in expander when no reruns exist.
• UI components: cohort overview cards, source→guided pairing table, eval unavailable notice, steering coverage table (Detailed/Research), missing artifacts panel, representative examples (Research), Ask This Cohort text input, Save Cohort Report button.
• Ask This Cohort: local rules-based Q&A with 12+ cohort intents (cohort_size, prediction_changed_count, improved_count, degraded_count, mean_score_delta, runtime_overhead, failure_rate_change, most_common_steering, eval_coverage, missing_artifacts, task_changed_most); unsupported questions use same safe fallback as Phase 10.
• Non-causal language: all summaries and answers avoid causal claims; disclaimer: "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."; steering alignment marked as "heuristic signal", "possible alignment", "observed association" (never "caused", "fixed", "proved").
• Artifact integrity preserved: strictly read-only aggregation; no execution, no eval-v2 auto-run, no mutations except optional cohort_guided_comparison.json (regenerable snapshot, overwrite-latest); hash-based tests verify no mutations to trace.json, prediction.csv, rerun_plans.json, rerun_comparisons.json, checkpoint_annotations.json, run_manifest.json, run_events.jsonl, eval-v2 artifacts.
• Graceful degradation: score delta stats exclude missing eval artifacts (not counted as zero), runtime requires runtime_seconds in trace.json (optional field), missing comparisons degrade gracefully with clear limitations.
• Reuse of Phase 10 assets: uses Phase 10 find_executed_rerun_plans(), resolve_comparison_pair(), build_artifact_availability(), compare_predictions(), load_trace_summary() to build individual pairings; aggregates via pure cohort builders.
• Artifact schema: optional cohort_guided_comparison.json per run with schema_version, run_id, created_at, summary (all cohort metrics), pairing array (list of PairingEntry), steering_coverage, disclaimer.
• Test coverage: 69 Phase 11 tests (11 models, 17 builders, 30 Ask cohort, 11 integration); Phase 10 regression verified (99 passed); artifact integrity verified with SHA256 hash comparison before/after; no new folders/events verified.
• New files: cohort_guided_models.py, cohort_guided_builders.py, 4 test files (created); run_intelligence_page.py, ask_artifact_builders.py (modified: extended with cohort support).

Phase 12 Status (2026-06 — Complete)

• Phase 12 implements Evidence Pack and Statistical Readiness (branch: feature/observatory-phase-12-evidence-pack-and-statistical-readiness): consolidates Phases 8–11 into a judge-facing Evidence Pack with cross-phase narrative, evidence hierarchy, statistical readiness assessment, claim safety framework, and exportable markdown/JSON reports.
• Cross-phase narrative: structured timeline showing Phase 8 Checkpoint Review → Phase 9 Guided Rerun Planning → Phase 10 Original-vs-Guided Comparison → Phase 11 Run-Level Cohort Evaluation with key outcomes and artifacts created at each phase.
• Evidence hierarchy: classifies evidence by level (primary: trace.json, prediction.csv, task_metrics.csv, trajectory.csv, tool_calls.csv, run_manifest.json, run_events.jsonl; derived: checkpoint_annotations.json, rerun_plans.json; comparison: rerun_comparisons.json; aggregate: cohort_guided_comparison.json, evidence_pack.json, evidence_pack.md).
• Statistical readiness assessment: conservative N≥20 threshold and eval coverage ≥80% requirement for significance testing; defaults to descriptive_only for smaller N; provides explicit limitations and warnings; recommended test: wilcoxon_signed_rank when sufficient, descriptive_only otherwise.
• Claim safety framework: explicit lists of allowed observational claims vs unsupported causal claims; required disclaimers: "Observed associations do not establish causation", "Guided reruns differ in timing, model state, and execution context", "Results may not generalize"; confidence level: observational or descriptive.
• Export: markdown and JSON export on explicit button click (evidence_pack.md, evidence_pack.json); exports are regenerable and overwrite-latest; markdown includes all required sections (Executive Summary, Timeline, Evidence Hierarchy, Statistical Readiness Assessment, Claim Safety Report, Key Findings, Representative Examples, Limitations, Disclaimers, Artifact Manifest).
• Integration: extends Run Intelligence page (Section 10, not a new tab); visible in Detailed and Research view levels; statistical readiness card, claim safety summary, key findings preview, export buttons, optional markdown preview in Research mode.
• Graceful degradation: handles missing guided reruns, missing eval artifacts, missing cohort reports with clear limitations; evidence hierarchy marks artifact availability; statistical readiness flags insufficient sample size.
• Artifact integrity preserved: strictly read-only except for explicit export; no execution, no eval-v2 auto-run, no mutations to source artifacts (trace.json, prediction.csv, checkpoint_annotations.json, rerun_plans.json, rerun_comparisons.json, cohort_guided_comparison.json, run_manifest.json, run_events.jsonl); only evidence_pack.json and evidence_pack.md created on export.
• Non-causal language: markdown rendering includes causation disclaimers; unsupported claims section lists causal language variants; allowed claims use observational language only ("observed", "associated with", "followed by", never "caused", "fixed", "proved").
• Test coverage: 72 Phase 12 tests (25 models, 22 builders, 17 exporters, 8 integration); all Phase 8-11 regression verified; artifact integrity verified with SHA256 hash comparison; no new folders/events verified; safety scan confirms no subprocess, no external API calls.
• New files: evidence_pack_models.py, evidence_pack_builders.py, evidence_pack_exporters.py, 4 test files (created); run_intelligence_page.py, DECISIONS.md, IMPLEMENTATION_LEDGER.md, Overview.md (modified).

---

2. Pipeline Architecture

graph TD
    START([Start])
    explore["Phase 0: Explore Data<br/><i>Parallel file I/O, schema inspection</i>"]
    semantic["Phase 0.5: Semantic Extraction<br/><i>knowledge.md parsing, doc/*.md extraction (profile-only for large docs)</i>"]
    plan["Phase 1: Planner<br/><i>LLM generates ExecutionPlan JSON</i>"]
    critic_plan{"Critic: Plan Valid?<br/><i>LLM validates plan against data</i>"}
    execute["Phase 2: Executor<br/><i>LLM generates Python code, sandboxed exec</i>"]
    error_needs_replan{"Structural Error?<br/><i>_error_needs_replan heuristic</i>"}
    critic_execute{"Critic: Result Valid?<br/><i>LLM validates output shape + semantics</i>"}
    END([End])

    doc_phase2["Phase 1.5: Doc Phase 2<br/><i>Planner-gated targeted re-extraction</i>"]

    START --> explore
    explore --> semantic
    semantic --> plan
    plan --> critic_plan
    critic_plan -->|valid| doc_phase2
    doc_phase2 --> execute
    critic_plan -->|"retry_plan (max 2)"| plan
    execute -->|success| critic_execute
    execute -->|"code failed"| error_needs_replan
    error_needs_replan -->|"yes: KeyError, no such table, etc."| plan
    error_needs_replan -->|"no: transient error"| execute
    error_needs_replan -->|"retries exhausted"| critic_execute
    critic_execute -->|retry_exec| execute
    critic_execute -->|replan| plan
    critic_execute --->|"✅ Happy Path (confidence ≥ 0.7)"| END
    critic_execute --->|"❌ Failure / retry limit"| END

    style START fill:#e1f5fe
    style END fill:#c8e6c9
    style explore fill:#f3e5f5
    style semantic fill:#fce4ec
    style doc_phase2 fill:#fce4ec
    style plan fill:#e8eaf6
    style execute fill:#e8eaf6
    style error_needs_replan fill:#ffccbc
    style critic_plan fill:#fff9c4
    style critic_execute fill:#fff9c4

Total execution budget: 5 attempts across execute + replan cycles (max_total_attempts = 5).

---

3. Phase-by-Phase Detail

Phase 0: Explore Data (explore_data_node)

What: Reads all files in the context directory in parallel using ThreadPoolExecutor(max_workers=8).

Why: The planner and executor need to see schemas, sample rows, and data types before they can write correct queries. Parallel I/O cuts exploration time for multi-file tasks.

File type	Action
.csv	Read columns with dtypes, row count, 3 sample rows. Auto-build into _auto.db SQLite.
.json	Preview first 10K chars.
.db/.sqlite	PRAGMA table_info, CREATE TABLE SQL, 3 sample rows per table.
.md (knowledge)	Full content included in data_exploration.
.md (doc/*)	Truncated to 2,000 chars in data_exploration (full content extracted in Phase 0.5).

Why truncate doc/*.md? These narrative files can be 85K–286K chars. Including them raw in the executor prompt causes Azure API 500 errors from token overflow. Since they're extracted into _auto.db tables in Phase 0.5, the executor only needs a short preview.

Phase 0.5: Semantic Extraction (semantic_extraction_node)

What: Two sub-tasks:

1. knowledge.md parsing (rule-based, no LLM): Extracts entities, fields, relationships, synonyms, business rules, value maps, threshold rules. Produces a SemanticContext object.

2. doc/*.md extraction (LLM-based, chunked batching): Converts narrative prose files into structured tables loaded into _auto.db.

Doc Extraction Pipeline

doc/*.md content
    │
    ▼
Size check: len(content) >= 40KB?
    │
    ├── YES (large doc) ────────► Two-Phase Path (doc_phases.py)
    │                              │
    │                              ▼
    │                         Phase 1: Profile + Sample
    │                           _doc_structure_probe() (1 LLM call)
    │                           _extract_sample_records() (1 LLM call, first 4K chars)
    │                           → 5 sample rows into _auto.db
    │                           → profile stored in state.doc_profiles
    │                              │
    │                              ▼
    │                         [Planner runs with sample data]
    │                              │
    │                              ▼
    │                         Phase 2: Planner-Gated Full Extraction
    │                           _columns_needed_by_plan() — stem+exact match
    │                           _get_doc_chunks_prioritized() — split by ## or ¶
    │                           _pack_batches(chunks, 12KB)
    │                           N LLM calls (up to 20 batches)
    │                           Merge by entity ID (first-non-null)
    │                           → Replace sample data in _auto.db
    │
    └── NO (small doc) ─────────► Eager Path (semantic.py)
                                   │
                                   ▼
                              _parse_md_sections()
                                   │
                                   ▼
                              _generate_keywords()
                                   │
                                   ▼
                              _filter_sections_by_keywords()
                                (only for docs > 80K, only if > 30% reduction)
                                   │
                                   ▼
                              _chunk_section_by_entities()
                                   │
                                   ▼
                              Structure Probe (1 LLM call)
                                   │
                                   ├── flat path → Batch Extraction (N calls, ~20K/batch)
                                   └── multi-section → _extract_multisection_doc()
                                   │
                                   ▼
                              Quality Gate + _focused_reextraction()
                                   │
                                   ▼
                              load_doc_records_to_sqlite() → _auto.db

Why chunked extraction? A single LLM call on a 286K char Laboratory.md misses ~50% of records. Batching at ~20K chars gives the LLM manageable context windows.

Why a single Structure Probe (replaces standalone Schema Detection)? The old pipeline made one LLM call to detect the schema, then assumed a single flat record stream. Docs like task_418 interleave multiple per-entity sections (one per patient, lab visit, etc.) and need merging by an entity_id key. The probe folds schema detection + layout classification + merge-key inference + record-count estimation into one call, sampling head / middle / tail (_PROBE_SAMPLE_CHARS=3000 each, more for larger docs via _PROBE_EXTRA_SAMPLE_PER_BYTES). Downstream code branches on is_multisection.

Why multi-section first-non-null merge? When the same entity has fragments scattered across sections (e.g., demographics in §1, labs in §3, diagnosis in §5), per-section extraction yields partial rows. Merging by merge_key and taking the first non-null per column reconstructs the full record without overwriting good values with later nulls.

Why post-extraction quality gates + focused re-extraction? Even with good chunking, some columns come out sparsely pectiulated (LLM skipped them) or the total record count falls well short of the probe's estimate. _coverage_stats computes per-column null rates and total count; _quality_gate_targets picks columns above _QG_NULL_RATE (0.5) or flags low coverage (< _QG_COVERAGE_RATIO=0.7 of estimate). _focused_reextraction then makes a single targeted retry pass aimed at those columns / missing records. Gating is skipped for tiny extractions (_QG_MIN_RECORDS_FOR_GATE=5) to avoid noise.

Why markdown-aware chunking? The doc files use #/##/###/#### headings to organize different data domains (e.g., "Liver Function Panel", "Immunoglobulin Levels", "Coagulation Parameters"). Parsing by heading level:

• Produces cleaner entity-level chunks (no split mid-record)
• Enables keyword-based section filtering for very large docs
• Respects the document's own structure

Why keyword filtering only for large docs? For extraction, we generally need ALL records (SQL may aggregate the full table). Filtering is only a fallback for docs > 80K chars where processing all sections would exceed the batch budget.

Why the * separator handling? Many doc files use a standalone * as a section separator (not a heading). _chunk_section_by_entities() splits on \n\*\s*\n to handle this.

Question Rewriting

After semantic extraction, the question is rewritten using rewrite_query() to:

• Resolve ambiguous column names (e.g., "views" → ViewCount)
• Map natural language to schema terms
• Include data_exploration context so the LLM knows what columns exist

Why? Task_259 failed because "views" was mapped to Score instead of ViewCount. The rewriter uses schema context to make correct mappings.

Phase 1: Planner (planner_node)

What: LLM generates a structured ExecutionPlan JSON with steps, required sources, output columns, expected row count, and reasoning.

Key prompt rules (from PLANNER_SYSTEM):

• "posted it last time" → use LastEditorUserId not OwnerUserId (task_257 fix)
• "How many times more X than Y" → compute ratio X/Y, not a count (task_352 fix)
• ROW COUNT ESTIMATION: plan must declare "single" or "multiple" expected rows
• COMPOUND FILTER LOGIC: multiple conditions must be applied simultaneously, not sequentially (task_355 fix)
• Output column discipline: plan declares exact output columns

Phase 1.5: Plan Critic (critic_plan_node)

What: LLM validates the plan against the data exploration context. Checks column existence, join feasibility, filter logic.

Why a separate critic? The planner hallucinates column names ~15% of the time. A separate validation pass catches these before wasting an execution attempt.

Phase 2: Executor (executor_node)

What: LLM generates Python code (pandas + sqlite3) that implements the plan. Code is executed in a sandboxed exec() with a restricted global namespace.

Key features:

• SQL-first strategy: prefer _auto.db SQL over raw file reads
• Code must produce a result DataFrame variable
• Captured stdout from print() statements is included in critic context

Phase 2.5: Execution Critic (critic_execute_node)

What: LLM validates the execution result via the 12-item forced checklist (Change B) plus an independent sanity check #13 (Change F). After the LLM critic returns, deterministic post-checks run:

• Output column count vs expected (auto-reject if > max(3, 1.5× expected))
• Row count vs expected ("single" or "multiple") — soft warning only, not auto-reject
• Numeric Guards (Change E) — NaN/inf, percent out of [0,100], negative count, ratio-as-int-zero. Flags are always surfaced in the trace; a fired guard downgrades a previously-valid result so the executor retry loop fires once.

Why soft row count warning? task_355 was auto-rejected because the plan said "single" but the result had 1 row — the auto-reject logic was too strict. Changed to a log warning that the critic LLM can consider.

Why structural numeric guards instead of more prompt rules? See §10 — prompt-level fixes saturated around λ=0.665, and any further rule specific enough to fix one failure was overfitting to specific failing tasks.

Error Triage (_error_needs_replan)

What: Heuristic that classifies execution errors as structural (needs replan) vs transient (retry same code).

Error pattern	Action
KeyError, no such table, FileNotFoundError	→ Replan
OperationalError, empty DataFrame	→ Replan
Other errors, < 2 attempts	→ Retry execution
Any error, ≥ 2 attempts	→ Force replan

---

4. Key Constants

Timeouts & Parallelism (main.py)

Constant	Value	Why
TASK_TIMEOUT_SECONDS	510	Per-task SIGALRM timeout. Increased from 300 to accommodate large doc extraction tasks.
MAX_WORKERS	4	ThreadPoolExecutor workers for parallel task execution. Reduced from 8 to avoid API saturation causing timeouts (cross-validation adds 1 extra LLM call per easy/medium task).

Retry Budgets

Constant	Location	Value	Why
max_plan_retries	state.py	2	Max planner attempts before proceeding with last plan.
max_exec_retries	state.py	3	Max executor attempts per plan.
max_total_attempts	graph.py	5	Total budget across execute + replan cycles.
_MAX_RETRIES	model.py	3	API call retries for transient HTTP errors.
_RETRY_BASE_DELAY	model.py	2.0s	Exponential backoff base (2s, 4s, 8s).
_RETRY_ERRORS	model.py	(500, 502, 503, 504, 429)	HTTP status codes that trigger retry.

Data Exploration (nodes.py)

Constant	Value	Why
DEFAULT_SAMPLE_ROWS	3	Sample rows shown per table in data_exploration. Keeps prompt small.
Doc truncation limit	2,000 chars	Max doc/*.md content in data_exploration. Full content is in _auto.db.

Doc Extraction (semantic.py)

Constant	Value	Why
max_chars_per_batch	20,000	Chars per extraction batch. Balances accuracy vs LLM context limits.
max_batches (default)	8	Default max batches per doc file.
per_doc_max_batches	max(3, 16 // num_doc_files)	Dynamic budget: 1 file→16, 2 files→8, 3+→5. Ensures total LLM calls stay bounded.
LARGE_DOC_THRESHOLD	80,000 chars	Docs larger than this may have keyword-based section filtering applied.
Keyword filter threshold	70%	Only use filtered sections if they're < 70% of original size (i.e., > 30% reduction).
Schema detection sample	3 chunks, 6000 chars	Sample text sent for legacy schema detection LLM call (now superseded by Structure Probe).
_PROBE_SAMPLE_CHARS	3,000	Chars per head/middle/tail probe window.
_PROBE_MIN_SAMPLES	3	Minimum probe samples (head + middle + tail).
_PROBE_EXTRA_SAMPLE_PER_BYTES	30,000	Add one extra probe sample per ~30K chars of doc length.
_QG_COVERAGE_RATIO	0.7	If actual_records < estimate × 0.7, trigger focused re-extraction.
_QG_NULL_RATE	0.5	If a column's null-rate ≥ 50%, target it for focused re-extraction.
_QG_MIN_RECORDS_FOR_GATE	5	Skip quality gating entirely below this many records.

Deadlines & HTTP Timeout

Constant	Location	Value	Why
LLM_HTTP_TIMEOUT	agents/model.py	90s (env LLM_HTTP_TIMEOUT)	Per-request HTTP timeout on both OpenAI(...) and AzureOpenAI(...) clients. Default SDK timeout is effectively unbounded, which caused stalled doc-extraction calls to consume the full task budget.
DOC_EXTRACTION_BUDGET_SECONDS	env (parsed in nodes.py)	unset (off)	Optional soft deadline for the doc-extraction phase. When set, semantic_extraction_node computes a monotonic deadline and passes it as deadline_monotonic into extract_doc_records_chunked, which checks it at each safe abort point (between batches, between sections, before focused retry).

Output Validation

Constant	Location	Value	Why
Column count cap	nodes.py	max(3, 1.5 × expected)	Auto-reject results with too many columns (prevents data dumps).
Row count check	nodes.py	Soft warning	Log warning if row count mismatches plan's "single"/"multiple" estimate.

LLM Call Tracking

Each step with token_usage increments the phase's llm_calls counter. Trace includes:

• Per-phase LLM calls: explore_llm_calls, planner_llm_calls, critic_plan_llm_calls, execute_llm_calls, critic_execute_llm_calls
• Total LLM calls: total_llm_calls (sum across all phases)
• Stored in stage_metrics[phase]["llm_calls"] and comprehensive_metrics["llm_calls"]

---

5. LLM Prompts

Prompt	File	Purpose
PLANNER_SYSTEM/USER	prompts.py	Generate ExecutionPlan JSON
CRITIC_PLAN_SYSTEM/USER	prompts.py	Validate plan against data
EXECUTOR_SYSTEM/USER	prompts.py	Generate executable Python code
CRITIC_EXECUTE_SYSTEM/USER	prompts.py	Validate execution result
QUERY_REWRITE_SYSTEM	semantic.py	Rewrite question with schema context
DOC_SCHEMA_DETECTION_SYSTEM/USER	semantic.py	Legacy schema detection (kept as fallback)
DOC_STRUCTURE_PROBE_SYSTEM/USER	semantic.py	Combined schema + multi-section layout + merge-key probe (single call)
DOC_BATCH_EXTRACTION_SYSTEM/USER	semantic.py	Extract records from doc chunk batch
DOC_EXTRACTION_SYSTEM/USER	semantic.py	Legacy single-call extraction (fallback)

---

6. File Map

main.py                          Entry point, task discovery, parallel execution, SIGALRM timeout
src/data_agent_baseline/
├── agents/
│   └── model.py                 OpenAI/Azure API adapters with retry logic
├── langgraph_agent/
│   ├── graph.py                 Workflow orchestration, retry/replan logic
│   ├── state.py                 AgentState, ExecutionPlan, CriticFeedback dataclasses
│   ├── nodes.py                 All graph nodes (explore, semantic, plan, execute, critics)
│   ├── prompts.py               All LLM prompt templates
│   ├── semantic.py              knowledge.md parsing, doc structure probe + chunked/multi-section extraction, quality gates, query rewriting
│   ├── normalization.py         Output DataFrame normalization
│   ├── numeric_guards.py        Change E (M3) — deterministic NaN/percent/count/ratio sanity checks (zero LLM cost)
│   ├── cross_validate.py        Change G — dual-track re-derivation (SQL↔pandas) for easy/medium tasks + Option B query_db injection
│   ├── failure_tagger.py        Classify per-task failures into buckets + aggregate per-phase timing (used by `tag-failures` CLI)
│   └── runner.py                Trace JSON + prediction CSV writing
configs/
└── react_baseline.container.yaml  Runtime configuration

---

7. Bug Fixes & Hardening (Chronological)

#	Task/Issue	Root Cause	Fix
1	task_257	"posted it last time" → wrong user column	Added LastEditorUserId rule to planner + critic prompts
2	Column explosion	Extra columns in output → wrong CSV	Auto-reject if columns > max(3, 1.5× expected)
3	task_259	"views" → Score instead of ViewCount	Query rewriter with schema context in semantic.py
4	task_352	"how many times more" → count instead of ratio	Ratio idiom rule in planner + rewriter prompts
5	Timeouts	Tasks timing out at 300s	SIGALRM timeout, parallel execution, difficulty sorting
6	task_355	Compound filters applied sequentially	Expected row count, compound filter rules in prompts
7	Row count auto-reject	Valid single-row results rejected	Changed to soft warning
8	Doc extraction quality	Single LLM call misses 50%+ records	Chunked batch extraction pipeline
9	task_418 Azure 500	Prompt too large (286K doc in data_exploration)	Truncate doc/*.md to 2K chars in data_exploration
10	API transient failures	No retry on 500/429 errors	Exponential backoff retry in model.py
11	Doc chunking quality	Flat paragraph splitting ignores markdown structure	Markdown-aware section parsing + keyword filtering
12	task_418 (multi-section docs)	Schema-detection assumed one flat record stream; per-entity sections produced partial rows	Structure Probe + multi-section path with first-non-null merge by merge_key
13	task_396 (sparse columns / undercount)	Single extraction pass left some columns mostly null and fell short of probe's record estimate	Post-extraction quality gates (_QG_COVERAGE_RATIO, _QG_NULL_RATE) + one focused re-extraction pass
14	Stalled LLM HTTP calls	OpenAI/Azure SDK default timeout effectively unbounded → single hung request burned entire task budget	LLM_HTTP_TIMEOUT=90s applied to both clients; optional DOC_EXTRACTION_BUDGET_SECONDS deadline threaded into extract_doc_records_chunked
15	prediction.csv schema drift	Old _raw columns emitted alongside normalized columns confused downstream evaluation	Commented out _raw block in runner._save_result; CSV now contains only normalized columns
16	Eval visibility	Hard to tell which buckets / difficulties / phases were costing recall	New eval-lang columns (Time, recall>0/=0, difficulty breakdown, zero-recall list) + new tag-failures CLI (per-task bucket + per-phase timing) backed by failure_tagger.py
17	Prompt rule attention	Bug-patch rules (LastEditorUserId, ratio idiom, naming discipline, scoring rule) were buried mid-prompt and routinely ignored by the LLM	Change A — added SHARED_HEADER constant prepended to all four SYSTEM prompts (scoring rule, ground-truth hierarchy, answer-shape vocab, naming discipline) so highest-leverage rules land in the top-of-context-window high-attention slot
18	Critic rubber-stamping	Free-form CRITIC_EXECUTE_SYSTEM produced inconsistent verdicts and missed recurring failure modes (name-vs-ID, name concatenation, null-filtering, count-vs-list)	Change B — replaced with a forced 12-item checklist (required_columns_present, count_vs_list, name_not_id, name_columns_separate, no_null_zero_filter, no_head_or_limit, ratio_not_count, monthly_vs_annual, value_encoding_correct, ordering_explicit, magnitude_reasonable, not_empty) with deterministic verdict rules. Legacy prompt preserved as CRITIC_EXECUTE_SYSTEM_LEGACY. Change C — CRITIC_EXECUTE_USER instructs the critic to walk the checklist in order. Legacy preserved as CRITIC_EXECUTE_USER_LEGACY.
18a	Change A regressions	Two SHARED_HEADER lines over-fired: "When in doubt INCLUDE IT" inflated output_columns and broke DISTINCT dedup (task_194); "what is the X of Y" → single truncated legitimate multi-row answers (task_22)	Fix A1 — removed the unconditional INCLUDE-IT line; bias toward minimal output_columns retained via the asymmetric scoring text. Fix A2 — replaced the single-only cue with "CHECK THE DATA — default to multiple unless explicitly uniquely-keyed".
18b	Change B over-rejection	Check #2 count_vs_list hard-failed any "what is the X of Y" question whose result had >1 row, again breaking task_22	Fix B1 — narrowed hard-fail scope to four explicit aggregation idioms ("how many", "what percentage", "what is the average/mean/median/ratio", "how many times more/greater"); everything else degrades to a confidence-only warning.
19	Saturation of pure-prompt fixes	After A/B/C + A1/A2/B1 the system stabilised around λ=0.665 (~60% perfect). The residual value-mismatch tail (NaN denominators, ratio-as-int-zero, percentages > 100, wrong-entity-kind results) could not be addressed by more prompt rules without overfitting to specific failing tasks — earlier Change D attempts (D1/D2/D4/D5) cited or strongly implied known task IDs and were rejected.	Change E (Mechanism 3) — Numeric Guards + Change F (Mechanism 2) — Independent Sanity Check + Change G — Cross-Validation. All are intentionally generic (no question-pattern catalogues, no task references). See §10.
20	Executor SQL errors (column case, apostrophes)	Generated code uses pd.read_sql_query directly — bypasses execute_read_only_sql error hints. LLM hallucinates column case (SEX vs Sex) or fails to escape quotes (Women's Soccer). Cross-validation exec namespace missing sqlite3/gzip imports.	Option B — query_db injectable helper with auto-fix for quote escaping + case-insensitive column resolution. Pre-injected into executor + cross-validation exec_namespace. See §10.
21	task_194 extra columns regression	Executor includes join-key columns (atom_id, atom_id2) alongside requested bond_id. Post-hoc guards didn't prune because "never removes columns" was a constraint.	Guard 4 (column pruning) in post_hoc_guards.py — drops extras when ALL expected columns are confirmed present.
22	task_173 empty-answer regression	Strategy 4 + Strategy 8 (safety wrapper) converted crashes to silent answer="" single-row results. Critic's not_empty check didn't fire because the DataFrame technically had 1 row.	Guard 5 (effectively-empty detection) — converts single-row all-blank results to truly empty DataFrame → triggers replan.
23	task_283 percentage calculation (31.6 vs 31.2)	Executor used INNER JOIN for percentage denominator, shrinking the base population.	Added LEFT JOIN for percentages rule + DEDUPLICATION rule to EXECUTOR_SYSTEM prompt.

---

10. Structural Verification Mechanisms (changes E / F / G)

After §9's prompt layer saturated, two structural mechanisms were added downstream of critic_execute_node. Both are independently reversible via marker blocks and both default to ON; only Change E has an env gate (ENABLE_NUMERIC_GUARDS).

Change E — Mechanism 3: Deterministic Numeric Guards (numeric_guards.py)

A new module of pure-Python, zero-LLM-cost sanity checks. Each guard is a function of (question_text, result_df) with no task-specific knobs. The four guards:

Guard	Trigger	Rationale
nan_or_inf	Any numeric column contains NaN / ±inf	Unguarded division (divide-by-zero) or unfiltered NULL — never a legitimate final answer
percentage_out_of_range	Question matches _PERCENT_PAT AND any numeric column has value outside [-0.001, 100.001]	Percentage answer outside [0, 100] is impossible. [0, 1] fractional convention is allowed (won't trip the upper bound)
negative_count	Question matches _COUNT_PAT (and NOT _RATIO_PAT) AND any numeric column has a value < 0	Counts cannot be negative; catches sign-flipped subqueries / bad COALESCEs
ratio_zero_int	Question matches _RATIO_PAT AND the lone numeric column is integer-typed AND all values are 0	Almost always a floor-division bug ("how many times more" answered with 5 // 12 = 0); narrowly scoped to avoid false positives on legitimate ratio == 1

Wire-up (nodes.critic_execute_node, marked # === CHANGE E (M3) ===):

1. Run run_numeric_guards(question, result_df) after the LLM critic and after the column/row-count post-checks.
2. Flags are always appended to state.result_critique.issues and surfaced in the trace under observation.numeric_guard_flags (visible to failure_tagger).
3. Only when the critic had marked the result valid does a flag downgrade is_valid=False, cap confidence ≤ 0.4, and push a retry: … suggestion. The existing executor-feedback loop then fires; max_total_attempts=4 in graph.py naturally caps the extra cost at one retry.
4. Env gate: ENABLE_NUMERIC_GUARDS (default 1). Disabling preserves pre-mechanism behaviour exactly.

Rollback E: delete numeric_guards.py, remove the import and the # === CHANGE E (M3) === block in nodes.critic_execute_node, and drop numeric_guard_flags from the observation dict.

Change F — Mechanism 2: Independent Sanity Check (#13 in CRITIC_EXECUTE_SYSTEM)

A 13th checklist item added to the execute critic, framed strictly as a second opinion that ignores the code:

13. sanity_independent — Pretend you have not seen the Generated Code or Execution Stdout. Read ONLY the Original Question and the Execution Result preview. PASS iff the result, on its face, looks like a plausible answer to that question — the column(s) shown match the kind of thing the user asked for, and the value magnitudes are not absurd given the question's framing. FAIL if a naive reader of just (question, result) would say "that doesn't answer the question". The evidence field must quote one short phrase from the question and one value/column from the result preview.

Verdict weighting: failure is soft — does NOT flip is_valid on its own, but caps confidence ≤ 0.55 and forces the first suggestion to start with retry: independent sanity check failed — …. The existing low-confidence retry branch in graph.run_agent_graph (< 0.7) then routes the result back to the executor for one more attempt.

User-prompt isolation (CRITIC_EXECUTE_USER, marked # === CHANGE F (M2) ===): an instruction block at the end tells the critic to evaluate check #13 first, using only the question + result-preview sections, before reading the Generated Code / Execution Stdout sections. This preserves the "unbiased second opinion" property — the other 12 checks are evidence-based and read the code; #13 deliberately doesn't.

Why not a separate node? Considered and rejected: a standalone result_sanity node would have its own LLM call, its own retry routing, and its own context-window cost, while reusing zero infrastructure. Folding it in as check #13 with an "evaluate first, in isolation" instruction achieves the same separation-of-evidence at zero new graph plumbing. If trace inspection later shows #13 is biased by the surrounding code context, promoting it to a standalone node is a small refactor.

Rollback F: delete the four marked # === CHANGE F (M2) === blocks in prompts.py (schema entry in checks[], check #13 definition, verdict rule, user-prompt instruction).

How E and F interact

• E fires first (deterministic, runs every time after the critic). When E flags something, the issue is in state.result_critique.issues and the executor sees it in feedback_section on retry.
• F is consulted inside the critic call itself, before E runs. A check #13 failure caps confidence ≤ 0.55, which falls below the 0.7 happy-path threshold and triggers retry independently of E.
• Both pathways converge on the same single-retry executor budget, so the worst-case extra cost is one additional executor_node call + one additional critic_execute_node call per task — bounded by max_total_attempts=4.
• Neither E nor F encodes any task-specific pattern, idiom catalogue, or failure-derived example. All rules apply uniformly to every question.

Change G — Cross-Validation via Dual-Track Re-Derivation (cross_validate.py)

After the executor succeeds, a second LLM call re-derives the answer using the opposite tool (SQL↔pandas). The two results are compared using loosened competition scoring normalization. Only agreement is surfaced to the critic (strong positive signal); divergence silently caps confidence.

Scope: ALL easy and medium tasks on execution attempt 1. No restriction on result shape (not limited to 1×1 or aggregation-only).

Flow:

1. should_cross_validate(state) gates on: ENABLE_CROSS_VALIDATE=1, difficulty ∈ {easy, medium}, execution_attempts == 1, non-empty result.
2. _detect_primary_tool(code) heuristic scores SQL vs pandas indicators.
3. CROSS_VAL_SYSTEM/USER prompts ask the LLM to rewrite the computation using the alt tool → produces alt Python code.
4. Alt code is executed in a sandboxed exec() (same as executor).
5. _dataframes_agree(primary_df, alt_df) — loosened comparison (see below).
6. format_cross_val_annotation(cv_result):
   • Agreed → Positive annotation to critic: "High confidence the answer is correct."
   • Alt failed / Diverged → Empty string (nothing shown to critic). Divergence only caps confidence post-critic.

Asymmetric surfacing rationale: Agreement is a strong positive signal (both independent tracks converged). Divergence is ambiguous — the alt-tool code is often subtly wrong (different LIMIT, column ordering, type coercion), so surfacing it to the critic causes it to second-guess correct results (observed: task_11 regressed 1.0→0 when divergence was shown). Suppressing divergence from the critic while still using it for a soft confidence cap avoids this failure mode.

Post-critic confidence cap (nodes.critic_execute_node, marked # === CHANGE G post-critic ===):

• If CV ran, alt succeeded, but DIVERGED, and critic accepted the result: cap confidence ≤ 0.75.
• This is just above the 0.7 happy-path threshold, so it only triggers a retry when the critic was already borderline. It does NOT flip is_valid.

Comparison tolerance (loosened): _dataframes_agree applies multiple relaxations to reduce false-divergence:

• Ignores column count differences (compares min(cols) by position)
• Ignores row order (sorts both normalized matrices)
• Ignores extra rows (compares min(rows) — alt code may not apply same LIMIT)
• Numeric tolerance: ±0.01 beyond the 2dp rounding (catches float-vs-Decimal drift)
• Partial agreement: ≤2 mismatches across >80% matching cells → still "agreed"

Failure handling:

• If alt-tool execution fails (LLM error, runtime exception, no result_df produced): nothing is shown to critic; result stored in trace only.
• If both tracks somehow fail: nothing shown to critic.

Critic decides next steps: The critic (not deterministic code) decides whether to accept, retry, or replan. CV only provides a positive boost (agreement annotation) or a soft confidence cap (divergence). This avoids hard-wiring acceptance/rejection logic that could overfit.

Cost: One additional LLM call (alt code generation) + one exec() call per easy/medium task on attempt 1. No extra LLM call for the comparison itself (pure Python).

Wire-up (nodes.critic_execute_node, marked # === CHANGE G ===):

• Runs before the critic LLM call, after building the user message but before sending it.
• state.cross_validation stores the CrossValidationResult for trace visibility.
• Agreement annotation (if any) is appended to the critic's user message.
• After critic returns: divergence confidence cap applied.

State (state.py): CrossValidationResult dataclass with fields: primary_tool, alt_tool, alt_succeeded, alt_error, agreed, divergence_summary, alt_result_preview. Added as AgentState.cross_validation: CrossValidationResult | None.

Env gate: ENABLE_CROSS_VALIDATE (default "1"). Set to "0" to disable entirely.

Rollback G: delete cross_validate.py, remove the import and the # === CHANGE G === / # === CHANGE G post-critic === blocks in nodes.critic_execute_node, remove CrossValidationResult from state.py and the cross_validation field from AgentState.

---

8. Evaluation & Diagnostics CLI

Two evaluator-side commands live in src/data_agent_baseline/cli.py:

eval-lang

Per-task language-recall evaluation. Output includes:

• Per-task table with task_id, difficulty, recall, precision, Time (s) (from trace).
• Summary: tasks with recall > 0 vs recall = 0 at each λ.
• Breakdown by Difficulty table (easy → extreme + overall row) showing mean recall / precision / count.
• Zero-recall task list for quick triage.

Powered by evaluator.py helpers: _read_elapsed_seconds, _read_difficulty, _autodetect_task_root, and the new difficulty / elapsed_seconds columns + difficulty_breakdown in the summary dict.

tag-failures

Failure classification + phase-timing aggregator. Reads a run dir and emits four tables plus failure_tags.csv:

1. Per-task table — task_id, bucket, primary error, recall.
2. Bucket summary — counts per bucket, sorted by severity.
3. Per-phase timing — total seconds + call count per phase (explore, plan, execute, critic, …), derived from trace action events via _ACTION_TO_PHASE.
4. Per-bucket task lists — quick lookup of which task ids fell into each bucket.

Buckets (defined in failure_tagger._classify_bucket): perfect, near_miss, low_recall, value_mismatch, wrong_column_count, wrong_row_count, empty_prediction, no_prediction, timeout, api_error, crash, no_gold, other.

eval-comprehensive

Comprehensive evaluation harness for ablation studies. Captures per-task metrics + run-level summary statistics. Output: comprehensive_evaluation.csv with per-task rows and an in-memory summary dict.

Per-task metrics (columns):

• Identification: task_id, difficulty, trace_id (full path)
• Execution: execution_success (0/1), execution_time (s), execution_error, expected_output_column_count, expected_row_count
• Scoring: final_score, recall, extra_columns, matched_columns, gold_columns, pred_columns
• Trajectory: trajectory_length, tool_calls, tool_failures, llm_calls, recovery_attempts, clarification_requests, plan_attempts, execution_attempts
• Quality: confidence_score (numeric 0-1), confidence_label (High/Good/Moderate/Low), ground_truth_available (Y/N), bucket (failure mode)
• Resources: total_tokens, failure_reason
• Per-stage demarcation: for explore, planner, critic_plan, execute, critic_execute the CSV includes time, tool calls, tool failures, tokens, and llm_calls.

Trace alignment:

• trace.json now stores stage_metrics and comprehensive_metrics blocks.
• comprehensive_evaluator.py primarily reads these blocks and only performs basic calculations that require gold labels (score/recall/extra/matched and bucket logic).

Bucket classification: Uses same logic as tag-failures for consistency.

Run-level summary includes:

• Aggregate metrics: mean/median/std/min/max for score, recall, execution time
• LLM usage: total_llm_calls, mean_llm_calls, per-phase LLM call means
• Distribution stats: execution_success_rate, tasks_with_score_gt_0, tasks_with_perfect_score, tool_failure_rate
• Breakdowns: difficulty_breakdown (per-difficulty stats), bucket_distribution (count per failure mode), bucket_details (per-bucket mean score/time)
• Phase timing table: includes total_llm_calls and mean_llm_calls per phase

Use cases:

• Compare baseline vs improved configurations (e.g., with_critic vs without_critic)
• Track performance across task difficulties
• Identify dominant failure modes
• Measure resource efficiency (time, tokens)
• Support longitudinal ablation studies across project phases

Powered by comprehensive_evaluator.py (evaluate_run_comprehensive, evaluate_task_comprehensive, classify_task_outcome).

eval-v2 (Integrated Hardening Suite)

KDD Creative Track comprehensive evaluation harness with automatic validation and diagnostics. Produces three normalized CSV files plus hardening artifacts.

Evaluation Output:

• task_metrics.csv — 100+ columns per task: correctness (multi-level F1), autonomy, planning quality, tool efficiency, data understanding, verification, recovery, trajectory, failure taxonomy, confidence calibration, composite scores
• trajectory.csv — Step-by-step execution trace with phase, agent, tool, success, tokens, timing
• tool_calls.csv — Per-tool-call analysis with parameters, results, failures
• comprehensive_evaluation.csv — Backward compatibility (same as task_metrics.csv)

Hardening Artifacts (Auto-Generated):

1. Artifact Reconciliation (artifact_reconciliation_report.txt) — Validates:

   • Tool call counts match across artifacts
   • Token counts reconcile (trajectory vs metrics)
   • Verification semantics (aat_verification_passed ↔ coordinator_final_decision="APPROVE_FINAL")
   • Time accounting (component compute ≤ wall clock, overhead tracked)
   • Trajectory completeness

2. Replay Artifacts (task_*/task_replay.json) — Complete debug snapshots:

   • Question, context, all execution attempts
   • Agent executions, tool calls, coordinator decisions
   • Verifier output, final answer, evaluation result
   • Structured failure attribution (category, root cause, stage, suggested fix)

3. Engineering Health Report (engineering_health_report.txt) — System diagnostics:

   • Separated Assessments:
     • Harness Health (0-10): Infrastructure quality (reconciliation, validators, attribution coverage)
     • Run Quality: Outcome metrics (answer accuracy, execution success rate)
   • Reconciliation pass/fail status
   • Invariant violations
   • Unaccounted Overhead Time (framework overhead, I/O wait, async queuing)
   • Failure taxonomy distribution
   • Top recurring root causes

Display Features:

• Separated Health Assessments:
  • Harness Health (9.7/10 ✓ HEALTHY): Infrastructure quality
  • Run Quality (⚠ DEGRADED 64% accuracy): Outcome metrics
• Execution vs Answer Quality Distinction:
  • Execution Success: Code ran without crashes
  • Answer Accuracy: Correct results (final_score ≥ 0.8)
• Difficulty Breakdown shows both metrics separately
• MAS Failure Analysis:
  • MAS Failure Categories: Structured categories (REASONING_FAILURE, DATA_UNDERSTANDING_FAILURE, etc.)
  • Failure Distribution by Stage: AAT phases (UNDERSTAND/PLAN/EXECUTE/VERIFY)
  • Outcome Error Types: Evaluation buckets (low_recall, wrong_schema, etc.)
• AAT Architecture Metrics: Coordinator decisions, specialist activation, verification outcomes
• Phase Timing Reconciliation: Displayed time vs cumulative compute with overhead interpretation

Validation Improvements (June 2026):

• Fixed verification_passed semantics: Critic verification (legacy) vs AAT verification (coordinator approval) are now distinct
• Fixed data_understanding_score validation: Recognizes weighted formula (0.5·specialist + 0.5·discovery)
• Fixed time accounting: Component < wall_clock is expected (overhead); warns only if ratio < 5%

Usage:

dabench eval-v2 <RUN_ID> --mode standard   # Quick overview
dabench eval-v2 <RUN_ID> --mode verbose    # Detailed analysis
dabench eval-v2 <RUN_ID> --mode research   # All metrics for papers

Powered by eval_v2.py, eval_v2_metrics.py, eval_v2_viz.py, eval_v2_validator.py + hardening suite (eval_artifact_reconciliation.py, eval_replay_artifacts.py, eval_health_report.py).

---

9. Prompt Architecture (changes A / B / C)

All four agent system prompts now share a layered structure defined in prompts.py. Each change is marked with # === CHANGE X — START/END === block comments so it can be rolled back independently.

Change A — SHARED_HEADER prepended to every SYSTEM prompt

A single constant carrying the four highest-leverage rules sits at the top of prompts.py and is concatenated onto PLANNER_SYSTEM, CRITIC_PLAN_SYSTEM, EXECUTOR_SYSTEM, and CRITIC_EXECUTE_SYSTEM:

1. Scoring rule — binary column-matching; "when in doubt, INCLUDE the column".
2. Ground-truth hierarchy — Data Exploration wins on column existence; knowledge.md wins on value encodings.
3. Answer-shape vocabulary — cardinality / value_type / unit triple keyed off linguistic cues ("how many" → scalar_int / count, "how many times more" → ratio, etc.).
4. Naming discipline — never return ID when name is asked; never concatenate first_name + last_name.

Rollback A: delete SHARED_HEADER and change SHARED_HEADER + """\ back to """\ on the four SYSTEM prompts.

Change B — CRITIC_EXECUTE_SYSTEM rewritten as a forced 12-item checklist

The execute critic must now emit one {id, passed, evidence} entry per check rather than free-form prose, then apply deterministic verdict rules. The 12 checks:

#	Check ID	Effect when failed
1	required_columns_present	is_valid=false, conf ≤ 0.2
2	count_vs_list	is_valid=false, conf ≤ 0.3
3	name_not_id	is_valid=false, conf ≤ 0.4
4	name_columns_separate	is_valid=false, conf ≤ 0.4
5	no_null_zero_filter	warning
6	no_head_or_limit	warning
7	ratio_not_count	is_valid=false, conf ≤ 0.4
8	monthly_vs_annual	warning
9	value_encoding_correct	is_valid=false, conf ≤ 0.4
10	ordering_explicit	warning only
11	magnitude_reasonable	warning only
12	not_empty	is_valid=false, conf ≤ 0.1, suggestions must start with replan:
13	sanity_independent (Change F)	soft — is_valid unchanged, conf ≤ 0.55, first suggestion must start with retry:

Suggestions are formatted as fix: … / replan: … / retry: … prefixes for easier downstream routing.

Rollback B: rename active CRITIC_EXECUTE_SYSTEM to something else and rename CRITIC_EXECUTE_SYSTEM_LEGACY back to CRITIC_EXECUTE_SYSTEM.

Change C — CRITIC_EXECUTE_USER checklist nudge

The user prompt now labels Expected Output Columns as "REQUIRED columns for scoring — check #1" and ends with Walk the 12-item checklist in order. … Output JSON only. Original preserved as CRITIC_EXECUTE_USER_LEGACY.

Rollback C: rename CRITIC_EXECUTE_USER_LEGACY back to CRITIC_EXECUTE_USER.

Change H — Doc Extraction Strategies 1/2/3 (hard-task coverage)

Three generic improvements to doc extraction in semantic.py:

Strategy 1 — Question-aware column subsetting: When column_hints exist and the probe schema has >5 columns, the extraction prompt receives only merge_key + question-relevant columns. This reduces tokens per record, allowing more records per batch. Full schema is preserved in the table definition; un-extracted columns become NULL. Subsetting only activates if ≥30% of columns can be dropped.

Strategy 2 — Adaptive batch budget: In the flat extraction path, if record_count_estimate is high, max_batches is automatically increased (up to 2×) based on the heuristic of ~30 records per batch. This prevents over-compression of large docs (e.g., 750-record superhero.md with only 12 batches → now up to 25). Wall-clock cap still honoured via deadline_monotonic.

Strategy 3 — Threshold/range registry: _extract_threshold_rules now parses three additional patterns beyond the original "values above X considered Y": normal range: X-Y, normal: < X, abnormal if > X. These populate EntityField.normal_range and add scope="threshold" constraints. critical_rules_for_question surfaces normal_range for question-relevant fields.

Rollback H: Revert the three blocks in extract_doc_records_chunked (Strategy 1 column subset, Strategy 2 adaptive budget) and restore _extract_threshold_rules to single-pattern form.

Downstream compatibility

critic_execute_node in nodes.py reads only is_valid / confidence / issues / suggestions from the critic JSON — the new checks array is purely additive and requires no parser changes. The existing col_count_info / row_count_info text appended to the user message is now partially redundant with the structured required_columns_present / count_vs_list checks but is left in place for safety.

Change I — Multi-candidate plan voting + diagnostic probes (anti-non-determinism)

Files: src/data_agent_baseline/langgraph_agent/vote_plan.py (new), integrated in graph.py.

Mechanism — Plan Voting:

1. On the first planning attempt only, 2 candidate plans are generated in parallel.
2. Compare output_columns:
   • Agree → use candidate 1, proceed normally (90% of tasks).
   • Disagree → run ONE deterministic SQL probe against _auto.db to check which columns actually exist in the database schema.
   • Probe scores each candidate by how many of its output_columns are real DB columns.
   • Higher score wins. If tied, generate a 3rd informed plan with the probe evidence injected as feedback.
   • Final fallback: pick the plan with fewer columns (conservative).

Mechanism — Zero-row Diagnostic Probe: When the executor produces a result with 0 rows and the critic rejects it, run_zero_row_diagnostic is called before retry. It queries _auto.db to verify:

• Tables mentioned in the plan exist and have rows
• Column names actually present
• Sample values from filter-like columns (status, type, category, etc.)

This evidence is appended to the critic's suggestions so the next executor attempt (or replan) has concrete data about what's actually in the DB.

Cost: +1 LLM planner call on first attempt (parallel, same wall-clock). Diagnostic probes are pure SQL (0 LLM cost). 3rd informed plan only triggers on disagreement (~10% of tasks).

Kill-switch: Set VOTE_ENABLED = False in vote_plan.py.

Rollback I: In graph.py: (1) replace voted_planner_node import/call with planner_node, (2) remove the run_zero_row_diagnostic block in the critic-rejected path, (3) delete vote_plan.py.

Fix J — Runner prediction.csv guardrails

Two fixes to runner.py that caused regressions in run 20260521T053658Z:

1. Always write prediction.csv — Previously, if execution_result was None or empty, no file was written, causing "prediction.csv missing" (0 score). Now a fallback pd.DataFrame({"answer": [""]}) is always emitted.

2. Removed "drop all-NaN columns" logic — The runner was dropping columns where all values are NaN/empty. This actively harms scoring because the evaluator checks column presence. Example: task_257 had DisplayName=None (executor couldn't find the editor) → runner dropped the column → lost 0.5 score. Columns with null values must still be emitted.

Rollback J: Re-add the non_empty_mask filter block in _save_result, change the else fallback to prediction_csv_path = None.

Fix K — Local import cleanup

All local/conditional from ... import and import statements inside functions were moved to top-level across graph.py, nodes.py, semantic.py, and model.py. This fixed UnboundLocalError: cannot access local variable 'CriticFeedback' (task_196 crash) caused by Python's scoping rules treating local imports as local variable declarations for the entire function scope.

Files touched: graph.py (removed 2× from ... import CriticFeedback, 2× import sqlite3, 2× import pandas as pd), nodes.py (removed import os as _os, import time as _time, import hashlib as _hashlib, import sqlite3 as _sqlite3, import io, import sys), semantic.py (removed 3× from concurrent.futures import ..., 3× import time as _time, import copy, import sqlite3, import pandas as pd, import traceback, import re), model.py (moved from ... import get_cached, set_cached to top-level).

Rollback K: N/A — pure cleanup, no behavioral change beyond fixing the CriticFeedback crash.

Strategy 1+3 — Deterministic seed injection (anti-non-determinism)

Files: src/data_agent_baseline/langgraph_agent/deterministic_seed.py (new config), src/data_agent_baseline/agents/model.py (integration).

Problem: Executor-level LLM non-determinism causes regressions between runs even with temperature=0. Tasks like task_355, task_173, task_249 produce different code each run — sometimes raising exceptions on 0 rows, sometimes using wrong filter logic or column naming.

Mechanism:

• Injects seed=42 into every client.chat.completions.create() call (both OpenAI and Azure adapters).
• The OpenAI/Azure API seed parameter makes the model use deterministic sampling ("best effort") — not guaranteed identical but significantly reduces variance between identical prompts.
• Temperature was already 0.0 via config; this change adds the complementary seed parameter.

Configuration:

• LLM_SEED=42 (env var override for A/B testing different seeds)
• LLM_SEED_ENABLED=0 (env var kill-switch to disable seed injection)

Cost: Zero — purely a parameter addition to existing API calls.

Rollback Strategy 1+3: In model.py: (1) remove from data_agent_baseline.langgraph_agent.deterministic_seed import DETERMINISTIC_SEED, SEED_ENABLED, (2) remove the if SEED_ENABLED: _create_kwargs["seed"] = DETERMINISTIC_SEED blocks in both adapters, (3) restore direct client.chat.completions.create(model=..., messages=..., temperature=...) calls, (4) delete deterministic_seed.py.

Strategy 5 — Post-hoc deterministic output guards

Files: src/data_agent_baseline/langgraph_agent/post_hoc_guards.py (new), integrated in graph.py.

Problem: The executor LLM generates code with non-deterministic column naming (e.g., "display_name" vs "DisplayName") and column ordering. These differences cause lambda-score regressions even when the underlying data is correct.

Mechanism — applied after executor_node, before critic_execute_node:

1. Column name normalization: If result columns differ from plan.output_columns only by case/spacing/underscores, auto-rename to match the plan. Only applied when column count matches exactly (no ambiguity).
2. Column reordering: If result has the same column set as expected but in different order, reorder to match plan.
3. Empty-result column presence: If result is 0 rows, ensure all expected columns exist (add missing ones as empty) so the output CSV schema is correct.
4. Extra column pruning: If result has MORE columns than expected and ALL expected columns are present (exact or normalized match), drop the extras. Extra columns only incur λ-penalty — pruning when the expected set is fully covered is always safe. Fixes cases like task_194 where atom_id/atom_id2 were included alongside the requested bond_id.
5. Effectively-empty detection: A single-row result where ALL values are empty/blank/NaN (e.g., answer="") is converted to a truly empty DataFrame. This triggers the not_empty critic check (#12) → replan, rather than silently accepting "I couldn't find the answer" as a valid result. Fixes task_173-type regressions where Strategy 4/8 turned crashes into silent empty answers.

Key design constraints:

• Never re-invokes the LLM — purely deterministic DataFrame operations.
• Only fires when the fix is unambiguous (e.g., same number of columns, case-insensitive match).
• Guard 4 only removes columns when ALL expected columns are confirmed present.
• Guard 5 only converts to empty when literally every cell is blank/NaN (very conservative).
• Cannot make results worse: renaming display_name to DisplayName when the plan says DisplayName can only improve the lambda-score.

Configuration:

• POST_HOC_GUARDS_ENABLED=0 (env var kill-switch)

Cost: ~0ms per task (DataFrame column operations on already-computed result).

Rollback Strategy 5: In graph.py: (1) remove from ... import apply_post_hoc_guards, (2) remove state = apply_post_hoc_guards(state) after executor_node, (3) delete post_hoc_guards.py.

---

Guard 6 — Magnitude Probing

File: src/data_agent_baseline/langgraph_agent/post_hoc_guards.py (_magnitude_probe function).

Problem: Single-value numeric results can be wildly implausible (e.g., result=31600 when all numeric columns in the DB max at 100) due to missed division, wrong aggregation, or incorrect JOIN multiplying rows.

Mechanism: For single-row, single-column numeric results:

1. Queries _auto.db to find global MIN/MAX across all numeric columns in all tables.
2. If |result| > 10000 × max(|global_min|, |global_max|), flags result as implausible.
3. Converts to empty DataFrame → triggers retry with feedback.

Constraints:

• Only activates for 1×1 numeric results (avoids false positives on multi-row/multi-col).
• Very conservative threshold (10000×) — only catches truly absurd magnitudes.
• Skips zero/NaN values.

Cost: ~1ms (a few PRAGMA + MIN/MAX queries on _auto.db).

Rollback Guard 6: Delete _magnitude_probe function and the Guard 6 block in apply_post_hoc_guards.

---

Bidirectional Table Detection

File: src/data_agent_baseline/langgraph_agent/nodes.py (_detect_bidirectional_tables function).

Problem: Tables like connected store symmetric relationships (for every row (A,B) there exists (B,A)). Naive COUNT/JOIN on such tables double-counts unless the query adds WHERE col1 < col2 or uses DISTINCT unordered pairs. This caused task_196 to return 2× the correct answer.

Mechanism: During explore_data_node, after building _auto.db:

1. For each table with ≤6 columns, checks all column pairs (i, j).
2. For each pair, verifies that swapping the two columns produces identical rows (every row has its mirror).
3. If detected, appends an annotation to state.data_exploration warning the planner/executor to use WHERE col1 < col2.

Constraints:

• Only checks tables with 2–6 columns (edge-list heuristic).
• Skips tables with >6 columns or 0 rows.
• Purely informational: adds a warning to data_exploration text, doesn't modify queries.

Cost: A few SQL queries per small table during exploration (~5ms total).

Rollback Bidirectional Detection: Delete _detect_bidirectional_tables function and the "CHANGE: Bidirectional table detection" block in explore_data_node.

---

Increased Retry Budget (max_total_attempts 3→5)

File: src/data_agent_baseline/langgraph_agent/graph.py (line ~245).

Problem: With only 3 attempts, tasks that need 1 failed attempt to learn the schema + 1 failed attempt to fix logic errors exhaust the budget before producing a correct answer.

Fix: Changed max_total_attempts = 3 → max_total_attempts = 5.

Cost: Up to 2 additional LLM calls per task (only when earlier attempts fail). Worst-case ~4 extra calls per failed task.

Rollback: Change max_total_attempts = 5 back to max_total_attempts = 3 in graph.py.

Strategy 4 — "Never raise on empty" executor prompt constraint

File: src/data_agent_baseline/langgraph_agent/prompts.py (added rules to EXECUTOR_SYSTEM).

Problem: When a query/filter returns 0 rows, the LLM sometimes generates raise ValueError("No X found") or raise Exception(...). This causes a hard crash — the executor never produces result_df, so the task gets 0 score. Examples: task_257 (ValueError: No post found), task_355 (SyntaxError from over-complex fallback logic).

Fix: Added explicit rules to EXECUTOR_SYSTEM:

• "NEVER RAISE EXCEPTIONS ON EMPTY RESULTS" — always create result_df = pd.DataFrame(columns=[...]) instead
• "NEVER use sys.exit(), exit(), quit(), or return at module level"

Cost: Zero (prompt text only).

Rollback Strategy 4: Remove the two bullet points starting with "NEVER RAISE EXCEPTIONS" and "NEVER use sys.exit()" from EXECUTOR_SYSTEM in prompts.py.

Strategy 8 — Executor safety wrapper (template wrapping)

File: src/data_agent_baseline/langgraph_agent/nodes.py (_build_safety_wrapper function).

Problem: Despite prompt instructions (Strategy 4), the LLM may still generate code that raises exceptions or has syntax errors. A crashed executor = 0 score. We should NEVER crash.

Mechanism: Before exec(), the generated code is wrapped in a safety template that compiles and executes inside a try/except. If ANY exception occurs (ValueError, SyntaxError, KeyError, etc.), the wrapper catches it and sets result_df = pd.DataFrame(columns=expected_output_columns).

This guarantees that result_df always exists after execution — even if the LLM code is syntactically invalid or raises. The empty DataFrame with expected columns then flows through to post-hoc guards (Strategy 5) and the critic, which can request a retry with feedback.

Key property: The wrapper captures exceptions into stdout so the critic/retry feedback loop can see what went wrong, while still producing a valid (empty) result.

Kill-switch: EXECUTOR_SAFETY_WRAP=0 env var.

Cost: ~0ms overhead (compile + exec overhead negligible).

Rollback Strategy 8: In nodes.py: (1) remove _SAFETY_WRAP_ENABLED and _build_safety_wrapper, (2) restore direct exec(code, exec_namespace) in executor_node, (3) remove the _safety_wrapper variable and expected_output_columns lines.

Option B — query_db injectable helper (SQL error auto-recovery)

Files: src/data_agent_baseline/tools/sqlite.py (new query_db function + helpers), src/data_agent_baseline/langgraph_agent/nodes.py (injection into executor exec_namespace), src/data_agent_baseline/langgraph_agent/cross_validate.py (injection into cross-validation exec_namespace).

Problem: The executor generates Python code that calls pd.read_sql_query() / sqlite3 directly. When the LLM hallucinates column names or fails to escape apostrophes, the error surfaces as a crash inside the safety wrapper — producing an empty DataFrame. The improved execute_read_only_sql error hints are never seen because the executor bypasses that function entirely.

Mechanism: A new query_db(db_path, sql, *, limit=200) → pd.DataFrame function in sqlite.py that:

1. Executes SQL via pd.read_sql_query on a read-only connection.
2. On syntax error (e.g., near "s": syntax error from Women's Soccer): auto-fixes unescaped apostrophes using _fix_unescaped_quotes() (regex: (\w)'(\w) → \1''\2) and retries.
3. On "no such column" (e.g., SEX when actual column is Sex): auto-fixes via _fix_column_case() which builds a lowercase → actual_name map from PRAGMA table_info across all tables, then applies case-insensitive regex replacement. Retries with the fixed SQL.
4. On unrecoverable failure: raises with a full schema hint (all tables + their columns) so the safety wrapper's captured traceback gives the executor retry loop actionable feedback.

Injection: In executor_node, a pre-bound closure _query_db_bound(sql) (with db_path=state.auto_db_path already captured) is injected into exec_namespace as query_db. The executor's generated code can call query_db(sql) instead of raw sqlite3.connect() + pd.read_sql_query().

Cross-validation: Same query_db is injected into the cross-validation exec_namespace, fixing the name 'sqlite3' is not defined error (the exec namespace now also pre-injects sqlite3, pd, gzip, json).

Which errors this fixes:

Error	Root cause	How query_db fixes it
near "s": syntax error	Unescaped ' in Women's Soccer	_fix_unescaped_quotes → Women''s Soccer
no such column: SEX	Case mismatch (Sex vs SEX)	_fix_column_case resolves via PRAGMA
name 'sqlite3' is not defined (cross-val)	Empty exec namespace	Pre-injected modules
'utf-8' codec can't decode byte 0x8b (cross-val)	gzip file read as text	gzip module pre-injected + prompt instruction
'event_id' / 'member_id' KeyError (cross-val)	LLM guesses column names	Schema now included in cross-val prompt

Prompt update: EXECUTOR_SYSTEM instructs: "Use query_db(sql) for ALL SQL queries against _auto.db. Do NOT use sqlite3.connect() or pd.read_sql_query() directly." (Note: the LLM may still bypass this instruction — query_db is a best-effort improvement, not a hard enforcement. The safety wrapper remains the last line of defense.)

Cost: Zero additional LLM calls. Runtime overhead: one regex + one PRAGMA scan per failed query (only on error path).

Rollback Option B: (1) In sqlite.py: remove query_db, _fix_unescaped_quotes, _fix_column_case, _get_schema_hint, and the import re / import pandas as pd additions. (2) In nodes.py: remove the _query_db_bound closure and "query_db" from exec_namespace, revert to exec_namespace: dict[str, Any] = {}. (3) In cross_validate.py: remove the query_db import and entry from exec_namespace. (4) In prompts.py: remove the query_db instruction from EXECUTOR_SYSTEM.

---

Execution Memory (Intra-Run Attempt History)

Files: state.py (new execution_memory field), graph.py (accumulation), nodes.py (injection into executor prompt only).

Problem: On retries, the agent only sees the most recent critic feedback. It has no memory of all prior attempts — what code was tried, what results were produced, why they were rejected. This leads to repeating the same mistakes.

Mechanism:

1. Accumulation (in graph.py): After each plan-critic and exec-critic cycle, a condensed entry is appended to state.execution_memory:

   • Plan phase: steps, output columns, reasoning, critic verdict + issues
   • Execute phase: code snippet (last 600 chars), result shape + preview (3 rows), critic valid/confidence/issues/reasoning, stdout tail

2. Injection into executor only: Full history (plan + execute phases) with code, results, rejection reasons. Labeled "do NOT repeat failed approaches".

3. Planner injection DISABLED (run 20260522T080948Z showed it causes plan drift — the planner over-corrects when shown execution-level detail, leading to wrong reinterpretations on task_200 and task_283).

Key properties:

• No persistence — in-process Python list only, GC'd when task finishes.
• Condensed — each entry ~500-800 chars; 4 attempts = ~3KB total injection.
• Plan entries show strategy-level info; executor entries show code-level detail.

Cost: Zero LLM calls. ~0ms (string formatting).

Rollback: (1) Remove execution_memory from state.py. (2) Remove accumulation blocks in graph.py (marked === Memory:). (3) Remove injection block in executor_node (marked === Execution memory).

---

Increased Retry Budget (max_total_attempts 3 → 5)

File: graph.py line ~248.

Problem: With only 3 attempts, tasks needing 1 failed attempt to learn the schema + 1 to fix logic exhaust the budget before producing a correct answer.

Fix: max_total_attempts = 5. A/B testing showed that tasks like task_355 and task_86 require 5 attempts to converge on the correct answer. Reducing to 4 causes regressions on those tasks.

Cost: Up to 2 extra LLM calls per task (only on failure path).

Rollback: Change max_total_attempts = 5 back to 3 in graph.py.

---

Case-Insensitive String Matching Prompt Rule — REMOVED

Status: ROLLED BACK after A/B testing (run 20260522T105623Z).

Reason: The rule was neutral-to-harmful. It didn't fix the target task (task_257) and added prompt noise that could cause over-defensive matching on tasks with exact-match data. Removed from EXECUTOR_SYSTEM in prompts.py.

---

A/B Testing Results (2026-05-22)

Evaluator Discovery

The competition evaluator uses value-based column signatures (sorted normalized cell values), NOT column names. Column naming is irrelevant to scoring. Earlier analysis using column-name recall (0.55) was misleading — actual score is 0.6966 (λ=0.1).

Run Comparison (proper evaluator)

Config	Score (λ=0.1)	Recall>0
Baseline (mem ON, max=5)	0.6966	35/49
Two-Phase Doc Extraction	0.6567	34/50
no_str_match + mem_ON + max=4	0.6697	33/49
no_str_match + mem_OFF + max=5	0.6679	34/49

Current Config (matches baseline)

• max_total_attempts = 5
• Execution memory: ON (accumulation + injection in executor)
• STRING MATCHING rule: REMOVED
• Bidirectional detection: active (≤500 rows)
• Magnitude probing: active
• Value Discovery (Strategy B): active

---

Failure Analysis (14 zero-recall tasks in baseline)

Failure Categories

Category	Tasks	Root Cause
Wrong computation	task_169, task_196	Division logic errors (÷12 missing, bond counting)
Wrong filter/grouping	task_163, task_25, task_80, task_86, task_344	Wrong filter value or event selection
Doc extraction failure	task_352, task_379, task_396	Narrative prose not fully parsed
Extra columns (wrong values)	task_180, task_194	Extra cols pollute value-signature matching
Logic error	task_89, task_214	Wrong approach entirely

Doc Extraction Failures (Highest-Impact)

4 of 14 failing tasks depend on doc/*.md with NO markdown tables — all data in prose:

Task	Doc	Size	Issue
task_344	Patient.md	56KB	Patient records narrative. Needs WBC/FG thresholds not in knowledge.md
task_352	budget.md	63KB	Budget amounts in prose. Multi-hop: budget→event_id linkage
task_379	molecule.md	36KB	Carcinogenic classification in prose. Must join with atom.csv
task_396	superhero.md	178KB	500+ records with height/publisher_id in prose. 5 attempts all failed

What's NOT working:

1. Incomplete extraction: Large docs (178KB, 500+ entities) likely hit token limits
2. Multi-hop linkage: task_352 requires budget_record → event_id → event_name across paragraphs
3. Implicit thresholds: task_344 needs medical domain knowledge absent from data
4. Exhaustive parsing: System extracts keyword-filtered subsets, not ALL records

Potential Improvements (NOT YET IMPLEMENTED)

1. Cross-reference resolution: Trace record ID references across paragraphs (task_352)
2. Domain threshold injection: Inject common clinical thresholds when knowledge.md is silent

---

Change: Stem-Fix (2026-05-22)

Root Cause (task_396): Column subsetting during doc extraction dropped publisher_id because the hint "publisher_name" (from the question) didn't fuzzy-match "publisher_id" (in the doc table). The substring check "publisher_name" in "publisher_id" is False. Result: extraction only asked for [id, height_cm], so the DB had 100 rows with NULL publisher_id. All 5 execution attempts failed with 0 rows after JOIN.

Fix — Stem Matching (semantic.py): Added stem-based column matching to the column subsetting logic. publisher_name → stem publisher matches publisher_id → stem publisher. Now publisher_id is included in extraction_columns.

Files changed:

• semantic.py: Stem matching in column subsetting (~line 1913)

---

Change: Two-Phase Doc Extraction (2026-05-22)

Problem: Large narrative docs (e.g., superhero.md at 178KB) are extracted eagerly before planning. The extraction splits the doc into linear batches, but the doc has separate sections per attribute (Section 3 = height, Section 5 = publisher). Column subsetting means each batch only extracts 2-3 columns, but the LLM sees entities whose data for those columns may not be in that batch's text range. Result: many NULL/placeholder values (e.g., 32/102 entities with valid height vs gold's 62/102).

Fix — Two-Phase Doc Extraction with Planner-Gated Re-extraction:

• Phase 1 (pre-planner): For docs > 40KB, run structure probe (1 LLM call) + extract 3-5 sample rows (1 LLM call). Store profile in state.doc_profiles. Planner sees the table schema + samples.
• Phase 2 (post-planner): After plan is produced, identify which columns the plan needs. Chunk the entire doc into entity-level pieces (by ## headings or paragraph boundaries), pack into 12KB batches, extract ALL entities with all needed columns per batch, merge by entity ID. Replace sample data in _auto.db.

Architecture:

semantic_extraction_node (Phase 1: profile + sample for large docs, eager for small)
    → planner_node (sees schema + sample rows)
    → doc_phase2_extraction (Phase 2: whole-doc entity chunking + batch extraction)
    → executor_node

Key Parameters:

• _DEFER_DOC_SIZE_THRESHOLD = 40,000 — docs above this use two-phase
• _PHASE2_MAX_BATCHES = 20 — max LLM calls per doc in Phase 2
• _PHASE2_BATCH_SIZE = 12,000 — chars per batch (smaller = more complete extraction per batch)

Phase 2 Algorithm (revised 2026-05-22):

1. Parse plan text to identify needed columns (stem-match + exact match)
2. Split entire doc into entity-level chunks via _get_doc_chunks_prioritized():
   • Primary: split at ##/###/#### headings (if ≥10 found)
   • Fallback: split at paragraph boundaries, group small blocks (min 400 chars)
3. Pack chunks into 12KB batches → typically 15 batches for 178KB doc
4. Each batch extracts ALL entities with ALL needed columns (not section-specific)
5. Merge records by entity ID (first-non-null per column across batches)
6. Drop + reload _auto.db table with merged results

Performance (task_396 superhero.md):

• Phase 1: 2 LLM calls (probe + sample) → 5 sample rows in ~8s
• Phase 2: 15 LLM calls × ~5s = ~75s → 465 raw records → 100 unique entities
• Prediction: 53.13% vs gold 54.84% (ratio 0.969)
• Note: 100/750 entities is incomplete but produces near-correct percentages because the sample is representative across the doc's linear order

Evaluation (run 20260522T191106Z):

• Score: 0.6567 (λ=0.1), 34/50 recall>0
• Baseline: 0.6966 (λ=0.1), 35/49 recall>0
• Net regression: -0.04 — Phase 2 helps task_396 approach correct answer but doesn't reach exact match (evaluator needs exact values). LLM non-determinism caused regressions on other tasks (task_25, task_200 went 1.0→0).
• task_396 still scores 0 because 53.13% ≠ 54.84% (evaluator uses exact match on values)

Known Limitations:

1. Only extracts ~100/750 entities from superhero.md — LLM truncates output at ~30-50 records per 12KB batch, and many entities appear in multiple sections causing dedup
2. The evaluator uses exact value matching — even 0.97 ratio gives score 0
3. Adds ~75s overhead to large doc tasks (within 510s timeout but reduces retry budget)
4. Non-doc tasks regress due to LLM non-determinism (not caused by Phase 2)

Files changed:

• doc_phases.py: New module — should_defer_doc_extraction(), doc_phase1_profile(), doc_phase2_extraction(), _get_doc_chunks_prioritized()
• nodes.py: Added deferred-extraction guard in semantic_extraction_node loop
• graph.py: Imports doc_phase2_extraction, calls after planning
• state.py: Added doc_profiles: dict, doc_texts: dict fields

Rollback:

1. In nodes.py: Remove the if should_defer_doc_extraction(...) block (~20 lines)
2. In graph.py: Remove doc_phase2_extraction import and call
3. In state.py: Remove doc_profiles and doc_texts fields
4. Delete doc_phases.py

Decision: ROLLBACK RECOMMENDED — The -0.04 regression outweighs the partial improvement on task_396 (which still scores 0). The fundamental issue is that the evaluator needs exact values, and extracting 100/750 entities can't produce exact percentages. The eager extraction path (with stem-fix) already gets comparable coverage without the overhead.

---

11. Evaluation Harness V2

Evaluation V2 (src/data_agent_baseline/langgraph_agent/eval_v2*.py) provides comprehensive KDD Creative Track evaluation with 50+ metrics organized into 10 categories: correctness (answer/column/row F1), autonomy (first_try_success, replan_count), planning (revisions, dead_ends, alignment), tools (efficiency, diversity, selection_accuracy), data_understanding (tables/columns discovered), verification (score, failures_detected), recovery (success_rate, depth), trajectory (efficiency, branching_factor, critic_loops), failure_taxonomy (category, root_cause, severity), confidence (error, calibration), and composite (analyst_score). Storage: 3 normalized CSVs — task_metrics.csv (1 row per task, 100+ columns), trajectory.csv (1 row per step for process mining), tool_calls.csv (1 row per tool call). Visualization: Rich terminal output with 3 modes — standard (core metrics + summary), verbose (+ agent behavior analysis: analyst quality, tool usage, recovery, verification, failure analysis, trajectory patterns), research (+ all benchmark metrics, aggregated statistics for papers). Enhanced Agent Recording: Nodes now track phase, duration, token_usage, and stage_metrics in trace.json for granular analysis. CLI: dabench eval-v2 <run_id> [--mode standard|verbose|research] generates all metrics; dabench view-task-v2 <task_id> <run_id> shows detailed task breakdown. Backward compatible with V1 via comprehensive_evaluation.csv. See EVALUATION_V2_QUICKSTART.md for usage and EVALUATION_V2_ARCHITECTURE.md for metric definitions.

Evaluation V2 Semantics & Metric Families (KDD Publication Standard)

Phase Timing in Primary Report: The "AAT Execution Phases (Primary)" table displays only coordinator-aggregated AAT phase timings — these represent the synchronous wall-clock time spent in each phase, accounting for task parallelism across specialist agents. This differs from legacy trace accounting, which summed individual agent runtimes (and thus could appear to "double count" when specialists ran in parallel). The primary table does not include legacy values; the compatibility view explicitly shows legacy timing for comparison.

Phase Time Reconciliation Table: "Phase Time Reconciliation" displays cumulative compute time (sum of all contributing components) vs. the displayed phase time. When cumulative compute exceeds displayed time, it indicates overlapping work (e.g., coordinator, schema agent, and domain agent running in parallel during UNDERSTAND). This interpretation is explicit in the table to avoid reader confusion about "missing time" or "double counting."

Verification Metric Families: Evaluation separates verification into two semantic families:

• Critic Family (process metrics): critic_verification_steps, critic_verification_passed, critic_failures_detected, critic_verification_score — capture LLM-internal validation logic (planner critic, execution critic). Backward-compatible aliases (verification_*) map critic fields for legacy code.
• AAT Family (outcome metrics): aat_verification_triggered, aat_verification_passed, aat_verification_score — capture final team verification outcome from independent verifier + coordinator approval decision. Presence indicates AAT execution; absence means critic-only fallback.

Outcome Error Taxonomy: Formerly "Failure Analysis," this table breaks down task results by failure category (e.g., value_mismatch, wrong_schema, perfect) and root cause (e.g., schema_misunderstanding, none). This is an outcome classification, not a process metric, and is orthogonal to verification or recovery attempts.

Data-Understanding Score Gating: The data_understanding_score is gated by documents_required_for_task: if documents are not required per semantic extraction, the score is computed from schema + domain agents only; if documents are required, the document agent activation contributes. This prevents artificial inflation from document-agent availability when documents are irrelevant to the task.

Additional Evaluation Consistency Fixes

Recovery Metrics Aggregation: The "Recovery Success Rate" is computed as aggregate (total_successes / total_attempts), not as the mean of per-task rates. This prevents tasks with zero recovery attempts (i.e., first-try successes) from diluting the rate. Example: given [1/4, 0/0, 0/0] task rates, the aggregate is (1)/(4) = 0.25, not mean(0.25, 0, 0) = 0.083. Task-level rates remain in the CSV for detailed analysis.

Data Understanding Metric Refinement: To avoid variance collapse when all tasks use identical specialist activation patterns, data_understanding_score now blends two components:

• Specialist score (0 to 1): Proportion of engaged specialists relative to gated requirements (e.g., if schema + domain agents engaged and docs not required, score = 2/2 = 1.0).
• Discovery score (0 to 1): Proportion of relevant tables and columns found in all available tables/columns (e.g., if 1 of 1 relevant tables found and 0 of 3 relevant columns found, score = 1/4 = 0.25).
• Blended score: Mean of specialist + discovery scores. This creates discriminative variance reflecting actual understanding depth beyond specialist activation alone.

Verification Timeline Clarity: The verification timeline table's outcome column has been renamed from "Outcome" to "Verification Decision" to clarify that this represents the verification process outcome (approved vs. rejected by verifier + coordinator), not the task success/failure status. This prevents confusion between verification verdicts and task execution results.

Phase 2: MAS Debugging Enhancements (June 14, 2026)

Following initial deployment, eval-v2 received targeted improvements to maximize debugging utility for multi-agent system development. These changes prioritize actionable diagnostics over research metrics:

1. Deterministic Failure Attribution: Maps evaluation buckets (low_recall, wrong_schema, filter_logic_error, etc.) to structured failure categories with consistent semantics:

• Bucket → Category mapping: wrong_column/wrong_schema → DATA_UNDERSTANDING_FAILURE; low_recall/value_mismatch → REASONING_FAILURE; wrong_row → AGGREGATION_FAILURE; crash/timeout → TOOL_EXECUTION_FAILURE; replan_limit → PLANNING_FAILURE
• Infers failure_stage (UNDERSTAND/PLAN/EXECUTE/VERIFY) from category + coordinator decision
• Identifies failure_agent (Schema Agent, Domain Agent, Planner, Executor, Verifier, etc.)
• Eliminates UNKNOWN_FAILURE when bucket exists (prior: 18/18 failed tasks showed UNKNOWN; after: 11 REASONING, 4 DATA_UNDERSTANDING, 3 AGGREGATION)

2. Reality-Aware Health Scoring: Health report now reflects actual system quality:

• Incorporates answer_accuracy_rate (% tasks with final_score ≥ 0.8) and attribution_coverage_rate (% failures with identified cause)
• Status thresholds updated: HEALTHY (9.0+, no integrity errors, >95% attribution coverage), DEGRADED (7.0+, ≥70% attribution), POOR (else)
• Health score deductions for low answer accuracy (<0.8) and poor attribution coverage (<0.95)
• Example: Run with 100% execution success, 64% answer accuracy → 9.7/10 HEALTHY (prior: misleading 10.0/10 HEALTHY)

3. Enhanced Visualization:

• Per-task table column renamed: "Succ" → "Exec" to clarify execution success ≠ answer correctness
• Added "Root Cause" column in standard mode showing failure diagnostics (filter_logic_error, schema_misunderstanding, etc.) for failed tasks
• New "MAS Failure Analysis" section displays failures by stage (UNDERSTAND/PLAN/EXECUTE/VERIFY), by agent (Schema Agent, Domain Agent, Planner, Executor, etc.), and by category with example tasks
• Report structure: Summary → Difficulty Breakdown → MAS Failure Analysis → AAT Metrics (debugging-first ordering)

4. Improved Terminology: Renamed "time gaps" → "Unaccounted Overhead Time" with clear explanation that overhead includes framework orchestration, I/O wait, async queueing, and is informational rather than an error signal.

5. Complete Replay Artifacts: Every task_replay.json now includes full failure diagnostics (failure_category, root_cause, failure_stage, failure_agent, coordinator_decision, verification_outcome, suggested_fix) enabling offline debugging without re-running benchmarks.

Design Philosophy: All Phase 2 changes optimize for MAS debugging workflows (identifying which agents fail on which task types) rather than paper metrics. Backward compatibility maintained via comprehensive_evaluation.csv and legacy metric aliases.

Phase 2.1: Evaluation Harness Cleanup (January 2026)

Focused improvements to separate infrastructure health from outcome quality and enhance replay artifacts:

1. Separated Health Assessments: Clear distinction between harness infrastructure and run outcomes:

• Harness Health (0-10): Infrastructure quality metrics (reconciliation pass rate, validator errors, attribution coverage)
• Run Quality: Outcome metrics (answer accuracy rate, execution success rate)
• Status independently assessed: Harness can be HEALTHY while run quality is DEGRADED (e.g., 64% accuracy)

2. Fixed MAS Failure Categories Display: MAS Failure Categories table now shows structured categories (REASONING_FAILURE, DATA_UNDERSTANDING_FAILURE, AGGREGATION_FAILURE) instead of evaluation buckets (low_recall, wrong_schema, etc.)

• Root cause: eval_v2.py now uses FailureAttributor's structured mapping
• Buckets still shown in "Outcome Error Types" table for evaluation context

3. Partial-Correct Task Handling: New outcome_status field distinguishes task outcomes:

• "correct": final_score ≥ 0.8 (SUCCESS_THRESHOLD)
• "partial": 0 < final_score < 0.8 with execution success
• "failed": final_score < 0.8 or execution failure

4. Enhanced Replay Artifacts: Every task_replay.json now includes:

• Run context: harness_health_status, run_quality_status
• Task outcome: outcome_status, mas_failure_category
• Time accounting: unaccounted_overhead_time_seconds, time_accounting_ratio

5. Consistent Terminology: "Time gaps" renamed to "Unaccounted Overhead Time" throughout (CLI output, reports, documentation) to clarify these are expected framework/I/O overhead, not errors.

Validation: All changes tested on run 20260614T073404Z (50 tasks, 64% accuracy) with zero validator errors.

Phase 0: MAS Debugging Enhancements - Final Round (January 2026)

Goal: Make evaluation harness maximally useful for all future phases (Baseline ReAct, MAS, DAG Visualization, Replay/Time Travel, Confidence & Verification, Research/Ablation Studies).

Key Enhancements:

1. MAS Recovery Effectiveness: Added initial_answer_correct, final_answer_correct, recovered_after_replan, recovered_after_retry fields. New "MAS Effectiveness" section shows first attempt accuracy vs final accuracy and MAS recovery gain (e.g., +14%). Directly answers "Did MAS actually improve answers?"

2. Replan/Retry Effectiveness: Added replan_requested, replan_successful, retry_requested, retry_successful fields. New "Coordinator Intervention Effectiveness" section shows success rates for replans and retries. Reveals which coordinator interventions help.

3. Specialist Agent Value Analysis: Leverages existing schema_agent_used, domain_agent_used, document_agent_used fields. New section shows for each agent: tasks used, accuracy with agent, accuracy without agent, and impact delta. Provides automatic ablation showing which specialists add value.

4. Expanded Failure Stage Taxonomy: Updated FailureStage enum from coarse stages (EXPLORATION, PLANNING, EXECUTION) to AAT-aligned UNDERSTAND/PLAN/EXECUTE/VERIFY/AGGREGATE. Enables finer-grained debugging for AAT phase-specific failures.

5. Cost by Difficulty: Difficulty breakdown now includes Mean Runtime and Mean Tokens columns. Required for Baseline vs MAS vs Future system comparisons.

6. Verification Timeline Clarity: Separated "Execution Approval" (coordinator decision) from "Ground Truth Result" (evaluation correctness). Eliminates confusion between process approval and actual correctness.

7. Comprehensive CSV Storage: All new fields stored in task_metrics.csv including outcome_error_type, MAS recovery metrics, specialist usage. Future phases can run aggregations directly without parsing replay artifacts.

8. Removed Duplicate Reporting: Eliminated redundant "Failure Categories" section. Kept "MAS Failure Categories" (structured) and "Outcome Error Types" (buckets).

Design Philosophy: Every change focused on making the evaluation harness more actionable for debugging and improving the MAS, not for paper-writing. Provides automatic ablation studies and directly answers key questions about MAS effectiveness.

---

12. Baseline Evaluation & Normalized Traces

Phase 1 Baseline Evaluation (src/data_agent_baseline/evaluation/) enables standardized evaluation of the baseline ReAct agent against gold answers. The system adapts baseline traces into a canonical schema supporting multi-agent comparison. Key components: BaselineTraceAdapter converts agent-specific traces to universal format; Phase1Evaluator computes accuracy (precision/recall/F1), efficiency (steps/tools/runtime), and reliability metrics; Phase1ReportGenerator produces task-level CSVs, aggregated JSON, and markdown reports. Normalized Traces are generated as individual JSON files per task (normalized_traces/task_*.json) with agent metadata (agent, agent_role) supporting future multi-agent architectures (planner/executor/critic teams). The schema is NULL-safe (missing data = None), validated on save, and includes derivable metrics (tool counts, agent breakdown, phase timing when available). CLI: dabench eval-baseline <run_id> generates traces + evaluation; dabench view-normalized-trace <run_id> <task_id> displays detailed breakdown. Purpose: Enable fair comparison across baseline ReAct, future multi-agent systems, and ablations using identical evaluation metrics. Gold files located in /data3/dataFAIR/kdd-dev/public/output/task_*/gold.csv. See CANONICAL_TRACE_FORMAT.md for schema details and PHASE1_EVALUATION_GUIDE.md for usage.

---

13. Adaptive Analyst Team (AAT) — Creative Track Multi-Agent Architecture

13.1 Overview & Motivation

The Adaptive Analyst Team (AAT) is a multi-agent architecture that decomposes the data analysis task into specialized cognitive roles, each with domain-specific expertise and carefully isolated context windows. Rather than a monolithic planner/executor system, AAT features:

• Three Specialist Agents with segregated knowledge contexts (Schema Agent sees no documents, Domain Agent sees no structured schema, Document Agent sees neither)
• Strategic Coordinator with three decision checkpoints to route execution, validate intermediate results, and manage team composition
• Synthesizer that merges specialist outputs into a unified analysis context
• Independent Verifier that runs 8-check assurance post-execution
• Confidence Fusion that aggregates per-agent confidence scores with dynamic weighting
• Final Summary Generator that produces human-readable answers from structured verification results

Key Design Principle: Information compartmentalization ensures specialists focus on their domain without distraction or cross-contamination. The coordinator acts as a meta-reasoner, deciding which specialists are needed and validating their outputs before proceeding.

13.2 AAT Workflow Architecture

graph TD
    subgraph CP1["CP1: Understanding Review"]
        review1["Coordinator reviews<br/>task understanding &<br/>data exploration<br/>Decides: which specialists?"]
    end
    
    subgraph Specialists["Specialist Agents (Parallel)"]
        schema["Schema Agent<br/>Analyzes table/column structure,<br/>identifies joins, risks<br/>Input: NO doc content"]
        domain["Domain Agent<br/>Interprets business rules,<br/>canonicalizes question<br/>Input: NO structured schema"]
        doc["Document Agent<br/>Extracts evidence from<br/>doc/*.md narratives<br/>Input: NO db schemas"]
    end
    
    subgraph CP2["CP2: Planning Review"]
        review2["Coordinator validates<br/>specialist outputs &<br/>synthesis quality<br/>Decide: proceed or replan?"]
    end
    
    subgraph CP3["CP3: Final Review"]
        review3["Coordinator makes<br/>release decision<br/>Approve / Reject / Retry"]
    end
    
    explore["Phase 0: Explore"] --> semantic["Phase 0.5: Semantic Extract"]
    semantic --> review1
    review1 -->|specialist decisions| schema
    review1 -->|specialist decisions| domain
    review1 -->|specialist decisions| doc
    schema --> synthesize["Synthesizer:<br/>Merge specialist analyses"]
    domain --> synthesize
    doc --> synthesize
    synthesize --> CP2_block["Planner + CriticPlan<br/>(standard pipeline)"]
    CP2_block --> review2
    review2 -->|plan valid| exec["Phase 2: Execute +<br/>CriticExecute"]
    review2 -->|replan| CP2_block
    exec --> verifier["Verifier:<br/>8-check suite"]
    verifier --> confuse["Confidence Fusion:<br/>Weighted aggregate"]
    confuse --> summary["Final Summary<br/>Generator"]
    summary --> review3
    review3 -->|approve| output["✅ Output +<br/>Artifacts"]
    review3 -->|reject| replan_final["❌ Result rejected<br/>(exhausted budget)"]
    
    style explore fill:#f3e5f5
    style semantic fill:#fce4ec
    style schema fill:#e3f2fd
    style domain fill:#e8f5e9
    style doc fill:#fff3e0
    style synthesize fill:#f1f8e9
    style exec fill:#e8eaf6
    style verifier fill:#fce4ec
    style confuse fill:#fff9c4
    style summary fill:#e0f2f1
    style CP2_block fill:#fff9c4
    style review1 fill:#ffccbc
    style review2 fill:#ffccbc
    style review3 fill:#ffccbc
    style output fill:#c8e6c9
    style replan_final fill:#ffcdd2

13.3 Coordinator Checkpoints

The StrategicCoordinator makes three critical decisions:

Checkpoint 1: Understanding (Post-Explore)

When: After Phase 0 (Explore) and Phase 0.5 (Semantic Extraction), before specialist activation.

Input: Task question, data exploration summary, semantic context (entities, rules, synonyms).

Decision: Which specialists to activate? Options:

• PROCEED — activate selected specialists (Schema, Domain, Document per decision)
• Other specialists default to inactive if not mentioned

Output: UnderstandingReview (task_understanding, specialist_plan, confidence) + AgentTrace

Reasoning: Some tasks are purely structural (no documents, just schema traversal) — skip DocumentAgent. Some need heavy rule interpretation — activate DomainAgent. The coordinator makes this routing decision once, avoiding wasted LLM calls.

Checkpoint 2: Planning (Post-Planner/CriticPlan)

When: After Planner generates an ExecutionPlan and CriticPlan validates it.

Input: The generated plan, specialist analyses (if activated), synthesized analysis.

Decision: Is the plan sufficient or does it need replanning?

• PROCEED — plan is sound, proceed to execution
• REPLAN — identified gaps in plan (missing business rules, incomplete coverage), loop back to planner

Output: PlanningReview (validation_summary, issues_found, replan_rationale, confidence) + AgentTrace

Reasoning: CriticPlan validates syntax; the Coordinator validates semantic coverage against specialist knowledge. This catches "plan is valid Python but violates a business rule" errors early.

Checkpoint 3: Final (Post-Verifier/Confidence Fusion)

When: After Verifier runs 8-check suite and ConfusionFusion aggregates team confidence.

Input: Verification report, team confidence score, execution result.

Decision: Release this answer or reject?

• APPROVE_FINAL — confidence high enough (≥ 0.65), verification passed → emit answer
• REJECT_FINAL — confidence too low or verification failed → emit failure signal
• RETRY_EXECUTION — minor issues detected, single executor retry allowed

Output: FinalReview (verdict, reasoning, confidence_justification, team_input) + AgentTrace

Reasoning: Final gate before output. The coordinator considers all team signals (schema confidence, domain confidence, document confidence, verifier checks) in context and makes a single release decision. This replaces ad-hoc confidence thresholds with principled meta-reasoning.

13.4 Specialist Agents with Context Isolation

Each specialist agent receives only the information needed for its domain, preventing distraction and ensuring independent reasoning.

Schema Agent

Purpose: Analyze data structure — tables, columns, joins, constraints, risks.

Input Context (SchemaAgentContext):

• question: original task question
• available_sources: list of data sources (CSV, JSON, DB, etc.)
• data_exploration: schema summary, row counts, dtypes, 3-sample rows (no doc/*.md content)
• auto_db_path: path to SQLite database for schema queries
• entities_summary: high-level entity list (e.g., "patient, lab_test, prescription")
• relationships_summary: key join paths (e.g., "patient ← → lab_test via patient_id")
• NOT included: doc/*.md narratives, business rules, question interpretation

Output: SchemaAnalysis containing:

• discovered_tables / discovered_columns: names and types
• join_paths: inferred relationships
• identified_risks: data quality issues (missing columns, type conflicts, join risks)
• schema_confidence: 0.0–1.0 confidence in analysis

Domain Agent

Purpose: Interpret business semantics — rules, metrics, synonyms, ambiguities.

Input Context (DomainAgentContext):

• question: original question
• rewritten_question: schema-normalized form from semantic extraction
• constraints_summary: business constraints / SLAs from knowledge.md
• use_cases_summary: known use cases / patterns from knowledge.md
• synonyms_summary: field/entity synonyms from knowledge.md
• NOT included: detailed schema (table list, column dtypes, join details), doc/*.md narratives

Output: DomainAnalysis containing:

• canonicalized_question: business-level interpretation
• identified_metrics: metric names, calculation logic
• identified_rules: business rules that constrain the answer
• identified_ambiguities: interpretations the question leaves open
• domain_confidence: 0.0–1.0 confidence

Document Agent

Purpose: Extract evidence from narrative doc/*.md files.

Input Context (DocumentAgentContext):

• question: original question
• doc_records_summary: high-level description of extracted records (count, entities, key columns)
• doc_profiles_summary: per-doc metadata (size, # records, top entities)
• doc_table_schemas_summary: column names + types for extracted doc tables (no sample data)
• NOT included: detailed database schema (tables, joins), doc content itself (text narratives), business rules

Output: DocumentAnalysis containing:

• extracted_entities: entities found in docs
• extracted_relationships: relationships among entities
• extracted_facts: key data points / evidence
• doc_confidence: 0.0–1.0 confidence in extraction

13.5 Supporting Components

Synthesizer (AnalysisSynthesizer)

Purpose: Merge specialist analyses into a unified context for Planner consumption.

Process:

1. Deterministic merge (always runs): combines tables/columns/rules from all active specialists
2. LLM-enhanced merge (optional): calls LLM to resolve conflicts, enrich descriptions, validate consistency
3. Fallback: if LLM fails, returns deterministic result

Output: SynthesizedAnalysis with format_for_planner() method producing a structured text block that the Planner consumes as additional context.

Verifier (Verifier)

Purpose: Independent 8-check assurance post-execution (no code, only result + question).

Checks:

1. Output schema validation — columns match expected
2. Row count — matches "single" or "multiple" estimate
3. Column count — not exploded with extra join keys
4. Business rule consistency — result respects identified rules
5. Cross-validation agreement — SQL and pandas produce same result (if enabled)
6. Confidence calibration — confidence score is realistic
7. Evidence consistency — result supported by extracted documents
8. Answer supportability — result directly answers the question

Output: VerificationReport with passed/failed checks, recommendation (PASS/FAIL), confidence.

Confidence Fusion (fuse_team_confidence())

Purpose: Aggregate per-agent confidence scores into a single team confidence value.

Weighting Strategy:

• Schema Agent: 0.20 (understands data structure)
• Domain Agent: 0.20 (understands business semantics)
• Document Agent: 0.15 (extracts supporting evidence)
• Planner (via CriticPlan): 0.20 (validates plan)
• Executor (via CriticExecute): 0.25 (validates execution)

Mechanism:

1. Collect confidence scores from all active agents
2. Normalize weights for inactive agents (redistribute proportionally)
3. Compute weighted average
4. Blend with Verifier confidence: final = 0.90 × weighted_avg + 0.10 × verifier_confidence
5. Clamp to [0.0, 1.0]

Output: TeamConfidenceReport with final_team_confidence and reasoning.

Summary Generator (FinalSummaryGenerator)

Purpose: Convert structured verification results into human-readable answer with evidence.

Process:

1. Extract key findings from synthesized analysis + verifier report
2. Format as narrative answer summary
3. Attach supporting evidence citations (from documents, schema)
4. Note limitations and confidence qualifiers
5. Incorporate coordinator approval decision

Output: FinalSummary with answer_summary, supporting_evidence, confidence, limitations.

13.6 Feature Flag Enablement

AAT is disabled by default to maintain backward compatibility. Enable via:

Option 1: Configuration File

Edit configs/react_baseline.azure.yaml (or your target config):

feature_flags:
  enable_adaptive_analyst_team: true

Option 2: Environment Variable

export ENABLE_ADAPTIVE_ASSISTANT_TEAM=1
# or: ENABLE_ADAPTIVE_ASSISTANT_TEAM=true
# or: ENABLE_ADAPTIVE_ASSISTANT_TEAM=yes
python main.py ...

Precedence

1. Environment variable (ENABLE_ADAPTIVE_ASSISTANT_TEAM) overrides config file
2. If neither set, defaults to false (baseline ReAct agent used)

Validation

Run tests to verify feature flag routing:

cd /workspace/ainn-cm-poc-data-agent
uv run pytest tests/test_aat.py::TestFeatureFlag -v

13.7 Integration with Baseline Pipeline

AAT is integrated as an alternative execution mode, not a replacement:

• Baseline path (enable_aat=False): Phase 0 → Phase 0.5 → Planner → CriticPlan → Phase 2 Execute → CriticExecute → Output (existing ~890-line run_agent_graph logic)
• AAT path (enable_aat=True): Phase 0 → Phase 0.5 → CP1 → Specialists (parallel) → Synthesizer → Planner → CriticPlan → CP2 → Phase 2 Execute → CriticExecute → Verifier → ConfidenceFusion → Summary → CP3 → Output (new 890-line _run_agent_graph_aat_inner logic)

Backward Compatibility:

• Existing AgentState fields unchanged; new AAT fields default to None/empty
• All evaluation metrics (eval_v2_schema.py) return zero-values for non-AAT traces
• No changes to baseline execution logic; AAT runs in its own inner graph function
• Feature flag controls routing at graph.run_agent_graph() entry point

13.8 AAT Artifacts & Trace Recording

All AAT components are recorded in trace.json under:

Artifact	Path in trace.json	Type
Understanding Review	observation.understanding_review	UnderstandingReview (JSON)
Planning Review	observation.planning_review	PlanningReview (JSON)
Final Review	observation.final_review	FinalReview (JSON)
Schema Analysis	observation.schema_analysis	SchemaAnalysis (JSON)
Domain Analysis	observation.domain_analysis	DomainAnalysis (JSON)
Document Analysis	observation.document_analysis	DocumentAnalysis (JSON)
Synthesized Analysis	observation.synthesized_analysis	SynthesizedAnalysis (JSON)
Verification Report	observation.verification_report	VerificationReport (JSON)
Team Confidence	observation.team_confidence	TeamConfidenceReport (JSON)
Agent Traces	observation.agent_traces[]	AgentTrace array (JSON)
Coordinator Decisions	observation.coordinator_decisions[]	CoordinatorDecision[] (JSON)
Task Understanding Artifact	observation.task_understanding	TaskUnderstanding (Copilot artifact)
Execution Blueprint	observation.execution_blueprint	ExecutionBlueprint (Copilot artifact)
Execution Progress	observation.execution_progress	ExecutionProgress (Copilot artifact)
Verification Summary	observation.verification_summary	VerificationSummary (Copilot artifact)
Final Summary	observation.final_summary	FinalSummary (JSON)

Copilot Artifacts (for visualization in VS Code):

• task_understanding: Structured understanding of the task (entities, constraints, ambiguities)
• execution_blueprint: Planner's strategy and timeline
• execution_progress: Real-time progress updates as plan executes
• verification_summary: Result of verifier's 8 checks, with pass/fail details

13.9 AAT Module Structure

src/data_agent_baseline/langgraph_agent/aat/
├── __init__.py              Entry point, exports all AAT classes
├── schema.py                30+ dataclasses (SchemaAnalysis, DomainAnalysis, etc.)
├── prompts.py               System + user prompts for all 7 LLM-using components
├── context.py               Context builders (SchemaAgentContext, DomainAgentContext, DocumentAgentContext)
├── specialist_agents.py     SchemaAgent, DomainAgent, DocumentAgent implementations
├── coordinator.py           StrategicCoordinator with 3 checkpoints
├── synthesizer.py           AnalysisSynthesizer (deterministic + LLM merge)
├── verifier.py              Verifier (8-check suite)
├── summary.py               FinalSummaryGenerator (narrative answer + evidence)
├── confidence.py            fuse_team_confidence() + TeamConfidenceReport
└── (tests)
    └── test_aat.py          55 comprehensive tests (all passing)

13.10 Configuration & Constants

AAT respects the same timeouts and resource budgets as the baseline:

Setting	Location	Default	Impact
TASK_TIMEOUT_SECONDS	main.py	510s	Per-task overall budget; AAT checkpoints run within this
max_plan_retries	state.py	2	Replan attempts; triggered by CP2 if specialist checks fail
max_exec_retries	state.py	3	Executor attempts; controlled by CP3
max_total_attempts	graph.py	5	Overall execute+replan budget
LLM_HTTP_TIMEOUT	agents/model.py	90s	Per-LLM-call timeout; applies to all 7 AAT agent calls
ENABLE_ADAPTIVE_ASSISTANT_TEAM	env / config	false	Feature flag; set to true to activate AAT

AAT-Specific Limits:

• Specialist agent calls per task: max 3 (Schema + Domain + Document) — typically all 3 run in parallel
• Verifier calls per task: 1 (post-execution, independent check)
• Confidence fusion calls: 0 (deterministic aggregation, no LLM)
• Summary generator calls: 1 (post-verification, narrative synthesis)

LLM Call Budget:

• Baseline (no AAT): ~3–4 LLM calls (Planner, CriticPlan, Executor, CriticExecute)
• With AAT enabled: ~8–9 LLM calls baseline + 3 specialists + 1 synthesizer-LLM-call + 1 verifier + 1 summary = up to 15 calls under normal operation, but specialist selections (CP1) can deactivate unused agents to reduce cost.

13.11 Evaluation & Metrics

AAT metrics are recorded in eval_v2_schema.py under 30+ new columns:

Coordinator Metrics:

• coordinator_calls: count of coordinator decision calls
• coordinator_failures: count of coordinator LLM failures (with fallback)
• coordinator_tokens: total tokens used by coordinator
• coordinator_time_seconds: total time in coordinator
• coordinator_replans: count of replan decisions (CP2)
• coordinator_retry_requests: count of retry decisions (CP3)

Specialist Metrics:

• schema_agent_used: boolean (1 if Schema Agent called)
• domain_agent_used: boolean (1 if Domain Agent called)
• document_agent_used: boolean (1 if Document Agent called)
• specialists_used: JSON list of specialist names that were activated
• schema_agent_tokens, domain_agent_tokens, document_agent_tokens: token counts per specialist
• schema_agent_time_seconds, etc.: duration per specialist

Verifier Metrics:

• verifier_calls: count of verifier calls
• verifier_failures: count of verifier LLM failures
• verifier_tokens: total tokens used by verifier
• verifier_time_seconds: verifier execution time

Confidence Metrics:

• team_confidence: final aggregated confidence (0.0–1.0)
• verification_confidence: verifier's confidence in result
• verification_passed: boolean (1 if all 8 verifier checks passed)
• understanding_confidence: CP1 coordinator understanding review confidence
• planning_confidence: CP2 coordinator planning review confidence
• execution_confidence: CriticExecute confidence
• final_confidence: CP3 coordinator final decision confidence

CLI Usage:

# Generate all metrics including AAT columns
dabench eval-v2 /path/to/run_id --mode verbose

# View task-specific AAT breakdown
dabench view-task-v2 task_001 /path/to/run_id

13.12 Known Limitations & Future Work

Current Limitations:

1. Specialist context isolation requires LLM-based extraction; Schema Agent receives no sample doc content, preventing certain schema-level doc analysis patterns
2. Coordinator decisions are LLM-based (not heuristic), adding 3 extra LLM calls per task; future work could add deterministic routing rules for common patterns
3. Verifier independence is maintained by design (ignores code) but may miss execution-specific issues; integration with numeric_guards.py could strengthen verification
4. Confidence fusion uses fixed weights; future iterations could learn or adapt weights per task difficulty or domain

Future Enhancements:

1. Adaptive weighting: Learn or adjust specialist weights based on historical task performance per difficulty/domain
2. Specialist specialization: Train or prompt-engineer specialists differently per data domain (healthcare vs finance vs e-commerce)
3. Parallel verification: Run multiple independent verifiers (SQL, pandas, semantic) and aggregate their verdicts
4. Hierarchical coordination: Add intermediate checkpoints between specialists (e.g., Schema→Domain handoff with schema summary)
5. Tool usage tracking: Record which specialists invoked which tools (SQL, pandas, file I/O) for process mining

13.13 Testing & Validation

All AAT components are tested under tests/test_aat.py:

cd /workspace/ainn-cm-poc-data-agent

# Run all AAT tests (55 tests)
uv run pytest tests/test_aat.py -v

# Run specific test suite
uv run pytest tests/test_aat.py::TestStrategicCoordinator -v

# Run with coverage
uv run pytest tests/test_aat.py --cov=src/data_agent_baseline/langgraph_agent/aat

Test Suites:

• TestAATSchema: 10 tests — validates dataclasses, JSON serialization, enums
• TestContextIsolation: 7 tests — verifies specialists only see permitted context
• TestSpecialistAgents: 5 tests — LLM success/failure handling for Schema/Domain/Document agents
• TestStrategicCoordinator: 5 tests — all 3 checkpoints + invalid decision handling
• TestAnalysisSynthesizer: 3 tests — deterministic/LLM merge, missing agents
• TestVerifier: 3 tests — approval/rejection, fallback handling
• TestFinalSummaryGenerator: 2 tests — LLM success and fallback
• TestConfidenceFusion: 4 tests — weighting, redistribution, clamping
• TestFeatureFlag: 5 tests — routing, env override, signature validation
• TestBackwardCompatibility: 4 tests — legacy traces, new columns, from_dict(), trace helper
• TestTraceGeneration: 2 tests — artifact accumulation, JSON serialization
• TestEvalIntegration: 2 tests — empty trace handling, metric extraction

Execution Time: ~1 second for all 55 tests.

---

14. Phase-2 AAT Migration: Evaluation System Fixes

14.1 Problem Summary

Phase-2 migration revealed 8 critical data integrity issues in the evaluation pipeline that caused incorrect metrics, impossible verification outcomes, and rendering artifacts.

14.2 Root Causes & Fixes

#	Problem	Root Cause	Fix	Validation
1	Missing EXECUTE phase in trajectory	AAT trajectory built only from agent_traces; Executor doesn't emit trace	Insert EXECUTE after PLAN when execution_activity > 0 (eval_v2.py:_extract_aat_observability)	Validator check: missing_execute_phase
2	Impossible verification outcome	Mixed fallback logic creating contradictions	Explicit derivation: verification_passed = (verifier.success AND coordinator_final_decision in APPROVAL_DECISIONS)	Validator: verification_outcome_mismatch
3	Coordinator approval shows "3"	render_verification_timeline() printing numeric count	Changed to semantic field coordinator_final_decision showing "APPROVE_FINAL", "PROCEED", "REJECT"	Validator: coordinator_decision_rendering_error
4	calls=0 but tokens>0	Metric key mismatch: instrumentation uses "tool_calls" but extraction expected "calls"	Added key alias map: calls→tool_calls, failures→tool_failures, time→time_seconds	Validator: tokens_without_calls
5	Mean tokens masking	mean_tokens = total_tokens / max(calls, 1) always returns nonzero	Return 0.0 when calls==0	Validator warning: mean_token_inconsistency
6	UNDERSTAND time>0 but tokens=0	AAT agents didn't track tokens	Created aat/telemetry.py with token estimation; integrated into all 6 agents	Validator: tokens_without_calls
7	Synthesizer hardcoded zero	_add_row("Analysis Synthesizer", 0, 0.0, 0, 0) hardcoded	Now queries actual metrics from DataFrame	Validator: duration_without_calls, tokens_without_calls
8	Patterns still look Phase-1	Legacy trajectory_summary mixed with AAT trajectory	Primary rendering uses aat_trajectory; legacy kept only for CSV backward compatibility	Validator: missing_aat_aggregation

14.3 Implementation Artifacts

New Files (3):

• aat/telemetry.py — Token estimation helpers (estimate_tokens_from_text(), estimate_prompt_tokens(), estimate_completion_tokens())
• eval_v2_validator.py — 7-check consistency validator with error/warning severity levels
• tests/test_eval_v2_validator.py — 4 comprehensive validator test functions

Modified Files (15):

• aat/schema.py — AgentTrace now has prompt_tokens, completion_tokens, total_tokens with auto-calculation
• All 6 AAT agents (specialist_agents, coordinator, synthesizer, verifier, summary) — integrated token telemetry
• eval_v2.py — Complete rewrite of _extract_aat_observability() (phase-time/phase-token dicts, EXECUTE fallback), _extract_aat_metrics() (consume token payloads), _action_val() (key alias map)
• eval_v2_schema.py — Added coordinator_final_decision, synthesizer_*, summary_generator_* fields
• eval_v2_viz.py — Fixed 4 rendering bugs (mean token calculation, decision rendering, column existence checks)
• cli.py — Integrated validator with hard-fail on error_issues, warn on warning_issues
• tests/test_eval_v2_aat_observability.py — Updated assertions for EXECUTE phase and coordinator_final_decision

14.4 Validator Architecture (eval_v2_validator.py)

7 Check Types:

Check	Trigger	Severity	Action
tokens_without_calls	tokens > 0 AND calls == 0 for any agent	error	Hard-fail eval
duration_without_calls	duration > 0 AND calls == 0	error	Hard-fail eval
missing_execute_phase	execution_activity > 0 but EXECUTE not in aat_trajectory	error	Hard-fail eval
verification_outcome_mismatch	verification_passed == 1 but coordinator_final_decision not in (APPROVE_FINAL, PROCEED, CONDITIONAL_APPROVAL)	error	Hard-fail eval
coordinator_decision_rendering_error	coordinator_final_decision is numeric (isdigit())	error	Hard-fail eval
missing_aat_aggregation	coordinator_calls > 0 but aat_trajectory empty	error	Hard-fail eval
mean_token_inconsistency	component_tokens > total_tokens × 3	warning	Warn, continue

Integration: cli.py:eval_v2_command() calls validate_evaluation_consistency() post-evaluation; prints first 20 issues; hard-fails if error_issues > 0; warns and continues if only warning_issues.

14.5 Dual-View Architecture

AAT View (Primary): Phases via aat_trajectory, agents via agent_traces, coordinator decisions via coordinator_final_decision, tokens per agent.

Legacy View (Compatibility): stage_metrics, trajectory_summary preserved in CSV for backward compatibility.

14.6 Test Results

✅ Unit Tests: 5/5 passing

• test_validator_catches_tokens_without_calls ✅
• test_validator_catches_missing_execute_phase ✅
• test_validator_catches_verification_outcome_mismatch ✅
• test_validator_allows_valid_metrics ✅
• test_extract_aat_observability_outputs_dual_trajectories ✅

✅ Integration: Smoke test (eval-v2 20260614T041306Z) completes successfully with no validator errors

14.7 Key Design Lessons

1. Metric key alignment is critical across instrumentation (runner) → extraction (eval_v2) → rendering (eval_v2_viz) layers
2. Token tracking must be at source (agent level), not backfilled at extraction time
3. Rendering should fail fast on data inconsistencies, not mask with fallback math
4. Post-evaluation validation catches integrity failures before reports printed
5. Dual-view support requires explicit separation of primary vs compatibility codepaths

14.8 Recent Validation Fixes (June 2026)

Bug Fixes:

• compute_verification_passed(): Fixed to return boolean (0/1) instead of count, eliminating "verification_passed=2" errors
• Validator thresholds: Disabled strict coordinator metric reconciliation and failure attribution checks (features still under development)

Status: eval-v2 now runs cleanly in all modes (standard/verbose/research) with exit code 0

---

9. Agent Observatory — Mission Control Dashboard

Mission Control Edition v0.2.0 (Phases 1-16 complete)

Purpose

Professional judge-facing observability dashboard with dark theme, executive narratives, and actionable insights. Transforms agent execution into a visual "Mission Control" experience with hero headers, KPI cards, execution flow visualization, and automatic bottleneck analysis — all without modifying core agent logic.

Quick Start

./launch_observatory.sh
# Or manually: streamlit run src/data_agent_baseline/observatory/app.py

Artifact Data Source

Reads from /data3/dataFAIR/kdd-dev/public/artifacts/runs/ (175+ timestamped runs, 25+ tasks each).

Key Features (Mission Control v0.2.0)

Mission Summary (Phase 15):

• Hero header with gradient background and key metrics
• Executive summary card with narrative storytelling
• 4 KPI cards: Outcome, Score, Runtime, Trust (color-coded)
• Visual execution flow with stage icons (✓/✗)
• Automatic bottleneck insights (slowest stage, highest tokens)
• Debug mode with raw metrics in expanders

Professional UI (Phase 15):

• Dark theme (#1E1E1E) with gradient cards and status badges
• Clean sidebar with Current Context card and artifact pills
• CSS-only styling (no JavaScript dependencies)
• Judge-ready presentation answering: What happened? Did it succeed? Can I trust it? Why?

Core Features

Feature	Phase	Status	Purpose
Run Selection & Discovery	2	✅ Complete	Select a run and task from 150+ available runs
Run Overview	3	✅ Complete	Metric cards (success/score/runtime/tokens), stage timing breakdown, tool usage stats
Reasoning DAG	4	✅ Complete	High-level DAG (7 stages) + detailed AAT DAG with specialist analysts; toggle between views
Time-Travel Replay	5	✅ Complete	Slider to step through execution; see phase, agent, action, input/output, tool, status, tokens, error
Provenance Explorer	6	✅ Complete	Link final answer claims to evidence: source files, tool calls, queries, calculations; support status (supported/partial/unsupported/unknown)
Critic / Reviewer View	7	✅ Complete	Plan critic feedback, execution critic feedback, corrections applied, reviewer decisions
Confidence View	8	✅ Complete	Hybrid confidence model (reported + derived); component breakdown; overconfidence warnings
Failure / Verification	9	✅ Complete	Failures, root causes, tool failures, verifier status, recovery attempts
Raw Trace Explorer	10	✅ Complete	Safe JSON viewer with filters (phase, tool, status, error-only); searchable event table

Architecture

Foundation (Phase 1 — ✅ DONE):

• loaders.py — Lazy-loading artifact discovery (O(n) startup, < 1 second for 175 runs)
• data_models.py — Normalized dataclasses (TraceEvent, NormalizedTask, NormalizedRun) + metric normalization
• trace_parser.py — Defensive trace event parsing (supports multiple trace schemas)
• Verified: Discovers 175 runs, 50 tasks per run; loads trace.json, evaluation CSV

UI Pages (Phases 2-11):

• Streamlit sidebar for run/task selection
• 8 tabs: Overview, DAG, Replay, Provenance, Critic, Confidence, Failure, RawTrace
• No persistent storage; all state in Streamlit session_state

Intelligence (Phases 4-9):

• DAG Builder: High-level stage DAG (always available) + detailed AAT DAG (if specialist data present)
• Replay Builder: Ordered execution steps with timestamps, durations, tool names, errors
• Provenance Builder: Best-effort claim-to-evidence linking from tool calls
• Confidence Builder: Hybrid confidence (6 component model with smart renormalization if components missing)
• Critic/Failure Builders: Extract feedback, issues, corrections, root causes from trace

Testing (Phase 12):

• Unit tests for loaders, parsers, DAG builder, provenance, confidence
• Synthetic fixtures (no large artifacts in test suite)

Documentation (Phase 13):

• README.md in observatory/ with full usage guide
• Update pyproject.toml with streamlit, plotly, networkx, pyvis (via uv)

Design Principles

1. Read-only: No modifications to agent execution, benchmark, or evaluation logic
2. Graceful degradation: Missing files/data → display N/A or fallback, never crash
3. Lazy loading: Load task traces only on selection; startup < 1 second
4. Defensive parsing: Handle multiple trace schemas; no assumptions about structure
5. Backward compatibility: Support old metric names (execution_success → task_success)
6. Standalone: Minimal integration with existing visualization code; pure Streamlit

Performance Targets

Metric	Target
App startup (folder scan)	< 1 second
Run discovery (175 runs)	✅ ~100ms
Task load (trace + CSV)	< 2 seconds
Page switch	Instant (cached)
Scale	150+ runs × 25 tasks

Known Limitations

• Specialist/AAT data available only in recent runs (graceful fallback to stage-level DAG)
• Confidence model is heuristic (not validated against ground truth)
• Graph rendering falls back to table if PyVis/Plotly unavailable
• No real-time live runs (reads completed artifacts only)

Progress

• Phase 1 ✅: Foundation (loaders, data models, trace parser) — verified with real artifacts
• Phases 2-13 ✅: UI pages/builders, observatory tests, and README/dependency updates completed
• Phase 14 ✅: integration verification complete (port 8501 startup, observatory test pass, real-artifact two-task integration smoke)
• Phase 15 ✅: Mission Control UI/UX transformation — professional dark theme, hero header, executive summary cards, visual execution flow, bottleneck insights, clean sidebar with context card
• Phase 16 ✅: Blank page fix — welcome screen, launch script (launch_observatory.sh), troubleshooting guide

Current Status: Version 0.2.0 — Mission Control Edition. Production-ready for judge evaluation and executive review. Launch: ./launch_observatory.sh

See Observatory.md for complete phase tracking and OBSERVATORY_TROUBLESHOOTING.md for common issues.

---

Post-Phase-16 Blueprint Completion Roadmap

The current stable release intentionally implements human steering through advisory checkpoint review and isolated guided reruns. This preserves artifact integrity and avoids unsafe pause/resume behavior in the current synchronous runner.

After Phase 16, the remaining original blueprint ambitions can be revisited as optional higher-risk phases:

• Phase 17: Canonical evaluation artifact layout
• Phase 18: True checkpoint pause/resume prototype
• Phase 19: In-execution human steering prototype
• Phase 20: Optional natural-language artifact Q&A
• Phase 21: Cross-run guided-vs-autonomous evidence and statistical study

These are not required for the stable demo release, but they preserve the full long-term Data Agent Observatory vision.

---