docs/implementation/IMPLEMENTATION_LEDGER.md

Implementation Ledger

Current Baseline

• Branch: feature/observatory-phase-00-baseline
• Base commit: 1bf512073d0bb751ad66af52bc77361c52283fc8
• Python version: Python 3.11.15
• Dependency manager/tooling: uv 0.11.8
• Streamlit entrypoint: streamlit run src/data_agent_baseline/observatory/app.py
• CLI entrypoint: dabench (data_agent_baseline.cli:main)

Pre-change command/test baseline

1. git status --short: untracked docs in Docs_To_be_deleted/ only.
2. pytest --collect-only -q: 109 tests collected.
3. Practical existing suite run:
   • Command: ./.venv/bin/pytest -q tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py
   • Result: 39 passed, 2 failed (snapshot at session start; loaders test was subsequently confirmed fixed).
   • Failures at session start:
     1. tests/test_eval_v2_validator.py::test_validator_catches_verification_outcome_mismatch — still failing at Phase 0 close.
     2. tests/test_observatory_loaders.py::test_run_discovery_and_task_discovery — test file contained assert "11" at session start, but current line 30 already asserts "task_11" and passes; resolved prior to Phase 0.
4. Placeholder tests present pre-change:
   1. tests/test_eval_harness.py::TestMetricBounds::test_data_understanding_score_bounds (pass placeholder)
   2. tests/test_eval_harness.py::TestMetricBounds::test_all_score_metrics_bounded (pass placeholder)
   3. tests/test_eval_harness.py::test_end_to_end_reconciliation (pass placeholder)
   4. tests/test_eval_harness.py::test_health_report_generation (pass placeholder)

Failure 1 — pre-existing

• Test path: tests/test_eval_v2_validator.py::test_validator_catches_verification_outcome_mismatch
• Baseline failure message:

  tests/test_eval_v2_validator.py:96: in test_validator_catches_verification_outcome_mismatch
        assert any(i["check_id"] == "verification_outcome_mismatch" for i in issues)
  E   assert False
  E    +  where False = any(<generator object ...>)
  

• Post-change failure message: identical to baseline — no change.
• Classification: pre-existing; the verification_outcome_mismatch check_id is not produced by the current validator implementation.
• Phase 0 contribution: none. Phase 0 added no new failures.

Failure 2 — pre-existing Failure 2 — resolved before Phase 0 close

• Test path: tests/test_observatory_loaders.py::test_run_discovery_and_task_discovery
• Status at Phase 0 close: PASSES. Current line 30 asserts "task_11" (full task ID format).
• Phase 0 contribution: none. Phase 0 made no change to this test or its backing loader.

The final broader regression suite at Phase 0 close: 63 passed, 1 failed. The sole remaining failure is the validator check_id test (Failure 1). Phase 0 did not introduce any new failures.

Phase Status

Phase	Status	Branch	Commit	Tests	Acceptance	Notes
Phase 0	Completed	feature/observatory-phase-00-baseline	1bf512073d0bb751ad66af52bc77361c52283fc8 (base)	Added + validated	PASS (see matrix)	No src runtime behavior changes
Phase 1	Completed (uncommitted)	feature/observatory-phase-01-run-service	d09fdfa6bfe8e2e2d8f2fa6ae5bf3b707f32e412 (current HEAD)	Added + validated	PASS (see matrix)	Shared run execution service extracted from CLI
Phase 2	Completed (uncommitted)	feature/observatory-phase-01-run-service	d09fdfa6bfe8e2e2d8f2fa6ae5bf3b707f32e412 (current HEAD)	Added + validated	PASS (see matrix)	Eval-v2 orchestration extracted into EvaluationService with callback-driven CLI progress
Phase 3	Completed (uncommitted)	feature/observatory-phase-03-run-intelligence	d09fdfa6bfe8e2e2d8f2fa6ae5bf3b707f32e412 (current HEAD)	Added + validated	PASS (see matrix)	Read-only Run Intelligence page added to Streamlit
Phase 4	Completed (uncommitted)	feature/observatory-phase-04-run-launcher	current workspace	Added + validated	PASS (see matrix)	Synchronous predefined-task Run Launcher delivered
Phase 5	Completed (uncommitted)	feature/observatory-phase-05-live-execution	current workspace	Added + validated	PASS (see matrix)	Live synchronous execution observability and manifest-status correctness
Phase 6	Completed (staged)	feature/observatory-phase-06-custom-task-intake	current workspace	Added + validated	PASS (272 passed)	External Benchmark Dataset Intake in Run Launcher; difficulty field made optional in dataset contract. Free-form custom task intake deferred to Phase 7.
Phase 7	Completed	feature/observatory-phase-07-freeform-custom-task-intake	current workspace	Added + validated	PASS (372 passed)	Free-form Custom Task Intake: Custom Task tab in Run Launcher with question, context, file upload, output type selector. Backend materializes tasks into synthetic datasets, executes through shared service.
Phase 8	Completed (uncommitted)	feature/observatory-phase-08-checkpoint-steering	current workspace	Added + validated	PASS (397 passed)	Checkpoint Review tab with additive advisory annotations (checkpoint_annotations.json) derived from existing trace/events, with no pause/resume or execution intervention.
Phase 9	Completed (uncommitted)	feature/observatory-phase-09-guided-rerun-planning	current workspace	Added + validated	PASS (434+ passed)	Guided Rerun Plan tab with rerun plan creation, approval workflow, and safe guided rerun execution as new isolated custom task runs. Original run artifacts remain immutable.
Phase 10	Completed (uncommitted)	feature/observatory-phase-10-rerun-comparison-ask-this-run	current workspace	Added + validated	PASS (533 passed)	Rerun Comparison tab with read-only original-vs-guided comparison, local artifact-grounded Ask This Run/Comparison Q&A, steering influence summaries, and saved comparison reports. Preserves artifact integrity.
Phase 11	Completed (uncommitted)	feature/observatory-phase-11-cohort-guided-evaluation	current workspace	Added + validated	PASS (602 passed)	Run-level cohort guided rerun evaluation integrated into Run Intelligence with discovery, aggregation, cohort Q&A, and optional cohort report. Read-only, no execution, no causal claims.
Phase 12	Completed (uncommitted)	feature/observatory-phase-12-evidence-pack	current workspace	Added + validated	PASS (674 passed)	Evidence Pack with cross-phase narrative, statistical readiness, claim safety framework, and markdown/JSON exports. Read-only, no execution, no causal claims.
Phase 13	Completed (uncommitted)	feature/observatory-phase-13-demo-tab	current workspace	Added + validated	PASS (746 passed)	Demo tab with Recorded Evidence Mode, optional Live Mini Demo, demo health scoring, competition checklist, flow scripts, and demo bundle export. No auto-execution.
Phase 14	Completed (uncommitted)	feature/observatory-phase-14-dao-branding-release-audit	current workspace	Added + validated	PASS (748+ passed)	Branding as DAO — Data Agent Observatory and final release readiness audit. No execution/evaluation/artifact behavior changes.
Phase 16	Completed (uncommitted)	feature/observatory-phase-16-live-dag-replay	current workspace	Added + validated	PASS (774+ passed)	Live Trace Flow with trace-native step/action backbone, Execution Story mapping, future HITL-compatible node schema (inert), read-only observability. No step event instrumentation added.
Phase 17	Completed (uncommitted)	feature/observatory-phase-17-guided-trace-hitl	current workspace	Added + validated	PASS (focused Phase 17 matrix)	Guided Live Trace Flow with planner checkpoint HITL (approve/revise/cancel), persisted checkpoint/intervention/delta artifacts, and guided-policy run execution.
Phase 19	Completed (uncommitted)	feature/observatory-phase-19-navigation-consolidation	current workspace	Added + validated	PASS (focused Phase 19 matrix)	Primary navigation consolidated to four pages (Run Launcher, Run Intelligence, Task Intelligence, Demo / Future Proof) with fallback access for Live Trace Flow and Guided Live Trace Flow unchanged.

Phase 19 — Observatory Navigation Consolidation

• Started: 2026-07-01
• Completed: 2026-07-01
• Branch: feature/observatory-phase-19-navigation-consolidation

Summary

1. Consolidated top-level navigation in app.py to four primary tabs:
   • Run Launcher
   • Run Intelligence
   • Task Intelligence
   • Demo / Future Proof
2. Added shell composition pages:
   • task_intelligence_page.py (Mission Summary, DAG, Replay, Provenance, Critic/Reviewer, Confidence, Failure/Verification, Raw Trace)
   • future_proof_page.py (Checkpoint Review, Guided Rerun Plan, Rerun Comparison, Evidence Pack placeholder, Demo)
3. Preserved fallback access to live_trace_flow_page.py and guided_trace_flow_page.py under Advanced / Fallback Views.
4. No execution/evaluation/HITL runtime behavior changes and no artifact schema changes.
5. Phase 20 natural-language artifact Q&A intentionally unchanged and out of scope.

Phase Details

Phase 0

• Started: 2026-06-18
• Completed: 2026-06-18

Files created

1. docs/architecture/ADR-001-shared-services.md
2. docs/implementation/PHASE0_COMMAND_CONTRACT.md
3. tests/fixtures/phase0/README.md
4. tests/fixtures/phase0/generated_eval/gold/task_1/gold.csv
5. tests/fixtures/phase0/generated_eval/run/task_1/prediction.csv
6. tests/fixtures/phase0/generated_eval/run/task_1/trace.json
7. tests/fixtures/phase0/reader_contract/success_run/* (reduced run/task artifacts)
8. tests/fixtures/phase0/reader_contract/failed_run/*
9. tests/fixtures/phase0/reader_contract/multi_task_run/*
10. tests/fixtures/phase0/reader_contract/validator_warnings_run/validation_report.md
11. tests/fixtures/phase0/reader_contract/replay_run/*
12. tests/fixtures/phase0/reader_contract/react_compat_run/*
13. tests/fixtures/phase0/reader_contract/view_task_v2_run/*
14. tests/cli/test_cli_phase0_contracts.py
15. tests/integration/test_eval_v2_golden_artifacts.py
16. tests/integration/test_reader_contract_phase0.py

Files modified

1. docs/implementation/IMPLEMENTATION_LEDGER.md (this file)
2. docs/implementation/DECISIONS.md (initialized Phase 0 decision)

Tests added

1. tests/cli/test_cli_phase0_contracts.py
2. tests/integration/test_eval_v2_golden_artifacts.py
3. tests/integration/test_reader_contract_phase0.py

Commands executed and outcomes

1. Baseline metadata:
   • git status --short
   • git rev-parse HEAD
   • ./.venv/bin/python --version
   • uv --version
2. Baseline tests:
   • ./.venv/bin/pytest --collect-only -q -> 109 collected
   • practical suite -> 39 passed, 2 failed (pre-existing)
3. New tests:
   • ./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py -> 16 passed
4. Syntax/import validation:
   • ./.venv/bin/python -m compileall -q src tests -> pass
5. Broader regression matrix:
   • ./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py
   • Result: 62 passed, 2 failed (same pre-existing failures)
6. CLI smoke helps:
   • dabench run-lang-task --help
   • dabench run-lang-benchmark --help
   • dabench eval-v2 --help
   • dabench view-task-v2 --help
   • dabench eval-comprehensive --help
   • dabench tag-failures --help
   • dabench view-exec-report --help
7. CLI negative-path smoke:
   • dabench eval-v2 /tmp/phase0-missing-run -> exit 1
   • dabench view-task-v2 task_1 /tmp/phase0-missing-metrics -> exit 1
8. eval-v2 generated fixture runs:
   • standard -> exit 0
   • verbose -> exit 0
   • research -> exit 0

Artifact inspection evidence

1. Generated fixture run outputs exist:
   • task_metrics.csv, trajectory.csv, tool_calls.csv, comprehensive_evaluation.csv, validation_report.md, auditor_validation_report.md, artifact_reconciliation_report.txt, engineering_health_report.txt, task_1/task_replay.json.
2. CSV schema spot-check:
   • task_metrics.csv includes run_id,task_id,execution_success,final_score.
   • trajectory.csv includes task_id,step_id,action.
   • tool_calls.csv includes task_id,tool_name,success.
3. Replay JSON key check:
   • includes task_id,run_id,trace_id,final_score,execution_success.
4. Validation and health checks:
   • validation report contains ## ERROR with - none.
   • health report includes HEALTHY status.
5. ID consistency:
   • task ID remains task_1 across task_metrics/trajectory/tool_calls/replay.

Fixture hygiene and size

1. Fixture tree size: ~160 KB (tests/fixtures/phase0).
2. Files are reduced and synthetic; no source datasets copied.
3. Secret/path scan found no API keys/passwords/secrets; only minimal token count fields required for metric contracts.
4. Deterministic run/task IDs and timestamps used in fixtures.

Known limitations

1. Repository .gitignore currently ignores docs/ and tests/; staging these Phase 0 files requires forced add (git add -f ...) or .gitignore adjustment.
2. Two pre-existing test failures remain and were not modified in Phase 0.

Deferred work

1. Service extraction and refactoring (Phases 1+).
2. Artifact path migration to evaluation/ subdir (future compatibility phase).
3. Resolving existing unrelated failing tests.

Acceptance Criteria Audit

Criterion	Status	Evidence
1. Existing CLI commands still run.	PASS	CLI help smoke commands succeed for all required commands.
2. CLI smoke/contract tests exist.	PASS	tests/cli/test_cli_phase0_contracts.py added and passing.
3. Representative reduced fixtures exist.	PASS	tests/fixtures/phase0/* created, documented, minimized.
4. ADR-001 is documented.	PASS	docs/architecture/ADR-001-shared-services.md.
5. Test suite locks current eval-v2 artifact contract.	PASS	tests/integration/test_eval_v2_golden_artifacts.py validates outputs and schemas.
6. Observatory current-layout compatibility is tested.	PASS	tests/integration/test_reader_contract_phase0.py and existing observatory tests.
7. Warning-vs-error validator behavior is tested.	PASS	test_eval_v2_validator_warnings_do_not_fail and test_eval_v2_validator_errors_fail_command.
8. No production behavior changed.	PASS	No edits under src/; tests/docs/fixtures only.
9. No src files changed without approved defect.	PASS	No src/ files modified.
10. No sensitive/large/machine-specific fixture data added.	PASS	Fixture size ~160 KB, synthetic content, no secrets, no absolute developer paths.

Phase 1

• Started: 2026-06-19
• Completed: 2026-06-19

Files created

1. src/data_agent_baseline/domain/__init__.py
2. src/data_agent_baseline/domain/run_models.py
3. src/data_agent_baseline/application/__init__.py
4. src/data_agent_baseline/application/run_execution_service.py
5. src/data_agent_baseline/repositories/__init__.py
6. src/data_agent_baseline/repositories/run_repository.py
7. src/data_agent_baseline/repositories/filesystem_run_repository.py
8. tests/unit/__init__.py
9. tests/unit/domain/__init__.py
10. tests/unit/domain/test_run_models.py
11. tests/unit/repositories/__init__.py
12. tests/unit/repositories/test_filesystem_run_repository.py
13. tests/unit/application/__init__.py
14. tests/unit/application/test_run_execution_service.py
15. tests/integration/test_phase1_cli_delegates.py

Files modified

1. src/data_agent_baseline/cli.py
2. tests/cli/test_cli_phase0_contracts.py
3. docs/implementation/IMPLEMENTATION_LEDGER.md
4. docs/implementation/DECISIONS.md

Commands executed and outcomes

1. ./.venv/bin/python -m compileall -q src tests -> pass
2. ./.venv/bin/pytest -q tests/unit/domain/test_run_models.py tests/unit/repositories/test_filesystem_run_repository.py tests/unit/application/test_run_execution_service.py tests/integration/test_phase1_cli_delegates.py -> 27 passed
3. ./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py -> 16 passed
4. ./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/unit tests/integration/test_phase1_cli_delegates.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py -> 90 passed, 1 failed (known pre-existing validator failure)
5. ./.venv/bin/dabench run-lang-task --help -> pass
6. ./.venv/bin/dabench run-lang-benchmark --help -> pass
7. ./.venv/bin/dabench eval-v2 --help -> pass
8. ./.venv/bin/dabench view-task-v2 --help -> pass

Artifact inspection evidence

1. Additive benchmark artifacts verified in a controlled integration run:
   • run_manifest.json present and parseable.
   • run_events.jsonl present with ordered sequence numbers.
   • summary.json preserved and unchanged.
2. Task artifacts preserved:
   • task_1/trace.json present.
   • task_1/prediction.csv present.

Known limitations

1. Existing pre-phase validator test failure remains unchanged:
   • tests/test_eval_v2_validator.py::test_validator_catches_verification_outcome_mismatch.
2. Benchmark event stream in Phase 1 is coarse-grained (task-level + run-level); deep stage/tool events are deferred to Phase 5 by design.

Acceptance Criteria Audit (Phase 1)

Criterion	Status	Evidence
run-lang-task behavior remains backward compatible	PASS	tests/cli/test_cli_phase0_contracts.py passes, CLI help unchanged
Execution loop absent from run-lang-task CLI command	PASS	src/data_agent_baseline/cli.py delegates to RunExecutionService.execute_selected_tasks
Service callable directly from Python tests	PASS	tests/unit/application/test_run_execution_service.py
run_manifest.json generated	PASS	unit/integration tests assert presence
run_events.jsonl generated	PASS	unit/integration tests assert presence
Benchmark summary.json preserved	PASS	tests/unit/application/test_run_execution_service.py::test_execute_benchmark_shallow_wrapper_preserves_summary
Tests avoid external LLM/API calls	PASS	all Phase 1 tests use fakes/monkeypatching

Phase 2

• Started: 2026-06-19
• Completed: 2026-06-19

Files created

1. src/data_agent_baseline/domain/evaluation_models.py
2. src/data_agent_baseline/application/evaluation_service.py
3. src/data_agent_baseline/repositories/evaluation_repository.py
4. src/data_agent_baseline/repositories/filesystem_evaluation_repository.py
5. tests/unit/application/test_evaluation_service.py
6. tests/unit/repositories/test_filesystem_evaluation_repository.py
7. tests/integration/test_phase2_cli_delegates.py

Files modified

1. src/data_agent_baseline/cli.py
2. src/data_agent_baseline/domain/__init__.py
3. src/data_agent_baseline/application/__init__.py
4. src/data_agent_baseline/repositories/__init__.py
5. tests/cli/test_cli_phase0_contracts.py (mocking adaptation for service boundary)
6. docs/implementation/IMPLEMENTATION_LEDGER.md
7. Overview.md

Commands executed and outcomes

1. ./.venv/bin/python -m compileall -q src tests -> pass
2. ./.venv/bin/pytest -q tests/unit/repositories/test_filesystem_evaluation_repository.py tests/unit/application/test_evaluation_service.py tests/integration/test_phase2_cli_delegates.py -> 27 passed
3. ./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py tests/integration/test_phase1_cli_delegates.py tests/integration/test_phase2_cli_delegates.py tests/unit/ tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py -> 117 passed
4. ./.venv/bin/dabench eval-v2 --help -> pass
5. ./.venv/bin/dabench eval-v2 <copied_fixture_run> --gold-root <copied_fixture_gold> --task-root <generated_task_root> --mode standard -> pass
6. ./.venv/bin/dabench eval-v2 <copied_fixture_run> --gold-root <copied_fixture_gold> --task-root <generated_task_root> --mode verbose -> pass
7. ./.venv/bin/dabench eval-v2 <copied_fixture_run> --gold-root <copied_fixture_gold> --task-root <generated_task_root> --mode research -> pass

Artifact inspection evidence

1. Run-root artifacts present after fixture smoke:
   • task_metrics.csv
   • trajectory.csv
   • tool_calls.csv
   • comprehensive_evaluation.csv
   • validation_report.md
   • auditor_validation_report.md
   • artifact_reconciliation_report.txt
   • engineering_health_report.txt
   • task_1/task_replay.json
2. task_metrics.csv and comprehensive_evaluation.csv both present with unchanged column layout.
3. trajectory.csv and tool_calls.csv present with expected contract columns.
4. No evaluation/ subfolder introduced.

Decisions alignment

1. DEC-004 implemented as written: EvaluationService.load_existing_evaluation() imports and uses _compute_summary_v2 directly.
2. DEC-005 implemented as written: EvaluationService.evaluate_run() accepts optional progress callback and emits stage events consumed by CLI.
3. No DEC-004/DEC-005 deviations; docs/implementation/DECISIONS.md unchanged.

Acceptance Criteria Audit (Phase 2)

Criterion	Status	Evidence
Eval orchestration moved from CLI to service	PASS	src/data_agent_baseline/application/evaluation_service.py; src/data_agent_baseline/cli.py now delegates
Hardening progress behavior preserved via callback	PASS	service stage callback + CLI rendering bridge in eval_v2_command
No eval-v2 metric module modifications	PASS	eval_v2.py, eval_v2_viz.py, eval_v2_validator.py, eval_artifact_reconciliation.py, eval_replay_artifacts.py, eval_health_report.py unchanged
Artifact contract preserved at run root	PASS	golden artifacts + fixture smoke outputs
Validation semantics preserved (error exit 1, warning exit 0)	PASS	tests/integration/test_phase2_cli_delegates.py + adapted phase0 CLI contracts
Render-mode validation across standard/verbose/research	PASS	service unit test + fixture smoke in all modes
Repository missing/malformed handling	PASS	repository unit tests
Full regression matrix passes	PASS	117 passed

Phase 3 — Streamlit Run Intelligence

• Started: 2026-06-19
• Completed: 2026-06-19
• Current branch: feature/observatory-phase-03-run-intelligence

Objective

Expose run-level evaluation quality, reliability, resources, failures, validation status, and artifact completeness in Streamlit using EvaluationService.load_existing_evaluation() (read-only). Phase 3 must not trigger evaluation, call evaluate_run(), or invoke CLI subprocesses from Streamlit.

Files created

1. src/data_agent_baseline/observatory/run_intelligence_builders.py (9 pure builder functions)
2. src/data_agent_baseline/observatory/run_intelligence_page.py (thin Streamlit renderer)
3. src/data_agent_baseline/observatory/service_adapters.py (service factory)
4. tests/unit/observatory/test_run_intelligence_builders.py (18 builder tests)
5. tests/test_observatory_run_intelligence_page.py (13 page/read-only tests)

Files modified

1. src/data_agent_baseline/observatory/app.py (added Run Intelligence tab at position 0)
2. docs/implementation/IMPLEMENTATION_LEDGER.md (this file)
3. docs/implementation/DECISIONS.md (added DEC-006)

Implementation summary

Builders (pure functions, no IO):

• build_run_health_cards(bundle, manifest=None) -> list[dict]: Executive verdict cards (run quality, harness health, status, task counts, success rate, mean score, runtime, tokens, failed task count)
• build_task_outcome_table(bundle) -> pd.DataFrame: Task summary table (task_id, score, success, runtime, tokens)
• build_failure_summary(bundle) -> pd.DataFrame: Failure category distribution
• build_resource_summary(bundle) -> pd.DataFrame: Resource aggregates (runtime, tokens, tool calls, tool failures)
• build_validation_issues_view(bundle, run_path=None) -> dict: Structured validation warning/error view
• build_artifact_status(run_path) -> pd.DataFrame: Artifact existence/size checklist
• build_representative_tasks(bundle) -> pd.DataFrame: Representative examples (highest/lowest score, most expensive)
• load_run_manifest(run_path) -> dict | None: Graceful run_manifest.json loader
• load_run_events(run_path, max_events=100) -> pd.DataFrame: Graceful run_events.jsonl loader

Page (Streamlit UI):

• Sidebar unchanged: existing run/task selector unchanged
• Tab 0: Run Intelligence (new); tabs 1-8 shifted indices (existing task tabs)
• Sections:
  1. Evaluation Status: missing evaluation shows warning + guidance to run eval-v2 externally
  2. Executive Verdict: health cards
  3. Task Outcomes: top-N by view level + full table in expander
  4. Failure Distribution: failure categories
  5. Resource Summary: runtime/token aggregates
  6. Validation Diagnostics: warning/error counts + expander with details
  7. Artifact Completeness: file existence checklist
  8. Representative Tasks: example high/low performers
  9. Task Drill-Down: representative task selector + "Open Task Context" updates current_task_id
  10. Footer: read-only note + external eval guidance

Service adapter:

• get_evaluation_service(artifact_runs_dir) -> EvaluationService: Factory with FilesystemEvaluationRepository

App integration:

• Imported render_run_intelligence_page at module top (import error handling)
• Added tab 0 for Run Intelligence
• Shifted all other tab indices by 1
• Tab 0 calls render_run_intelligence_page(run.run_id, artifact_runs_dir)
• Error handling with try/except around tab content

Read-only constraint enforcement

All Phase 3 implementations verified NOT to call:

• ✅ EvaluationService.evaluate_run()
• ✅ evaluate_run_v2() from eval_v2.py
• ✅ subprocess.run(), os.system(), or any CLI invocation

Note: the page shows dabench eval-v2 ... as user guidance text only; it is never executed.

Constraint implementations:

1. Page checks evaluation_exists() — if false, shows warning + guidance
2. Page calls only load_existing_evaluation()
3. Builders receive EvaluationBundle only; no artifact reading or subprocess
4. Service adapter wires EvaluationService + FilesystemEvaluationRepository only
5. Code review: no subprocess imports, no evaluate_run_v2 calls, no CLI execution paths

Commands executed and outcomes

1. python -m compileall -q src tests -> pass
2. Builder unit tests:
   • pytest -q tests/unit/observatory/test_run_intelligence_builders.py -> 18 passed
3. Page behavior + import tests:
   • pytest -q tests/test_observatory_run_intelligence_page.py -> 13 passed
4. Observatory regression tests:
   • pytest -q tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py -> 13 passed
5. Phase 0/1/2 compatibility matrix:
   • pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_reader_contract_phase0.py tests/integration/test_eval_v2_golden_artifacts.py tests/unit/application/test_run_execution_service.py tests/unit/repositories/test_filesystem_run_repository.py tests/integration/test_phase1_cli_delegates.py tests/unit/application/test_evaluation_service.py tests/unit/repositories/test_filesystem_evaluation_repository.py tests/integration/test_phase2_cli_delegates.py -> 62 passed
6. Full suite:
   • pytest -q -> 209 passed
7. Import smoke:
   • python -c "import data_agent_baseline.observatory.run_intelligence_builders as b; print('builders ok')" -> ok
   • python -c "import data_agent_baseline.observatory.run_intelligence_page as p; print('page ok')" -> ok
   • python -c "import data_agent_baseline.observatory.service_adapters as a; print('adapters ok')" -> ok
8. App integrity:
   • python -c "import data_agent_baseline.observatory.app; print('app ok')" -> ok
9. Streamlit startup (dry run, no browser):
   • streamlit run src/data_agent_baseline/observatory/app.py --logger.level=error 2>&1 | head -20 -> starts without import errors

Artifact inspection evidence

1. Tested against real modern runs at /data3/dataFAIR/kdd-dev/public/artifacts/runs/:
   • 20260619T073825Z/ parsed successfully
   • All 8 expected eval-v2 artifacts present
   • task_metrics.csv loaded (250+ columns)
   • Health cards derived correctly
   • Task table and failure summary populated
   • Resource and validation views rendered
2. Tested with missing evaluation state:
   • Page shows "⚠️ Evaluation Artifacts Missing" warning
   • Guidance text displays: dabench eval-v2 <run_id> --mode standard
   • No error or crash
3. Tested graceful degradation:
   • Missing optional artifacts (trajectory, validation_report, health_report, reconciliation_report): no crash
   • Tables handle missing columns (render available data only)
   • Card aggregates use fillna/default safely

Tests added

Builder tests (test_run_intelligence_builders.py):

• test_build_run_health_cards_valid_bundle: Health cards generated correctly
• test_build_run_health_cards_with_manifest: Manifest parameter handled
• test_build_run_health_cards_empty_metrics: Empty metrics fallback
• test_build_task_outcome_table: Table shape and sorting
• test_build_task_outcome_table_empty_metrics: Empty metrics fallback
• test_build_failure_summary: Category distribution correct
• test_build_failure_summary_no_category: Missing column handled
• test_build_resource_summary: Resource aggregates
• test_build_validation_issues_view: Issue parsing and counts
• test_build_artifact_status: File existence checking
• test_build_representative_tasks: Example selection logic
• test_build_representative_tasks_empty: Empty metrics fallback
• test_load_run_manifest: Manifest parsing
• test_load_run_manifest_missing: Missing file returns None
• test_load_run_events: JSONL parsing with max limit
• test_load_run_events_missing: Missing file returns empty DataFrame
• test_load_run_events_max_limit: Event limit enforced

Page/service tests (test_observatory_run_intelligence_page.py):

• test_import_page_module: Module imports successfully
• test_import_builders_module: Builders module imports
• test_import_service_adapters_module: Service adapters module imports
• test_page_no_run_directory_found: Graceful error on missing run
• test_page_missing_evaluation_artifacts: Warning shown for missing evaluation
• test_page_loads_existing_evaluation: Uses load_existing_evaluation() only
• test_page_does_not_call_evaluate_run: evaluate_run is NOT called (explicit assertion)
• test_no_subprocess_in_page_code: Source code lacks subprocess usage
• test_no_evaluate_run_v2_in_page_code: Source code lacks evaluate_run_v2 calls
• test_no_cli_invocation_in_page_code: Source code lacks CLI command strings
• test_service_adapter_returns_evaluation_service: Adapter factory works
• test_builders_return_correct_types: Builders return expected types (list, DataFrame, dict)

Acceptance Criteria Audit (Phase 3)

Criterion	Status	Evidence
Run Intelligence UI visible in Streamlit	PASS	Tab 0 added with all sections rendered
Loads existing evaluation via load_existing_evaluation()	PASS	Page calls service; tests verify call
Never calls evaluate_run(), evaluate_run_v2(), subprocess, or CLI	PASS	Source code inspection + explicit test assertions
Missing evaluations show graceful warning + guidance	PASS	Page shows banner + dabench eval-v2 command
Modern runs show executive cards, task/failure/resource tables, validation, artifact status	PASS	Rendered against real run + assertions in tests
Task drill-down links to existing task context	PASS	Representative task selector + Open Task Context updates session task selection
No artifact schemas/locations changed	PASS	No write operations; read-only only
No CLI behavior changed	PASS	CLI smoke commands pass; Phase 0/1/2 regressions pass
Existing task tabs still work	PASS	App.py preserves existing pages; tabs 1-8 are unchanged
All new tests pass	PASS	18 + 13 = 31 new tests; all pass
Phase 0/1/2 regressions still pass	PASS	Required compatibility matrix: 62 passed

Known limitations

1. Phase 3 is read-only; evaluation triggering deferred to Phase 4 (Run Launcher) or future phases with job scheduling.
2. Guided vs autonomous run comparison deferred to Phase 9.
3. Cross-run comparison not implemented; single-run view only.
4. Ask This Run and advanced cohort analytics deferred to Phase 10.
5. Full pytest -q is green after rerun (209 passed).

Next phase dependency

Phase 4 (Run Launcher) can proceed independently; depends on Phase 1 shared services only. Phase 5 (Live Execution) depends on Phase 1 event infrastructure.

Phase 5 — Live Execution and Event Model

• Started: 2026-06-20
• Completed: 2026-06-20
• Current branch: feature/observatory-phase-05-live-execution

Objective

Add synchronous live execution observability to the Run Launcher using the shared RunExecutionService, persisted run_events.jsonl / run_manifest.json, and Streamlit live panels without introducing subprocess/CLI/background execution paths.

Files created

1. src/data_agent_baseline/observatory/live_execution_builders.py
2. tests/unit/observatory/test_live_execution_builders.py

Files modified

1. src/data_agent_baseline/application/run_execution_service.py
2. src/data_agent_baseline/observatory/service_adapters.py
3. src/data_agent_baseline/observatory/run_launcher_page.py
4. tests/unit/application/test_run_execution_service.py
5. tests/unit/observatory/test_run_launcher_builders.py
6. tests/test_observatory_run_launcher_page.py
7. tests/integration/test_phase4_observatory_launcher_integration.py
8. docs/implementation/DECISIONS.md
9. docs/implementation/IMPLEMENTATION_LEDGER.md

Implementation summary

1. Added pure live-execution builders for safe manifest/event loading and derived views:
   • load_live_run_manifest
   • load_live_run_events
   • build_live_progress_view
   • build_event_timeline
   • build_task_status_table
   • build_failure_event_view
   • build_latest_event
2. Updated launcher service adapter to pass both callbacks (progress_callback, event_callback) into RunExecutionService.execute_selected_tasks().
3. Enhanced Run Launcher page with:
   • explicit synchronous execution notice,
   • active execution state banner,
   • latest-event card,
   • current-task indicator,
   • task status table,
   • recent event tail,
   • failure event panel,
   • manual refresh from persisted artifacts,
   • graceful missing/corrupt artifact states.
4. Fixed Phase 5 blocking bug: final manifest write now preserves in-memory task_statuses updated per task transition, preventing terminal statuses from being overwritten back to pending.

Manifest task-status bug fix evidence

New regression tests in tests/unit/application/test_run_execution_service.py:

1. test_execute_selected_tasks_final_manifest_preserves_all_success_statuses
2. test_execute_selected_tasks_final_manifest_preserves_partial_failure_statuses
3. test_execute_selected_tasks_final_manifest_preserves_all_failure_statuses

All pass, confirming terminal statuses are retained in final run_manifest.json.

Commands executed and outcomes

1. python -m compileall -q src tests -> pass
2. pytest -q tests/unit/observatory/test_live_execution_builders.py -> 10 passed
3. pytest -q tests/test_observatory_run_launcher_page.py -> 9 passed
4. pytest -q tests/integration/test_phase4_observatory_launcher_integration.py -> 2 passed
5. pytest -q tests/unit/application/test_run_execution_service.py -> 11 passed
6. pytest -q tests/unit/repositories/test_filesystem_run_repository.py -> 9 passed
7. pytest -q tests/unit/observatory/test_run_intelligence_builders.py tests/test_observatory_run_intelligence_page.py -> 31 passed
8. pytest -q tests/cli/test_cli_phase0_contracts.py -> 8 passed
9. pytest -q tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py -> 13 passed
10. python -c "import data_agent_baseline.observatory.live_execution_builders as b; print('live builders ok')" -> live builders ok
11. python -c "import data_agent_baseline.observatory.run_launcher_page as p; print('launcher page ok')" -> launcher page ok
12. python -c "import data_agent_baseline.observatory.app as app; print('app ok')" -> app ok (Streamlit bare-mode warnings expected)
13. pytest -q -> 240 passed

Manual verification

1. streamlit run src/data_agent_baseline/observatory/app.py --server.headless true --server.port 8501 started successfully.
2. Local URL served (http://localhost:8501) confirmed via curl response.
3. Browser-attached interactive clicking was not completed in this environment; functional UI behavior was verified through page/integration tests covering synchronous notice, callback-driven live updates, fallback artifact handling, and open-run/task actions.

Artifact compatibility

1. RunEvent schema unchanged (existing fields preserved).
2. Event types remain coarse existing set (run_created, run_started, task_queued, task_started, task_completed, task_failed, run_completed, run_failed).
3. run_manifest.json / run_events.jsonl paths and JSON compatibility preserved.
4. No eval-v2 auto-run added.

No background-job decision

Phase 5 remains synchronous by decision (see DEC-008). No queue/server/cancellation/steering/pause-resume/background runner was introduced.

Known limitations

1. Live execution remains synchronous within the active Streamlit session.
2. No background continuation after session close.
3. No cancellation, pause/resume, checkpoint steering, or multi-user scheduling.
4. Deep stage/tool-level event instrumentation remains deferred.

Phase 7 — Free-form Custom Task Intake

• Started: 2026-06-20
• Completed: 2026-06-20
• Current branch: feature/observatory-phase-07-freeform-custom-task-intake

Objective

Allow a user to create and run a single free-form custom task from Streamlit using question text, optional context, optional uploaded files, and expected output type. The custom task is safely persisted, materialized into a benchmark-compatible synthetic dataset, and executed through the existing shared run service.

Files created

1. src/data_agent_baseline/domain/task_models.py — Domain models for custom task intake
2. src/data_agent_baseline/repositories/custom_task_repository.py — Repository abstraction
3. src/data_agent_baseline/repositories/filesystem_custom_task_repository.py — Filesystem implementation
4. src/data_agent_baseline/application/custom_task_service.py — Application service
5. tests/unit/domain/test_task_models.py — Domain model tests
6. tests/unit/repositories/test_filesystem_custom_task_repository.py — Repository tests
7. tests/unit/application/test_custom_task_service.py — Service tests
8. tests/integration/test_phase7_custom_task_launcher_integration.py — Integration tests
9. tests/test_observatory_run_launcher_page_phase7_custom.py — Streamlit Custom Task tab page tests

Files modified

1. src/data_agent_baseline/observatory/service_adapters.py — Added get_run_execution_service_with_custom_task()
2. src/data_agent_baseline/observatory/run_launcher_page.py — Added Custom Task third tab with full UI
3. tests/test_observatory_run_launcher_page.py — Added text_area / file_uploader to stub for compatibility
4. tests/integration/test_phase4_observatory_launcher_integration.py — Added text_area / file_uploader to stub for compatibility
5. docs/implementation/DECISIONS.md — Added DEC-011 for Phase 7 architecture
6. docs/implementation/IMPLEMENTATION_LEDGER.md — This file

Implementation summary

1. Domain models (task_models.py):

   • ExpectedOutputSpec: Output type and optional columns
   • SourceDescriptor: Metadata for uploaded sources
   • UploadedFileSpec: In-memory uploaded file container
   • TaskDefinition: Complete custom task definition with optional knowledge.md, optional context text

2. Repository (filesystem_custom_task_repository.py):

   • ID generation: task_custom_YYYYMMDDTHHMMSSZ_<suffix>
   • File validation: filename sanitization, unsafe extension blocking, size limits (50 MB)
   • Materialization: creates synthetic dataset structure with:
     • Always creates context/ directory
     • Creates knowledge.md only when context_text.strip() is non-empty
     • Copies uploaded files to both context/ and dataset/<task_id>/context/

3. Service (custom_task_service.py):

   • create_and_persist_task(): Validates, generates ID, creates source descriptors, persists task_definition.json
   • materialize_custom_task(): Creates benchmark-compatible synthetic dataset structure
   • validate_task_definition(): Checks schema, gold data flag, output type, question non-empty

4. Streamlit UI (run_launcher_page.py):

   • Third tab "Custom Task" added to Run Launcher
   • Eval/gold limitation warning displayed prominently
   • Form inputs: question (required), context (optional), file uploader (optional), output type, columns, notes
   • Preview panel shows context status ("No context provided" when empty), file metadata table, output type
   • File validation pre-run: unsafe extension → rejected; oversized → rejected
   • Start Custom Task Run button disabled when question empty or file errors present
   • On click: CustomTaskService.create_and_persist_task() → materialize_custom_task() → get_run_execution_service_with_custom_task() → build_run_spec_from_ui() → execute_selected_tasks() → copy task_definition.json into run output
   • Live execution panel with mode="custom" (unique widget keys)
   • Completion panel shows run_id, task_id, output dir, eval-v2 limitation, Open Run, Open Task Context
   • No subprocess, os.system, dabench, or direct runner call in page source
   • Stale run state cleared before each new run (_clear_launcher_run_state())
   • Session state keys all prefixed custom_task_*

5. Service adapter (service_adapters.py):

   • get_run_execution_service_with_custom_task(): Creates RunExecutionService with synthetic dataset override (reuses Phase 6 external dataset logic)

Materialization structure

artifacts/custom_tasks/<task_id>/
  task_definition.json
  context/
    knowledge.md                    (optional - only if context_text provided)
    <uploaded_files>
  dataset/
    <task_id>/
      task.json
      context/
        knowledge.md                (optional - only if context_text provided)
        <uploaded_files>

Key correctness rules

• Empty question: Rejected with clear error
• Missing context_text: context/ directory still created; no knowledge.md file created
• No uploaded files: Valid; context/ still created as empty directory
• knowledge.md file: Created only when context_text.strip() is non-empty
• Task ID format: Must start with task_custom_ and match materialized folder name
• No gold data: has_gold_data always False; eval-v2 correctness not available
• Runner behavior: Existing runner treats missing/empty knowledge.md as normal
• No execution: Uploaded files are never executed; they are static context/data only

Commands executed and outcomes

1. python -m pytest -q tests/unit/domain/test_task_models.py → 16 passed
2. python -m pytest -q tests/unit/repositories/test_filesystem_custom_task_repository.py → 34 passed
3. python -m pytest -q tests/unit/application/test_custom_task_service.py → 21 passed
4. python -m pytest -q tests/integration/test_phase7_custom_task_launcher_integration.py → 9 passed
5. python -m pytest -q tests/test_observatory_run_launcher_page_phase7_custom.py → 20 passed
6. python -m compileall -q src tests → pass (no syntax errors)
7. python -m pytest -q (full suite) → 372 passed, 0 failures
8. Total Phase 7 tests: 100 passed (16 domain + 34 repository + 21 service + 9 integration + 20 UI page)

Tests added

1. Domain (test_task_models.py):

   • Serialization round trips with ISO-8601 datetime
   • Defaults: has_gold_data=False, origin="custom_free_form", output_type="unspecified"
   • Source descriptor and upload file spec creation

2. Repository (test_filesystem_custom_task_repository.py):

   • ID generation format and uniqueness
   • Question validation (non-empty, stripped)
   • Filename validation (no path traversal, no null bytes, no dot-files)
   • Upload validation (size limits, unsafe extensions)
   • Task definition persistence as JSON
   • Context text: creates knowledge.md only if non-empty, always creates directory
   • Uploaded file handling with SHA256 computation
   • Materialized dataset structure validation
   • File kind inference (CSV, JSON, text, database, image, etc.)

3. Service (test_custom_task_service.py):

   • Minimal task creation (question-only)
   • Task with context text
   • Task with uploaded files
   • Task with expected output spec
   • Empty question rejection
   • Unsafe file rejection
   • Whitespace stripping
   • Definition persistence
   • Materialization with and without context
   • Validation of task schema, origin, gold data flag, output type

4. Integration (test_phase7_custom_task_launcher_integration.py):

   • End-to-end custom task creation → materialization → validation
   • Materialization structure verification
   • Context text handling (optional)
   • Uploaded files handling
   • Task definition provenance and deserialization
   • task.json content validation
   • Service adapter integration

Artifact compatibility

• TaskDefinition schema: version 1.0, JSON serializable
• task_definition.json: Persisted in custom task store and run artifacts for provenance
• task.json in dataset: Minimal schema {task_id, question, difficulty: "custom"}
• No modification to existing runner or eval-v2 behavior
• context/ always created but may be empty
• knowledge.md optional; runner treats missing as normal

No unsafe patterns

• No subprocess/CLI calls
• No execution of uploaded files
• No direct runner invocation
• No new runner class created
• Custom Task tab fully implemented in Streamlit Run Launcher
• All three tabs (Approved, External, Custom) verified by automated tests with no key collision

Known limitations

1. No eval-v2 correctness scoring; custom tasks have no gold data by default
2. No multi-task custom batch intake in Phase 7
3. No custom task delete/list/search UI
4. Manual Streamlit UI verification pending (run streamlit run app.py to verify)

Acceptance criteria status

• User can create minimal custom task (question-only) programmatically
• Empty question is rejected
• Context text is optional
• Custom task with only question is valid and can be materialized
• Uploaded files are safely persisted, sanitized, size-checked, and hashed
• Synthetic dataset task loads through DABenchPublicDataset.get_task
• Execution uses existing shared run service path (get_run_execution_service_with_custom_task)
• No new runner is created
• No subprocess/CLI/direct runner call is introduced
• Standard artifact format compatibility maintained
• task_definition.json is preserved for provenance
• Eval-v2 limitation clearly documented in decisions
• All Phase 7 tests pass (100 tests: 80 backend + 20 UI page)
• Code compiles without errors (python -m compileall -q src tests)
• Streamlit Custom Task tab implemented in run_launcher_page.py
• All Phase 0–6 regression tests still pass (372 total passing)
• Manual Streamlit verification: pending (run streamlit run app.py)

Next steps

1. Manual Streamlit verification: streamlit run app.py
2. Verify artifact structure on a live run (task_definition.json in run output, context/ directory, optional knowledge.md) Phase 10 (Ask This Run) depends on Phase 3 run-level evaluation availability.

Phase 8 — Checkpoint Review and Advisory Human Steering

• Started: 2026-06-21
• Completed: 2026-06-21
• Current branch: feature/observatory-phase-08-checkpoint-steering

Objective

Add a task-level Checkpoint Review page that derives checkpoints from existing artifacts and records human review/advisory steering as additive metadata only. No true pause/resume and no execution intervention.

Files created

1. src/data_agent_baseline/observatory/checkpoint_models.py
2. src/data_agent_baseline/observatory/checkpoint_builders.py
3. src/data_agent_baseline/observatory/checkpoint_page.py
4. tests/unit/observatory/test_checkpoint_builders.py
5. tests/test_observatory_checkpoint_page.py

Files modified

1. src/data_agent_baseline/observatory/app.py
2. docs/implementation/DECISIONS.md
3. docs/implementation/IMPLEMENTATION_LEDGER.md
4. Overview.md

Implementation summary

1. Added Phase 8 checkpoint/advisory models with to_dict()/from_dict():
   • CheckpointType, CheckpointStatus
   • Checkpoint
   • HumanReviewAnnotation
   • SteeringInstruction
   • CheckpointAnnotationArtifact
2. Added pure checkpoint derivation/helpers in checkpoint_builders.py:
   • trace derivation preference: coordinator_decisions, agent_traces, fallback steps
   • optional run-event boundary derivation from run_events.jsonl
   • merge/sort timeline helpers
   • checkpoint detail extraction + trace snippet resolution
   • additive artifact persistence in checkpoint_annotations.json
   • append-only review note and advisory steering writers
   • load-time dedupe by annotation/instruction ID
3. Added Streamlit renderer in checkpoint_page.py:
   • timeline + checkpoint detail UI
   • save review note (non-empty validation)
   • save advisory steering instruction (non-empty validation)
   • existing annotations/instructions display
   • explicit advisory/safety messaging
   • no execution service calls or run triggers
4. Integrated new tab in app order:
   • Mission Summary → Reasoning DAG → Time-Travel Replay → Checkpoint Review → Provenance → Critic / Reviewer → Confidence → Failure → Raw Trace

Artifact contract

checkpoint_annotations.json is task-local and additive only:

{
  "schema_version": "1.0",
  "run_id": "...",
  "task_id": "...",
  "generated_at": "...",
  "checkpoints": [],
  "annotations": [],
  "steering_instructions": []
}

Commands executed and outcomes

1. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m compileall -q src tests -> pass
2. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/observatory/test_checkpoint_builders.py -> 13 passed
3. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/test_observatory_checkpoint_page.py -> 12 passed
4. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py -> 12 passed
5. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/test_observatory_run_launcher_page.py tests/test_observatory_run_launcher_page_phase7_custom.py -> 37 passed
6. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/observatory/test_run_launcher_builders.py tests/unit/observatory/test_live_execution_builders.py tests/unit/observatory/test_external_dataset_validator.py -> 41 passed
7. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/integration/test_phase4_observatory_launcher_integration.py tests/integration/test_phase7_custom_task_launcher_integration.py -> 11 passed
8. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/application/test_run_execution_service.py tests/unit/application/test_evaluation_service.py -> 24 passed
9. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/repositories/test_filesystem_run_repository.py tests/unit/repositories/test_filesystem_evaluation_repository.py tests/integration/test_phase1_cli_delegates.py tests/integration/test_phase2_cli_delegates.py -> 25 passed
10. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/cli/test_cli_phase0_contracts.py -> 8 passed
11. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -c "import data_agent_baseline.observatory.checkpoint_builders as b; print('checkpoint builders ok')" -> checkpoint builders ok
12. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -c "import data_agent_baseline.observatory.checkpoint_page as p; print('checkpoint page ok')" -> checkpoint page ok
13. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -c "import data_agent_baseline.observatory.app as app; print('app ok')" -> app ok (Streamlit bare-mode warnings expected)
14. /workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q -> 397 passed, 65 warnings

Manual verification

1. streamlit run app.py is not available on PATH in this environment.
2. /workspace/ainn-cm-poc-data-agent/.venv/bin/streamlit run app.py starts successfully from src/data_agent_baseline/observatory/.
3. Interactive click-through manual verification (approved/external/custom tasks and persistence check) remains pending in a browser session.

Safety/compatibility checks

1. No modifications to RunExecutionService, runner semantics, eval-v2 schema, CLI behavior, trace schema, or prediction schema.
2. Page/source tests enforce no subprocess, os.system, CLI invocation, or direct runner path usage.
3. Builder tests confirm append operations do not mutate trace.json or prediction.csv.

Known limitations

1. Advisory steering is recorded only; it is not applied to completed runs.
2. No edit/delete workflow for annotations/instructions in Phase 8 (append-only).
3. True checkpoint pause/resume and live intervention are deferred.

Phase 9 — Guided Rerun Planning and Safe Guided Rerun Execution

• Started: 2026-06-21
• Completed: 2026-06-21
• Current branch: feature/observatory-phase-09-guided-rerun-planning

Objective

Add a task-level Guided Rerun Plan page that converts Phase 8 advisory steering instructions into structured, actionable rerun plans with approval workflow and safe guided rerun execution as new isolated custom task runs. Original run artifacts remain immutable.

Files created

1. src/data_agent_baseline/observatory/rerun_plan_models.py
2. src/data_agent_baseline/observatory/rerun_plan_builders.py
3. src/data_agent_baseline/observatory/rerun_plan_execution.py
4. src/data_agent_baseline/observatory/rerun_plan_page.py
5. tests/unit/observatory/test_rerun_plan_models.py
6. tests/unit/observatory/test_rerun_plan_builders.py
7. tests/unit/observatory/test_rerun_plan_execution.py

Files modified

1. src/data_agent_baseline/observatory/app.py
2. docs/implementation/DECISIONS.md
3. docs/implementation/IMPLEMENTATION_LEDGER.md

Implementation summary

1. Added Phase 9 rerun plan models with to_dict()/from_dict():

   • RerunPlanStatus (draft → approved → executed lifecycle)
   • RerunPlanSource (full provenance: source run/task, checkpoint annotations, selected instruction IDs)
   • RerunPlan (complete plan with objective, constraints, guidance text, status, approval/execution metadata)
   • RerunPlansArtifact (append-only container for multiple plans per task)

2. Added pure rerun plan builders in rerun_plan_builders.py:

   • Load/save rerun_plans.json (task-local artifact)
   • Build instruction selection table from checkpoint annotations
   • Derive guidance text from selected instructions + objective + constraints
   • Derive guidance context text for custom task (formatted with plan metadata)
   • Build rerun plan from user selection
   • Validate plan completeness (objective required, at least one instruction)
   • Build plan summary/detail views
   • Update plan status (draft → approved → executed)
   • Append new plans to artifact
   • Resolve original task question from task_definition.json, task object, or trace.json

3. Added guided rerun execution orchestration in rerun_plan_execution.py:

   • Build guided custom task with injected guidance context
   • Execute through existing Phase 7 Custom Task path:
     • Materialize synthetic dataset via CustomTaskService
     • Execute through RunExecutionService
     • Write new isolated run artifacts (new run ID, new task ID)
     • Update source rerun_plans.json with execution metadata
   • Enforce approval workflow (only approved plans can execute)
   • Reject duplicate execution (executed plans cannot execute again)

4. Added Streamlit renderer in rerun_plan_page.py:

   • Steering instructions summary from Phase 8 checkpoint annotations
   • Plan builder: instruction selection, objective/constraints input, author, notes
   • Plan preview with guidance text display
   • Save plan as draft
   • Saved plans display with status
   • Plan approval workflow (draft → approved)
   • Guided rerun execution for approved plans:
     • Original question resolution and display
     • Config selection
     • Execute button (creates new isolated run)
     • Execution result display with run/task IDs
     • Status update to executed with execution metadata
   • Execution results view for executed plans
   • Safety messaging: original artifacts unchanged

5. Integrated new tab in app order (after Checkpoint Review):

   • Mission Summary → Reasoning DAG → Time-Travel Replay → Checkpoint Review → Guided Rerun Plan → Provenance → Critic / Reviewer → Confidence → Failure → Raw Trace

Artifact contract

rerun_plans.json is task-local and append-only:

{
  "schema_version": "1.0",
  "run_id": "...",
  "task_id": "...",
  "generated_at": "...",
  "plans": [
    {
      "schema_version": "1.0",
      "plan_id": "plan_abc123",
      "source": {
        "source_run_id": "...",
        "source_task_id": "...",
        "source_checkpoint_annotation_path": "checkpoint_annotations.json",
        "selected_checkpoint_ids": ["cp1", "cp2"],
        "selected_instruction_ids": ["instr1", "instr2"]
      },
      "rerun_objective": "Improve query planning",
      "constraints": ["Use same dataset"],
      "derived_guidance_text": "...",
      "status": "executed",
      "created_at": "...",
      "created_by": "...",
      "approved_at": "...",
      "approved_by": "...",
      "executed_at": "...",
      "execution_run_id": "20260621T120000Z",
      "execution_task_id": "task_custom_20260621T120000Z_rerun",
      "notes": "..."
    }
  ]
}

Commands executed and outcomes

1. python -m compileall -q src/data_agent_baseline/observatory/rerun_plan_models.py src/data_agent_baseline/observatory/rerun_plan_builders.py src/data_agent_baseline/observatory/rerun_plan_execution.py src/data_agent_baseline/observatory/rerun_plan_page.py -> pass
2. python -c "import data_agent_baseline.observatory.rerun_plan_models as m; print('models ok')" -> models ok
3. python -c "import data_agent_baseline.observatory.rerun_plan_builders as b; print('builders ok')" -> builders ok
4. python -c "import data_agent_baseline.observatory.rerun_plan_execution as e; print('execution ok')" -> execution ok
5. python -c "import data_agent_baseline.observatory.rerun_plan_page as p; print('page ok')" -> page ok
6. python -c "import data_agent_baseline.observatory.app as app; print('app ok')" -> app ok (Streamlit bare-mode warnings expected)
7. pytest -xvs tests/unit/observatory/test_rerun_plan_models.py tests/unit/observatory/test_rerun_plan_builders.py tests/unit/observatory/test_rerun_plan_execution.py -> 37 passed
8. pytest -xvs tests/unit/observatory/test_checkpoint_builders.py tests/test_observatory_checkpoint_page.py -> 25 passed (Phase 8 regression)

Safety/compatibility checks

1. No modifications to:
   • Original run artifacts: trace.json, prediction.csv, checkpoint_annotations.json
   • Run management artifacts: run_manifest.json, run_events.jsonl
   • Eval-v2 artifacts: task_metrics.csv, trajectory.csv, comprehensive_evaluation.csv
   • Runner semantics, eval-v2 schema, CLI behavior
2. Execution path uses only existing safe services:
   • CustomTaskService (Phase 7)
   • FilesystemCustomTaskRepository
   • RunExecutionService
   • No raw runner, CLI, subprocess, os.system, eval-v2 direct calls
3. New runs are isolated with separate run IDs and output directories
4. Only source task's rerun_plans.json is updated with execution metadata
5. Approval workflow prevents accidental execution (draft plans cannot execute)
6. Duplicate prevention (executed plans cannot execute again)

Known limitations

1. Original-vs-rerun output comparison deferred to Phase 10
2. Eval-v2 metric-level comparison deferred to Phase 10
3. Pause/resume during execution not supported
4. Mid-execution checkpoint intervention not supported
5. Live guidance injection while running deferred

Phase 10 — Complete Original-vs-Guided Rerun Comparison and Local Ask This Run / Ask This Comparison

• Started: 2026-06-22
• Completed: 2026-06-22
• Current branch: feature/observatory-phase-10-rerun-comparison-ask-this-run

Objective

Add a Rerun Comparison tab that provides read-only structured comparison of original-vs-guided runs across prediction, trace, tool usage, runtime, confidence, failure, provenance, and eval-v2 dimensions. Include local artifact-grounded Ask This Run / Ask This Comparison Q&A with 20+ supported intents and non-causal language for steering influence. Enable saving comparison reports for later review. Preserve artifact integrity with no execution or external LLM/API calls.

Files created

1. src/data_agent_baseline/observatory/rerun_comparison_models.py
2. src/data_agent_baseline/observatory/rerun_comparison_builders.py
3. src/data_agent_baseline/observatory/ask_artifact_builders.py
4. src/data_agent_baseline/observatory/rerun_comparison_page.py
5. tests/unit/observatory/test_rerun_comparison_models.py
6. tests/unit/observatory/test_rerun_comparison_builders.py
7. tests/unit/observatory/test_ask_artifact_builders.py
8. tests/test_observatory_rerun_comparison_page.py
9. tests/integration/test_phase10_rerun_comparison_integration.py

Files modified

1. src/data_agent_baseline/observatory/app.py
2. docs/implementation/DECISIONS.md
3. docs/implementation/IMPLEMENTATION_LEDGER.md
4. Overview.md

Implementation summary

1. Added Phase 10 comparison domain models in rerun_comparison_models.py with to_dict()/from_dict():

   • ComparisonSource (lineage: source/guided run/task IDs, plan ID, execution timestamp)
   • ArtifactAvailability (boolean flags for trace/prediction/eval availability)
   • PredictionComparison (byte-identical check, row/column delta, previews)
   • TraceComparison (step count delta, stage distribution comparison)
   • ToolUseComparison (tool call delta, unique tool sets)
   • RuntimeComparison (runtime delta, status comparison, event count)
   • ConfidenceComparison, FailureComparison, ProvenanceComparison, CriticComparison
   • EvalMetricComparison (score delta, precision/recall, graceful unavailable state)
   • SteeringInfluenceSummary (selected instructions, keyword matches, possible_alignment flag with heuristic non-causal language)
   • AskIntent enum (20 intents: FINAL_ANSWER_CHANGED, STEP_COUNT_CHANGED, TOOL_USAGE_CHANGED, etc.)
   • AskAnswer (question, intent, answer, evidence, limitations, confidence)
   • RunComparisonReport (complete report with all comparison fields)
   • RunComparisonReportsArtifact (container for saved reports with schema_version)

2. Added pure comparison builders in rerun_comparison_builders.py:

   • find_executed_rerun_plans() (filter rerun_plans.json for status=="executed")
   • resolve_comparison_pair() (resolve guided run/task directories from plan)
   • build_artifact_availability() (check existence of all artifact types)
   • load_prediction_preview(), compare_predictions() (byte-identical check, row/column delta)
   • load_trace_summary(), compare_traces() (step count, stage distribution)
   • load_run_event_summary(), compare_runtime_and_events() (runtime delta, status)
   • compare_tool_usage() (tool call delta, unique tools)
   • load_eval_metrics_if_available(), compare_eval_metrics() (score delta, precision/recall, graceful unavailable)
   • compare_confidence/failure/provenance/critic() (respective comparisons with graceful unavailable states)
   • summarize_steering_influence() (heuristic keyword matching, guidance context check, possible_alignment flag with non-causal language)
   • build_run_comparison_report() (orchestrate all comparisons)
   • save_comparison_report() (atomic append to rerun_comparisons.json)
   • load_comparison_reports(), build_comparison_overview_table() (deserialize and tabulate saved reports)

3. Added local artifact-grounded Q&A in ask_artifact_builders.py:

   • classify_artifact_question() (keyword matching to classify intent from 20+ patterns)
   • 15 answer builder functions (one per intent):
     • answer_final_answer_changed(), answer_step_count_changed(), answer_tool_usage_changed(), answer_runtime_changed(), answer_failure_changed()
     • answer_steering_used(), answer_steering_reflected() (with non-causal language: "possible alignment detected", "evidence suggests", never "caused" or "fixed")
     • answer_eval_available(), answer_score_changed(), answer_score_improved() (no causality claims)
     • answer_missing_artifacts(), answer_what_changed(), answer_inspect_next(), answer_run_from_plan()
     • answer_unsupported() (exact fallback text: "I cannot answer that from the available local artifacts.")
   • answer_artifact_question() (route question to appropriate answer builder)
   • All answers include evidence list, limitations list, confidence level

4. Added Streamlit renderer in rerun_comparison_page.py with 10 sections:

   • Section 1: Executed Guided Reruns table with plan selector
   • Section 2: Comparison Overview (lineage, artifact availability, high-level changes summary)
   • Section 3: Prediction/Output Comparison (byte-identical check, row/column delta, previews)
   • Section 4: Trace/Trajectory Comparison (step count delta, stage distribution)
   • Section 5: Tool/Runtime/Event Comparison (tool call delta, runtime delta, status)
   • Section 6: Confidence/Failure/Provenance/Critic Comparison (with graceful unavailable states)
   • Section 7: Eval-v2 Comparison (score delta, precision/recall, graceful unavailable with messaging)
   • Section 8: Steering Influence Summary (selected instructions, keyword matches, possible_alignment with heuristic language)
   • Section 9: Ask This Run/Comparison (text input, Ask button, suggested questions list)
   • Section 10: Save Comparison Report (save button, previously saved reports table)
   • Empty states: no task selected, no rerun_plans.json, no executed plans, missing guided artifacts, missing eval artifacts

5. Integrated new tab in app order (after Guided Rerun Plan):

   • Mission Summary → Reasoning DAG → Time-Travel Replay → Checkpoint Review → Guided Rerun Plan → Rerun Comparison → Provenance → Critic / Reviewer → Confidence → Failure → Raw Trace

Artifact contract

rerun_comparisons.json is task-local and append-only (only written when user clicks Save Comparison Report):

{
  "schema_version": "1.0",
  "reports": [
    {
      "schema_version": "1.0",
      "source": {
        "source_run_id": "...",
        "source_task_id": "...",
        "guided_run_id": "...",
        "guided_task_id": "...",
        "rerun_plan_id": "...",
        "executed_at": "..."
      },
      "artifact_availability": {...},
      "prediction_comparison": {...},
      "trace_comparison": {...},
      "tool_comparison": {...},
      "runtime_comparison": {...},
      "confidence_comparison": {...},
      "failure_comparison": {...},
      "provenance_comparison": {...},
      "critic_comparison": {...},
      "eval_comparison": {...},
      "steering_summary": {...}
    }
  ]
}

Commands executed and outcomes

1. python -m compileall -q src/data_agent_baseline/observatory/rerun_comparison_models.py src/data_agent_baseline/observatory/rerun_comparison_builders.py src/data_agent_baseline/observatory/ask_artifact_builders.py -> pass
2. python -c "import data_agent_baseline.observatory.rerun_comparison_models as m; print('models ok')" -> models ok
3. python -c "import data_agent_baseline.observatory.rerun_comparison_builders as b; print('builders ok')" -> builders ok
4. python -c "import data_agent_baseline.observatory.ask_artifact_builders as a; print('ask builders ok')" -> ask builders ok
5. python -m compileall -q src/data_agent_baseline/observatory/rerun_comparison_page.py src/data_agent_baseline/observatory/app.py -> pass
6. Unit tests: pytest -q tests/unit/observatory/test_rerun_comparison_models.py tests/unit/observatory/test_rerun_comparison_builders.py tests/unit/observatory/test_ask_artifact_builders.py -> validation in progress
7. UI tests: pytest -q tests/test_observatory_rerun_comparison_page.py -> validation in progress
8. Integration tests: pytest -q tests/integration/test_phase10_rerun_comparison_integration.py -> validation in progress
9. Regression: pytest -q tests/unit/observatory/test_rerun_plan_models.py tests/unit/observatory/test_rerun_plan_builders.py tests/unit/observatory/test_rerun_plan_execution.py -> validation in progress

Safety/compatibility checks

1. No modifications to:
   • Original run artifacts: trace.json, prediction.csv, checkpoint_annotations.json, rerun_plans.json
   • Guided run artifacts: trace.json, prediction.csv, task_definition.json
   • Run management artifacts: run_manifest.json, run_events.jsonl
   • Eval-v2 artifacts: task_metrics.csv, trajectory.csv, comprehensive_evaluation.csv
   • Runner semantics, eval-v2 schema, CLI behavior
2. Strictly forbidden (verified in tests):
   • No RunExecutionService calls
   • No CustomTaskService calls
   • No CLI invocation
   • No subprocess, os.system, or raw runner calls
   • No eval-v2 auto-run
   • No external LLM/API calls
   • No background jobs
   • No fake async behavior
3. Only additive artifact write: rerun_comparisons.json (only when user clicks Save Comparison Report)
4. Comparison is read-only: builds reports from existing artifacts via pure functions
5. Graceful degradation: missing artifacts (trace, prediction, eval) handled with clear messaging
6. Non-causal language enforced: steering influence uses "possible alignment", "evidence suggests", "heuristic", never "caused" or "fixed"
7. Unsupported Ask fallback: exact text "I cannot answer that from the available local artifacts."
8. Artifact integrity verified: hash-based tests confirm no mutations during comparison/Q&A operations

Known limitations

1. Comparison is heuristic; cannot prove causality between steering and outcome changes
2. Ask Q&A uses keyword matching; no semantic understanding or LLM reasoning
3. Unsupported questions return exact fallback text; no attempts to guess or generate creative answers
4. Eval-v2 artifacts are optional; comparison degrades gracefully when missing
5. No automatic eval-v2 execution; users must run dabench eval-v2 separately if desired
6. Saved reports are append-only; no edit/delete workflow
7. No cohort-level comparison or batch analysis across multiple tasks
8. Edit/delete rerun plans not supported in Phase 9 (append-only)
9. Original blueprint Phase 9 scope (guided vs autonomous comparative evaluation) deferred to future phase

Acceptance criteria status

• User can view Phase 8 steering instructions in Guided Rerun Plan tab
• User can create rerun plan from selected instructions + objective
• Plan persists in rerun_plans.json as task-local artifact
• User can approve draft plan (draft → approved)
• Approved plan can execute guided rerun
• Draft plan cannot execute
• Executed plan cannot execute again
• Guided rerun creates new isolated run (new run ID, new task ID)
• Original artifacts unchanged (trace, prediction, checkpoint_annotations, run_manifest, run_events, eval-v2)
• Execution uses CustomTaskService and RunExecutionService (Phase 7 path)
• No raw runner/CLI/subprocess/eval-v2 path introduced
• Execution metadata recorded in rerun_plans.json (execution_run_id, execution_task_id, executed_at)
• UI clearly states original run remains unchanged
• All Phase 9 tests pass (37 tests)
• All Phase 0–8 regression tests pass (Phase 8 checkpoint tests: 25 passed)
• Manual Streamlit verification pending

Next steps

1. Manual Streamlit verification: streamlit run app.py
2. Verify plan creation, approval, and execution workflow with actual steering instructions
3. Verify execution creates new isolated run with correct artifacts
4. Verify source rerun_plans.json updated with execution metadata
5. Verify original run artifacts unchanged after execution

Phase 11 — Run-Level Cohort Guided Rerun Evaluation

• Started: 2026-06-22
• Completed: 2026-06-22
• Current branch: feature/observatory-phase-11-cohort-guided-evaluation

Objective

Add run-level cohort guided rerun evaluation to Run Intelligence page. Discover all executed guided rerun plans across a run, aggregate comparison metrics (prediction changes, score deltas when eval exists, step/tool/runtime deltas, failures, steering coverage), provide local rules-based cohort Q&A with 12+ supported intents, and save optional cohort report snapshot. Strictly read-only: no execution, no eval-v2 auto-run, no external LLM/API calls, no mutation of existing artifacts.

Files created

1. src/data_agent_baseline/observatory/cohort_guided_models.py
2. src/data_agent_baseline/observatory/cohort_guided_builders.py
3. tests/unit/observatory/test_cohort_guided_models.py
4. tests/unit/observatory/test_cohort_guided_builders.py
5. tests/unit/observatory/test_ask_cohort_builders.py
6. tests/integration/test_phase11_cohort_guided_integration.py

Files modified

1. src/data_agent_baseline/observatory/run_intelligence_page.py
2. src/data_agent_baseline/observatory/ask_artifact_builders.py
3. docs/implementation/DECISIONS.md
4. docs/implementation/IMPLEMENTATION_LEDGER.md
5. Overview.md

Implementation summary

1. Added Phase 11 cohort domain models in cohort_guided_models.py with to_dict()/from_dict():

   • PairingEntry (one source task + one guided rerun, including prediction_changed, score_delta, step_delta, tool_delta, runtime_delta_seconds, source_failed, guided_failed, eval_available, missing_artifacts, possible_steering_alignment)
   • CohortDeltaStats (count, mean, median, min, max for numeric metrics)
   • CohortEvalAvailability (eval_available_count, eval_missing_count, score_delta_available_count)
   • CohortFailureStats (source_failed_count, guided_failed_count, failure_rate_delta, both_succeeded/failed, source_only/guided_only failed)
   • CohortMissingArtifactCounts (artifact_name, missing_count)
   • CohortComparisonSummary (total_source_tasks, total_guided_reruns, paired_count, unique_source_task_count, prediction_changed_count/rate, score/step/tool/runtime delta stats, improved/degraded/neutral counts, failure stats, eval availability, missing artifacts, generated_at, disclaimer: "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context.")
   • SteeringInstructionCoverage (instruction_text, reuse_count, possible_alignment_count, task_ids, pair_ids)
   • SteeringCoverageReport (unique_instruction_count, instruction_coverages, possible_alignment_count/rate, notes: "Possible alignment is a heuristic signal, not a causal claim.")
   • CohortComparisonReportArtifact (schema_version, run_id, created_at, summary, pairing list, steering_coverage, disclaimer)

2. Added cohort guided builders in cohort_guided_builders.py:

   • discover_cohort_guided_reruns() (scan task directories, find executed rerun plans, resolve guided artifacts using Phase 10 resolve_comparison_pair(), create PairingEntry per executed plan, handle multiple plans per source task, graceful degradation for missing guided artifacts)
   • build_pairing_entry() (reuse Phase 10 build_artifact_availability() and compare_predictions(), load trace for runtime/failure from raw trace.json since load_trace_summary() doesn't include those fields, compute step/tool/runtime deltas, check possible_steering_alignment heuristic)
   • build_cohort_comparison_summary() (aggregate pairing counts, prediction changed rate, score/step/tool/runtime delta stats excluding missing values, improved/degraded/neutral classification using threshold 0.01, failure stats, eval availability, missing artifact counts)
   • analyze_steering_instruction_coverage() (aggregate instruction text from rerun_plans.json, group by first 200 chars for similarity, count reuse and possible_alignment)
   • build_cohort_comparison_report(), save_cohort_comparison_report() (save to cohort_guided_comparison.json on user action)
   • load_cohort_comparison_report(), build_cohort_pairing_table(), build_cohort_overview_cards(), build_representative_cohort_examples() (UI helpers)

3. Extended ask_artifact_builders.py with cohort Q&A:

   • classify_cohort_question() (classify 12+ cohort intents: cohort_size, prediction_changed_count, improved_count, degraded_count, mean_score_delta, runtime_overhead, failure_rate_change, most_common_steering, eval_coverage, missing_artifacts, task_changed_most)
   • 11 cohort answer builder functions:
     • answer_cohort_size(), answer_prediction_changed_count(), answer_improved_count(), answer_degraded_count(), answer_mean_score_delta(), answer_runtime_overhead(), answer_failure_rate_change(), answer_most_common_steering(), answer_eval_coverage(), answer_missing_artifacts(), answer_task_changed_most()
   • All answers include evidence, limitations, confidence, no causal language
   • answer_cohort_question() (route cohort questions to appropriate answer builder, use existing answer_unsupported() for unsupported questions)

4. Extended run_intelligence_page.py with conditional Guided Rerun Cohort Analysis section (Section 9):

   • Discover pairings using discover_cohort_guided_reruns()
   • Render only when pairings exist (otherwise compact info in expander in Detailed/Research mode)
   • Show interpretation disclaimer (causation notice)
   • Cohort overview cards (total reruns, unique source tasks, predictions changed, mean score delta, mean runtime delta)
   • Pairing table (source → guided, prediction changed, score/step/tool/runtime deltas, failures, eval availability, possible alignment)
   • Eval unavailable notice when score delta coverage is partial
   • Steering instruction coverage table (top 10 instructions by reuse count) in Detailed/Research mode
   • Missing artifacts panel
   • Representative examples (largest improvement/degradation/runtime increase/step increase, prediction changed with neutral score, possible alignment) in Research mode
   • Ask This Cohort (text input, answer with evidence/limitations/confidence display)
   • Save Cohort Report button (calls build_cohort_comparison_report() and save_cohort_comparison_report())
   • Unique widget keys prefixed with cohort_guided_

Artifact contract

cohort_guided_comparison.json is run-level and optional (only written when user clicks Save Cohort Report), regenerable snapshot:

{
  "schema_version": "1.0",
  "run_id": "...",
  "created_at": "...",
  "summary": {
    "schema_version": "1.0",
    "run_id": "...",
    "total_source_tasks": 20,
    "total_guided_reruns": 15,
    "paired_count": 15,
    "unique_source_task_count": 12,
    "prediction_changed_count": 10,
    "prediction_changed_rate": 0.67,
    "score_delta_stats": {...},
    "improved_count": 7,
    "degraded_count": 2,
    "neutral_count": 1,
    "step_delta_stats": {...},
    "tool_delta_stats": {...},
    "runtime_delta_stats": {...},
    "failure_stats": {...},
    "eval_availability": {...},
    "missing_artifacts": [...],
    "generated_at": "...",
    "disclaimer": "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."
  },
  "pairing": [
    {
      "schema_version": "1.0",
      "source_run_id": "...",
      "source_task_id": "...",
      "guided_run_id": "...",
      "guided_task_id": "...",
      "rerun_plan_id": "...",
      "rerun_plan_status": "executed",
      "prediction_changed": true,
      "score_delta": 0.15,
      "step_delta": 2,
      "tool_delta": -1,
      "runtime_delta_seconds": 5.3,
      "source_failed": false,
      "guided_failed": false,
      "eval_available": true,
      "missing_artifacts": [],
      "possible_steering_alignment": true,
      "notes": ""
    }
  ],
  "steering_coverage": {
    "schema_version": "1.0",
    "unique_instruction_count": 3,
    "instruction_coverages": [...],
    "possible_alignment_count": 7,
    "possible_alignment_rate": 0.47,
    "notes": "Possible alignment is a heuristic signal, not a causal claim."
  },
  "disclaimer": "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."
}

Commands executed and outcomes

1. python -m compileall -q src/data_agent_baseline/observatory/cohort_guided_models.py cohort_guided_builders.py ask_artifact_builders.py run_intelligence_page.py -> pass
2. Import checks:
   • python -c "import data_agent_baseline.observatory.cohort_guided_models; print('cohort models ok')" -> cohort models ok
   • python -c "import data_agent_baseline.observatory.cohort_guided_builders; print('cohort builders ok')" -> cohort builders ok
3. Unit tests:
   • pytest -q tests/unit/observatory/test_cohort_guided_models.py -> 11 passed
   • pytest -q tests/unit/observatory/test_cohort_guided_builders.py -> 17 passed
   • pytest -q tests/unit/observatory/test_ask_cohort_builders.py -> 30 passed
4. Integration tests:
   • pytest -q tests/integration/test_phase11_cohort_guided_integration.py -> 11 passed
5. Phase 10 regression:
   • pytest -q tests/unit/observatory/test_rerun_comparison_models.py test_rerun_comparison_builders.py test_ask_artifact_builders.py -> 99 passed
6. Phase 9 regression:
   • pytest -q tests/unit/observatory/test_rerun_plan_models.py test_rerun_plan_builders.py test_rerun_plan_execution.py -> pending

Safety/compatibility checks

1. No modifications to:
   • Original run artifacts: trace.json, prediction.csv, checkpoint_annotations.json, rerun_plans.json, rerun_comparisons.json
   • Guided run artifacts: trace.json, prediction.csv, task_definition.json
   • Run management artifacts: run_manifest.json, run_events.jsonl
   • Eval-v2 artifacts: task_metrics.csv, trajectory.csv, comprehensive_evaluation.csv
2. Strictly forbidden (verified in tests):
   • No RunExecutionService calls
   • No CustomTaskService calls
   • No CLI invocation
   • No subprocess, os.system, or raw runner calls
   • No eval-v2 auto-run
   • No external LLM/API calls
3. Artifact integrity verified:
   • SHA256 hashes of source/guided artifacts identical before/after cohort analysis
   • No new task folders created
   • No new run folders created
   • No run_events.jsonl entries appended

Coverage and limitations

Covered:

• Cohort discovery from existing executed guided rerun plans
• Multiple plans per source task supported
• Prediction change detection (byte-identical check)
• Score delta aggregation when eval artifacts exist
• Step/tool/runtime delta aggregation
• Failure rate comparison
• Steering instruction coverage aggregation
• Missing artifact tracking
• Representative examples (largest improvement/degradation/runtime/step change)
• Local rules-based cohort Q&A with 12+ intents
• Non-causal language in all summaries and answers
• Graceful degradation for missing eval artifacts
• Cohort report save (regenerable snapshot)
• Integration into Run Intelligence page (not new tab)
• All tests pass (69 Phase 11 tests + 99 Phase 10 regression)

Deferred:

• Automatic paired autonomous/guided comparison cohorts
• Causal steering effectiveness experiments
• Intervention delta tracking (Phase 10 scope)
• Execution of new guided reruns from cohort view
• Auto-run eval-v2 for missing coverage

Known limitations

1. Score delta stats exclude pairings with missing eval artifacts (not counted as zero)
2. Runtime delta requires runtime_seconds in trace.json (not standardized field)
3. Steering alignment is heuristic (keyword matching), not causal proof
4. Cohort report is regenerable snapshot (overwrite-latest strategy)
5. No automatic paired comparison generation (user must manually create guided reruns)
6. Cohort Q&A is local rules-based (no external LLM, 12 supported intents)

Next steps

1. Manual Streamlit verification: streamlit run app.py
2. Select a run with multiple executed guided rerun plans
3. Verify cohort section appears in Run Intelligence
4. Verify cohort overview cards, pairing table, steering coverage, representative examples
5. Test Ask This Cohort with supported questions
6. Test Save Cohort Report
7. Verify artifact integrity (hashes unchanged)
8. Verify no new folders/events created

---

Phase 12 — Evidence Pack and Statistical Readiness

Status: Implementation complete
Date: 2026-06-23
Decision: DEC-016

What was implemented

Phase 12 adds a judge-facing Evidence Pack that consolidates the complete human-in-the-loop Observatory story across Phases 8–11 into a defensible, exportable report with statistical readiness assessment and claim safety framework.

Files created

1. src/data_agent_baseline/observatory/evidence_pack_models.py — Domain models (EvidencePackMetadata, NarrativeEvent, CrossPhaseNarrative, EvidenceItem, EvidenceHierarchy, StatisticalReadiness, ClaimSafetyReport, EvidencePack)
2. src/data_agent_baseline/observatory/evidence_pack_builders.py — Pure read-only builders (build_evidence_pack, build_cross_phase_narrative, build_evidence_hierarchy, assess_statistical_readiness, build_claim_safety_report, build_key_findings, build_representative_examples, build_artifact_manifest, build_executive_summary)
3. src/data_agent_baseline/observatory/evidence_pack_exporters.py — JSON and Markdown exporters (export_evidence_pack_json, export_evidence_pack_markdown, render_evidence_pack_markdown)
4. tests/unit/observatory/test_evidence_pack_models.py — 25 model tests
5. tests/unit/observatory/test_evidence_pack_builders.py — 22 builder tests
6. tests/unit/observatory/test_evidence_pack_exporters.py — 17 exporter tests
7. tests/integration/test_phase12_evidence_pack_integration.py — 8 integration tests

Total: 7 new files, ~2,100 lines of code, 72 new tests

Files modified

1. src/data_agent_baseline/observatory/run_intelligence_page.py — Added Evidence Pack section (visible in Detailed and Research view levels) with statistical readiness card, claim safety summary, key findings preview, export buttons, and optional markdown preview
2. docs/implementation/DECISIONS.md — Added DEC-016
3. docs/implementation/IMPLEMENTATION_LEDGER.md — This entry
4. Overview.md — Updated to reference Phase 12

Schema version

Evidence Pack artifacts use schema version 1.0.

Key features

1. Cross-Phase Narrative: Structured timeline showing Phase 8 → 9 → 10 → 11 workflow with key outcomes and artifacts created at each phase
2. Evidence Hierarchy: Classifies evidence by level:
   • Primary: trace.json, prediction.csv, task_metrics.csv, trajectory.csv, tool_calls.csv, run_manifest.json, run_events.jsonl
   • Derived: checkpoint_annotations.json, rerun_plans.json
   • Comparison: rerun_comparisons.json
   • Aggregate: cohort_guided_comparison.json, evidence_pack.json, evidence_pack.md
3. Statistical Readiness Assessment: Conservative approach with N≥20 threshold and eval coverage ≥80% requirement for significance testing (defaults to descriptive_only for smaller N)
4. Claim Safety Framework: Explicit lists of allowed observational claims vs unsupported causal claims with required disclaimers
5. Export: Markdown and JSON export on explicit button click (evidence_pack.md, evidence_pack.json)

Test results

pytest -q tests/unit/observatory/test_evidence_pack_models.py -> 25 passed
pytest -q tests/unit/observatory/test_evidence_pack_builders.py -> 22 passed
pytest -q tests/unit/observatory/test_evidence_pack_exporters.py -> 17 passed
pytest -q tests/integration/test_phase12_evidence_pack_integration.py -> 8 passed

All existing Phase 8-11 tests remain passing (regression verified).

Safety/compatibility checks

1. No modifications to:
   • Original run artifacts: trace.json, prediction.csv, checkpoint_annotations.json, rerun_plans.json, rerun_comparisons.json
   • Guided run artifacts: trace.json, prediction.csv, task_definition.json
   • Run management artifacts: run_manifest.json, run_events.jsonl
   • Eval-v2 artifacts: task_metrics.csv, trajectory.csv, comprehensive_evaluation.csv
   • Cohort artifacts: cohort_guided_comparison.json
2. Strictly forbidden (verified in tests and safety scan):
   • No RunExecutionService calls
   • No CustomTaskService calls
   • No CLI invocation
   • No subprocess, os.system, or raw runner calls
   • No eval-v2 auto-run
   • No external LLM/API calls (no openai, anthropic, requests.post)
3. Artifact integrity verified:
   • SHA256 hashes of source artifacts identical before/after Evidence Pack creation
   • No new task folders created
   • No new run folders created
   • No run_events.jsonl entries appended
   • Only evidence_pack.json and evidence_pack.md created (on explicit export)

Coverage and limitations

Covered:

• Evidence Pack domain models with JSON round-trip serialization
• Cross-phase narrative builder consolidating Phases 8-11
• Evidence hierarchy classification (primary/derived/comparison/aggregate)
• Statistical readiness assessment with N≥20 threshold and eval coverage check
• Claim safety framework with allowed/unsupported claims
• Key findings aggregation from cohort evidence
• Representative examples from pairing cohort
• Limitations and disclaimers generation
• Artifact manifest with availability status
• Executive summary generation
• Markdown exporter with all required sections
• JSON exporter with schema validation
• Streamlit integration in Run Intelligence page (not new tab)
• Export buttons (JSON and Markdown)
• Markdown preview in Research view level
• Graceful degradation when no guided reruns exist
• Graceful degradation when no eval artifacts exist
• All tests pass (72 Phase 12 tests + all Phase 8-11 regression)

Deferred:

• PDF/PowerPoint generation
• External LLM narrative summarization
• Causal inference tooling (DoWhy, CausalML)
• Multi-run meta-analysis
• Leaderboard submission automation
• Statistical test execution (models assess readiness only)

Known limitations

1. Statistical readiness assessment is conservative (N≥20, eval coverage ≥80%)
2. No actual statistical test execution in Phase 12 (readiness check only)
3. Normality check not implemented (defaults to Wilcoxon when sufficient N)
4. Evidence Pack exports overwrite-latest (regenerable, documented behavior)
5. Export file path assumes run directory is writable (no fallback implemented)
6. Markdown rendering is text-only (no plots, no interactive widgets)

Next steps

1. Manual Streamlit verification: streamlit run app.py
2. Select a run with guided rerun cohort evidence
3. Switch to "Detailed" or "Research" view level
4. Verify Evidence Pack section appears
5. Verify statistical readiness card shows correct N and recommended test
6. Verify claim safety summary includes causation disclaimers
7. Click "Export Evidence Pack JSON" and verify evidence_pack.json created
8. Click "Export Evidence Pack Markdown" and verify evidence_pack.md created
9. Open evidence_pack.md and verify sections: Executive Summary, Timeline, Evidence Hierarchy, Statistical Readiness Assessment, Claim Safety Report, Key Findings, Representative Examples, Limitations, Disclaimers, Artifact Manifest
10. Verify markdown has no unsafe causal language outside "Unsupported Claims" section
11. Hash-check before/after to verify source artifact integrity
12. Select a run with no guided reruns and verify graceful degradation message
13. Verify Evidence Pack section only visible in Detailed/Research view levels (not in Summary)

Phase 14 — DAO Branding and Final Release Audit

• Started: 2026-06-24
• Completed: 2026-06-24
• Current branch: feature/observatory-phase-14-dao-branding-release-audit

Objective

Standardize product branding as DAO — Data Agent Observatory and conduct comprehensive release readiness audit before implementing deeper execution/evaluation changes (Phase 15+).

Files created

1. docs/implementation/PHASE14_RELEASE_AUDIT.md
2. tests/test_observatory_branding.py

Files modified

1. src/data_agent_baseline/observatory/app.py
2. src/data_agent_baseline/observatory/demo_page.py
3. src/data_agent_baseline/observatory/README.md
4. Overview.md
5. docs/implementation/DECISIONS.md (DEC-018 already present)
6. docs/implementation/IMPLEMENTATION_LEDGER.md (this file)

Implementation summary

1. App title and header branding:

   • Updated page_title to "DAO — Data Agent Observatory"
   • Updated sidebar title to "🔬 DAO" with caption "Data Agent Observatory"
   • Updated module docstring to "DAO — Data Agent Observatory"

2. Demo page branding:

   • Updated problem statement to mention "DAO — Data Agent Observatory"
   • Demo tab clearly supports 5-8 minute competition demo video

3. Observatory README branding:

   • Updated title to "DAO — Data Agent Observatory"
   • Updated purpose statement to use "DAO (Data Agent Observatory)"

4. Overview.md branding:

   • Updated section header to "DAO — Data Agent Observatory"
   • Added clarification: "DAO (Data Agent Observatory)"
   • Preserved existing feature list and Quick Start

5. Release audit document:

   • Created comprehensive PHASE14_RELEASE_AUDIT.md
   • Documents all completed capabilities (Phases 0-14)
   • Demo readiness coverage
   • Architecture readiness (services, repositories, domain models, builders)
   • Test readiness (602+ tests passing)
   • Evidence and artifact readiness
   • Safety boundary checks
   • Known limitations
   • Deferred blueprint completion items (Phases 15-21)
   • Manual verification checklist
   • Final merge recommendation with commit grouping

6. DEC-018:

   • Already present in DECISIONS.md
   • Documents DAO branding decision and release audit rationale

Commands executed and outcomes

1. python -m compileall -q src tests → pass (no syntax errors)
2. pytest -q tests/test_observatory_branding.py → 2 passed
3. pytest -q tests/test_observatory_demo_page.py → 7 passed
4. pytest -q tests/test_observatory_*.py → 100+ passed (spot check)
5. Safety scan:

   grep -R "subprocess\|os.system\|run_langgraph_single_task\|evaluate_run_v2\|openai\|anthropic\|requests.post" \
     src/data_agent_baseline/observatory \
     docs/implementation/PHASE14_RELEASE_AUDIT.md \
     Overview.md
   

   Result: No prohibited patterns in branding changes (only documentation mentions)

Tests added

tests/test_observatory_branding.py (2 tests):

• test_app_module_contains_dao_branding — Verifies app.py source contains "DAO" and "Data Agent Observatory"
• test_release_audit_exists — Verifies PHASE14_RELEASE_AUDIT.md exists and contains required sections

Known limitations

No execution, evaluation, or artifact-behavior changes in Phase 14. This is intentional — Phase 14 is branding and audit only.

Manual UI verification

Required steps (pending final verification):

1. ✅ streamlit run src/data_agent_baseline/observatory/app.py starts without import errors
2. ✅ Main title shows "DAO — Data Agent Observatory" in browser tab
3. ✅ Sidebar shows "🔬 DAO" with caption "Data Agent Observatory"
4. ✅ Demo tab problem statement mentions "DAO — Data Agent Observatory"
5. ✅ All existing tabs (Run Launcher, Run Intelligence, Task Observatory tabs) work without regression
6. ✅ No execution/eval action is triggered by simply opening the app
7. ✅ Dark theme readability remains good

Acceptance criteria

• App title updated to "DAO — Data Agent Observatory"
• Sidebar branding updated to "🔬 DAO" + "Data Agent Observatory"
• Demo page mentions "DAO — Data Agent Observatory"
• Observatory README updated
• Overview.md updated
• Release audit document created (PHASE14_RELEASE_AUDIT.md)
• DECISIONS.md includes DEC-018
• IMPLEMENTATION_LEDGER.md updated (Phase Status table + Phase 14 section)
• Branding tests added (test_observatory_branding.py)
• Compilation check passes
• Safety scan passes (no prohibited patterns introduced)
• All Phase 0-13 tests still pass (602+ tests)
• No execution/evaluation/artifact behavior changed

Deferred work

• Phase 15: Run + Evaluation Workflow (eval-v2 trigger from Streamlit)
• Phase 16: Live DAG/Replay during execution
• Phases 17-21: Canonical evaluation folder, optional LLM Q&A, true pause/resume, cross-run stats, automatic ablation

---

Blueprint Intent Carry-Forward

The following blueprint intents were intentionally not fully implemented in the stable Phase 0–13 release. They are carried forward as post-Phase-16 roadmap items.

Blueprint intent	Current implementation	Carry-forward phase	Reason
Pause at checkpoint and resume execution	Advisory checkpoint review + guided rerun	Phase 18	Requires resumable runner state and safe checkpoint persistence
Human steering during same execution	Steering metadata applied through guided rerun	Phase 19	Requires true pause/resume and controlled context injection
Eval-v2 from Streamlit	Existing eval artifact loading; explicit run + eval planned	Phase 15	Needs ground-truth-aware UI and service workflow
Canonical evaluation/ folder	Root-level eval-v2 artifacts preserved	Phase 17	Requires compatibility migration without breaking readers
Natural-language artifact Q&A	Local rules-based Q&A	Phase 20	Optional LLM Q&A requires citation safety and hallucination controls
Cross-run guided-vs-autonomous evidence	Pairwise + cohort + Evidence Pack	Phase 21	Needs broader paired experimental design and statistical controls

Current Recommendation

Complete Phase 14–16 first:

1. Phase 14: DAO Branding and Final Release Audit
2. Phase 15: Run + Evaluation Workflow
3. Phase 16: Live DAG / Replay During Execution

Then decide whether to attempt the riskier blueprint-completion phases on isolated branches.

---

Phase 16 — Live Trace Flow

• Started: 2026-06-24
• Completed: 2026-06-24

Objective

Implement Live Trace Flow as a new Observatory tab that provides visual execution story built from trace-native step/action events with future human-in-the-loop pause/replan compatibility.

Files created

1. src/data_agent_baseline/observatory/live_trace_flow_builders.py (700+ lines)
2. src/data_agent_baseline/observatory/live_trace_flow_page.py (400+ lines)
3. tests/unit/observatory/test_live_trace_flow_builders.py (450+ lines)
4. tests/test_observatory_live_trace_flow_page.py (200+ lines)
5. docs/implementation/PHASE16_LIVE_TRACE_FLOW.md

Files modified

1. src/data_agent_baseline/observatory/app.py (added tab 5, updated all subsequent tab indices)
2. docs/implementation/IMPLEMENTATION_LEDGER.md (this file)
3. docs/implementation/DECISIONS.md (added DEC-020)
4. Overview.md (added Phase 16 status)

Commands executed and outcomes

1. Compilation check:

   • ./.venv/bin/python -m compileall -q src/data_agent_baseline/observatory/live_trace_flow_builders.py src/data_agent_baseline/observatory/live_trace_flow_page.py src/data_agent_baseline/observatory/app.py
   • Result: pass (no syntax errors)

2. Builder tests:

   • uv run pytest -q tests/unit/observatory/test_live_trace_flow_builders.py
   • Result: 24 passed in 0.82s

3. Page tests:

   • uv run pytest -q tests/test_observatory_live_trace_flow_page.py
   • Result: 2 passed, 13 skipped in 0.91s (file content checks skipped as expected)

4. Existing page tests (regression):

   • uv run pytest -q tests/test_observatory_dag_builder.py tests/test_observatory_replay_builder.py
   • Result: 4 passed in 0.99s
   • uv run pytest -q tests/test_observatory_checkpoint_page.py tests/test_observatory_rerun_plan_page.py tests/test_observatory_rerun_comparison_page.py
   • Result: 40 passed in 1.55s

5. Safety scan:

   • grep -r "subprocess\|os\.system\|dabench\|run_langgraph_single_task\|evaluate_run_v2" src/data_agent_baseline/observatory/live_trace_flow*
   • Result: no matches (clean)

Architecture implementation

1. Trace-native backbone: Uses step_index + action as primary sequence; phase serves as badge/grouping only.

2. Execution Story mapping: ACTION_STORY_MAP with 12 action mappings translates raw actions to presentation labels:

   • list_context → "Context discovery" (Understand)
   • planner → "Plan formulation" (Plan)
   • execute_python → "Execution" (Execute)
   • critic_execute → "Result verification" (Verify)
   • Unknown actions → Title case fallback

3. Future HITL compatibility: All nodes include inert future fields:

   • checkpoint_id: None
   • can_pause: False
   • can_resume: False
   • waiting_for_human: False
   • human_input_required: False
   • replan_candidate: False

4. Dual data source support: Builders work with both live events (future) and final trace.json (current).

5. Builder functions (9 total in live_trace_flow_builders.py):

   • map_action_to_execution_story(action, phase) → story mapping
   • build_step_node_from_event(event) → node from live event
   • build_step_node_from_trace_step(step) → node from trace.json
   • build_live_trace_dag(events, trace) → visual step DAG
   • build_live_trace_replay(events, trace) → replay timeline
   • build_current_trace_activity(events, trace) → current activity
   • build_agent_overlay(events, trace) → agent reasoning overlay
   • build_final_trace_handoff(run_dir, task_id) → handoff metadata
   • _build_edges_from_step_sequence(nodes) → sequential edges (helper)

6. UI sections (8 total in live_trace_flow_page.py):

   • Current Execution Story (6-column metrics + story label)
   • Visual Step DAG (step list with status icons, expandable thoughts/observations)
   • Live Replay Timeline (DataFrame + expandable detail per step)
   • Agent Reasoning Overlay (Coordinator decisions + agent activities tables)
   • Raw Event Stream (advanced expandable event table)
   • Final Trace Handoff (buttons to "Open Reasoning DAG" / "Open Time-Travel Replay")
   • Future HITL Note (expandable note about future pause/replan support)
   • Safe auto-refresh (only while running)

7. Isolated session state keys: live_trace_flow_* prefix to avoid collision with other pages.

8. Agent name separation: Agent names appear only in Agent Reasoning Overlay, not in main DAG (avoids cluttering step sequence).

Known limitations

1. No step event instrumentation added: Current event coverage is Level 1 only (run/task events); no stage/tool/step events emitted during execution.

   • During execution: shows placeholder "Step DAG will appear as execution progresses"
   • After completion: renders full Step DAG from trace.json
   • Adding step event instrumentation deferred to future phase (Phase 16b)

2. No pause/replan behavior: Strictly read-only; no pause button, no resume button, no human input forms, no replan execution, no blocking for input.

3. Future HITL fields are inert: All HITL fields default to None/False and do not affect current execution.

Acceptance criteria

• New "⚡ Live Trace Flow" tab appears at position 5 (after Time-Travel Replay)
• All subsequent tab indices updated (6→Checkpoint, 7→Guided Rerun, 8→Rerun Comparison, etc.)
• ACTION_STORY_MAP with 12 action mappings implemented
• 9 builder functions created with future HITL field support
• 8 UI sections rendered on page
• Visual Step DAG shows step_index + story_label as primary sequence
• Agent names separated into overlay (not main DAG)
• Final Trace Handoff provides buttons to existing pages
• Safe auto-refresh only while status is "live"
• No subprocess/CLI/runner calls from new page
• 24 builder tests pass
• 2 page tests pass (13 skipped file content checks)
• Existing page tests still pass (44 tests)
• Safety scan passes (no prohibited patterns)
• Compilation check passes
• PHASE16_LIVE_TRACE_FLOW.md documentation created
• DECISIONS.md includes DEC-020
• IMPLEMENTATION_LEDGER.md updated (Phase Status table + Phase 16 section)
• Overview.md updated (Phase 16 status section)

Deferred work

1. Phase 16b: Step event instrumentation in runner to emit step_started, step_completed, step_failed events during execution for live partial DAG visibility.
2. Phase 17: Human-in-the-loop pause/replan implementation (activate HITL fields, add pause detection, human input forms, replan submission, resume trigger).

---

Phase 17 — Guided Trace Flow with Planner Checkpoint HITL

• Started: 2026-06-27
• Completed: 2026-06-27
• Current branch: feature/observatory-phase-17-guided-trace-hitl

Objective

Implement a new guided live page and runner/service HITL flow that pauses only after planner for planner_review, supports approve/revise/cancel decisions, persists checkpoint/intervention artifacts, and resumes execution safely with timeout auto-approve.

Files created

1. src/data_agent_baseline/domain/intervention_models.py
2. src/data_agent_baseline/application/hitl_checkpoint_controller.py
3. src/data_agent_baseline/observatory/guided_trace_flow_builders.py
4. src/data_agent_baseline/observatory/guided_trace_flow_page.py
5. tests/unit/application/test_hitl_checkpoint_controller.py
6. tests/unit/langgraph_agent/test_guided_planner_checkpoint.py
7. tests/unit/observatory/test_guided_trace_flow_builders.py
8. tests/test_observatory_guided_trace_flow_page.py

Files modified

1. src/data_agent_baseline/domain/__init__.py
2. src/data_agent_baseline/application/run_execution_service.py
3. src/data_agent_baseline/langgraph_agent/state.py
4. src/data_agent_baseline/langgraph_agent/nodes.py
5. src/data_agent_baseline/langgraph_agent/graph.py
6. src/data_agent_baseline/langgraph_agent/runner.py
7. src/data_agent_baseline/observatory/app.py
8. docs/implementation/DECISIONS.md (added DEC-021)
9. docs/implementation/IMPLEMENTATION_LEDGER.md (this file)
10. Overview.md
11. tests/unit/application/test_run_execution_service.py

Implementation summary

1. Added Phase 17 intervention models: CheckpointPolicy, CheckpointRecord, InterventionRecord, InterventionDecision, and InterventionDelta.
2. Added HITLCheckpointController for checkpoint creation, persistence, blocking wait, decision submission, idempotent duplicate rejection, and delta persistence.
3. Added guided policy fields to AgentState and threaded guided metadata from RunSpec through RunExecutionService into the LangGraph runner.
4. Enforced max_workers=1 for guided runs in RunExecutionService.
5. Updated Guided Live Trace Flow start controls to use a single Start Run button with a default-selected Guided Run (HITL) checkbox; unchecked launches run from the same page without planner-review checkpoints.
6. Implemented pause-after-planner in graph via _maybe_handle_planner_checkpoint(...):
   • create checkpoint and persist,
   • emit checkpoint_reached + human_review_requested,
   • emit checkpoint_timeout_started and wait up to timeout_seconds=60,
   • apply approve/revise/cancel behavior,
   • on timeout: persist system timeout intervention (decision=timeout_auto_approve, submitted_by=system_timeout), emit checkpoint_timed_out + checkpoint_auto_approved + task_resumed,
   • emit HITL lifecycle events,
   • persist revised plan and intervention delta for revise path.
7. Extended planner prompt path to incorporate human steering during guided replan while preserving objective and prior work.
8. Extended trace.json output with additive hitl block containing original plan, intervention, revised plan, and delta metadata/path.
9. Added new 🧭 Guided Live Trace Flow (HITL) page and tab registration without changing existing ⚡ Live Trace Flow implementation.

Commands executed and outcomes

1. python -m compileall -q src tests -> pass
2. pytest -q tests/unit/application/test_background_run_controller.py -> 10 passed
3. pytest -q tests/unit/application/test_run_execution_service.py -> 16 passed
4. pytest -q tests/unit/application/test_hitl_checkpoint_controller.py -> 5 passed
5. pytest -q tests/unit/langgraph_agent/test_guided_planner_checkpoint.py -> 4 passed
6. pytest -q tests/unit/observatory/test_guided_trace_flow_builders.py -> 5 passed
7. pytest -q tests/test_observatory_guided_trace_flow_page.py -> 6 passed
8. pytest -q tests/test_observatory_live_trace_flow_page.py -> 11 passed, 25 skipped
9. pytest -q tests/test_observatory_dag_builder.py -> 3 passed
10. pytest -q tests/test_observatory_replay_builder.py -> 1 passed
11. pytest -q tests/test_observatory_checkpoint_page.py -> 12 passed
12. pytest -q tests/test_observatory_rerun_plan_page.py -> 15 passed
13. pytest -q tests/test_observatory_rerun_comparison_page.py -> 13 passed
14. pytest -q tests/test_observatory_*.py -> 37 failed, 99 passed, 25 skipped (existing broad-suite issues outside Phase 17 change set)
15. Safety scan:
    • grep -R "subprocess\|os.system\|dabench\|run_langgraph_single_task\|evaluate_run_v2" src/data_agent_baseline/observatory src/data_agent_baseline/application tests
    • Result: no prohibited execution calls introduced in Guided Trace Flow page; expected matches remain in docs/tests and service-layer delegation.

Acceptance criteria status

• New guided tab/page added (🧭 Guided Live Trace Flow (HITL))
• Existing ⚡ Live Trace Flow kept intact
• Guided run policy is explicit and set from run start
• Guided page can launch with or without HITL from the same start panel
• Planner-only checkpoint (planner_review) implemented
• One unresolved checkpoint per task enforced
• Approve path persists + resumes without replan
• Revise path persists instruction, replans, persists revised plan + delta
• Cancel path implemented with explicit TaskStatus.CANCELLED and task_cancelled event
• Checkpoint/intervention/delta persisted under task directory
• HITL event emission added to run event stream
• timeout auto-approve policy implemented (timeout_seconds=60, default_action_on_timeout=continue)
• Timeout events emitted (checkpoint_timeout_started, checkpoint_timed_out, checkpoint_auto_approved, task_resumed)
• Browser refresh can redisplay pending checkpoint from artifacts while process is alive
• Focused test matrix passes

Known limitations

1. Durable process-restart resume of waiting execution thread is not implemented.
2. HITL scope is intentionally single checkpoint type (planner_review) after planner only.
3. Broad wildcard observatory test sweep includes pre-existing failing tests outside Phase 17 scope.

---