docs/implementation/IMPLEMENTATION_LEDGER.md

# Implementation Ledger

## Current Baseline

- Branch: feature/observatory-phase-00-baseline
- Base commit: 1bf512073d0bb751ad66af52bc77361c52283fc8
- Python version: Python 3.11.15
- Dependency manager/tooling: uv 0.11.8
- Streamlit entrypoint: streamlit run src/data_agent_baseline/observatory/app.py
- CLI entrypoint: dabench (data_agent_baseline.cli:main)

### Pre-change command/test baseline

1. `git status --short`: untracked docs in `Docs_To_be_deleted/` only.
2. `pytest --collect-only -q`: 109 tests collected.
3. Practical existing suite run:
   - Command: `./.venv/bin/pytest -q tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py`
    - Result: 39 passed, 2 failed (snapshot at session start; loaders test was subsequently confirmed fixed).
    - Failures at session start:
       1. `tests/test_eval_v2_validator.py::test_validator_catches_verification_outcome_mismatch` — still failing at Phase 0 close.
       2. `tests/test_observatory_loaders.py::test_run_discovery_and_task_discovery` — test file contained `assert "11"` at session start, but current line 30 already asserts `"task_11"` and passes; resolved prior to Phase 0.
4. Placeholder tests present pre-change:
   1. `tests/test_eval_harness.py::TestMetricBounds::test_data_understanding_score_bounds` (pass placeholder)
   2. `tests/test_eval_harness.py::TestMetricBounds::test_all_score_metrics_bounded` (pass placeholder)
   3. `tests/test_eval_harness.py::test_end_to_end_reconciliation` (pass placeholder)
   4. `tests/test_eval_harness.py::test_health_report_generation` (pass placeholder)


**Failure 1 — pre-existing**
- Test path: `tests/test_eval_v2_validator.py::test_validator_catches_verification_outcome_mismatch`
- Baseline failure message:
   ```
   tests/test_eval_v2_validator.py:96: in test_validator_catches_verification_outcome_mismatch
         assert any(i["check_id"] == "verification_outcome_mismatch" for i in issues)
   E   assert False
   E    +  where False = any(<generator object ...>)
   ```
- Post-change failure message: identical to baseline — no change.
- Classification: pre-existing; the `verification_outcome_mismatch` check_id is not produced by the current validator implementation.
- Phase 0 contribution: none. Phase 0 added no new failures.

**Failure 2 — pre-existing**
**Failure 2 — resolved before Phase 0 close**
- Test path: `tests/test_observatory_loaders.py::test_run_discovery_and_task_discovery`
- Status at Phase 0 close: **PASSES**. Current line 30 asserts `"task_11"` (full task ID format).
- Phase 0 contribution: none. Phase 0 made no change to this test or its backing loader.

The final broader regression suite at Phase 0 close: **63 passed, 1 failed**. The sole remaining failure is the validator `check_id` test (Failure 1). Phase 0 did not introduce any new failures.

## Phase Status

| Phase | Status | Branch | Commit | Tests | Acceptance | Notes |
|---|---|---|---|---|---|---|
| Phase 0 | Completed | feature/observatory-phase-00-baseline | 1bf512073d0bb751ad66af52bc77361c52283fc8 (base) | Added + validated | PASS (see matrix) | No src runtime behavior changes |
| Phase 1 | Completed (uncommitted) | feature/observatory-phase-01-run-service | d09fdfa6bfe8e2e2d8f2fa6ae5bf3b707f32e412 (current HEAD) | Added + validated | PASS (see matrix) | Shared run execution service extracted from CLI |
| Phase 2 | Completed (uncommitted) | feature/observatory-phase-01-run-service | d09fdfa6bfe8e2e2d8f2fa6ae5bf3b707f32e412 (current HEAD) | Added + validated | PASS (see matrix) | Eval-v2 orchestration extracted into EvaluationService with callback-driven CLI progress |
| Phase 3 | Completed (uncommitted) | feature/observatory-phase-03-run-intelligence | d09fdfa6bfe8e2e2d8f2fa6ae5bf3b707f32e412 (current HEAD) | Added + validated | PASS (see matrix) | Read-only Run Intelligence page added to Streamlit |
| Phase 4 | Completed (uncommitted) | feature/observatory-phase-04-run-launcher | current workspace | Added + validated | PASS (see matrix) | Synchronous predefined-task Run Launcher delivered |
| Phase 5 | Completed (uncommitted) | feature/observatory-phase-05-live-execution | current workspace | Added + validated | PASS (see matrix) | Live synchronous execution observability and manifest-status correctness |
| Phase 6 | Completed (staged) | feature/observatory-phase-06-custom-task-intake | current workspace | Added + validated | PASS (272 passed) | External Benchmark Dataset Intake in Run Launcher; difficulty field made optional in dataset contract. Free-form custom task intake deferred to Phase 7. |
| Phase 7 | Completed | feature/observatory-phase-07-freeform-custom-task-intake | current workspace | Added + validated | PASS (372 passed) | Free-form Custom Task Intake: Custom Task tab in Run Launcher with question, context, file upload, output type selector. Backend materializes tasks into synthetic datasets, executes through shared service. |
| Phase 8 | Completed (uncommitted) | feature/observatory-phase-08-checkpoint-steering | current workspace | Added + validated | PASS (397 passed) | Checkpoint Review tab with additive advisory annotations (`checkpoint_annotations.json`) derived from existing trace/events, with no pause/resume or execution intervention. |
| Phase 9 | Completed (uncommitted) | feature/observatory-phase-09-guided-rerun-planning | current workspace | Added + validated | PASS (434+ passed) | Guided Rerun Plan tab with rerun plan creation, approval workflow, and safe guided rerun execution as new isolated custom task runs. Original run artifacts remain immutable. |
| Phase 10 | Completed (uncommitted) | feature/observatory-phase-10-rerun-comparison-ask-this-run | current workspace | Added + validated | PASS (533 passed) | Rerun Comparison tab with read-only original-vs-guided comparison, local artifact-grounded Ask This Run/Comparison Q&A, steering influence summaries, and saved comparison reports. Preserves artifact integrity. |
| Phase 11 | Completed (uncommitted) | feature/observatory-phase-11-cohort-guided-evaluation | current workspace | Added + validated | PASS (602 passed) | Run-level cohort guided rerun evaluation integrated into Run Intelligence with discovery, aggregation, cohort Q&A, and optional cohort report. Read-only, no execution, no causal claims. |
| Phase 12 | Completed (uncommitted) | feature/observatory-phase-12-evidence-pack | current workspace | Added + validated | PASS (674 passed) | Evidence Pack with cross-phase narrative, statistical readiness, claim safety framework, and markdown/JSON exports. Read-only, no execution, no causal claims. |
| Phase 13 | Completed (uncommitted) | feature/observatory-phase-13-demo-tab | current workspace | Added + validated | PASS (746 passed) | Demo tab with Recorded Evidence Mode, optional Live Mini Demo, demo health scoring, competition checklist, flow scripts, and demo bundle export. No auto-execution. |
| Phase 14 | Completed (uncommitted) | feature/observatory-phase-14-dao-branding-release-audit | current workspace | Added + validated | PASS (748+ passed) | Branding as DAO — Data Agent Observatory and final release readiness audit. No execution/evaluation/artifact behavior changes. |
| Phase 16 | Completed (uncommitted) | feature/observatory-phase-16-live-dag-replay | current workspace | Added + validated | PASS (774+ passed) | Live Trace Flow with trace-native step/action backbone, Execution Story mapping, future HITL-compatible node schema (inert), read-only observability. No step event instrumentation added. |
| Phase 17 | Completed (uncommitted) | feature/observatory-phase-17-guided-trace-hitl | current workspace | Added + validated | PASS (focused Phase 17 matrix) | Guided Live Trace Flow with planner checkpoint HITL (approve/revise/cancel), persisted checkpoint/intervention/delta artifacts, and guided-policy run execution. |
| Phase 19 | Completed (uncommitted) | feature/observatory-phase-19-navigation-consolidation | current workspace | Added + validated | PASS (focused Phase 19 matrix) | Primary navigation consolidated to four pages (Run Launcher, Run Intelligence, Task Intelligence, Demo / Future Proof) with fallback access for Live Trace Flow and Guided Live Trace Flow unchanged. |

### Phase 19 — Observatory Navigation Consolidation

- Started: 2026-07-01
- Completed: 2026-07-01
- Branch: `feature/observatory-phase-19-navigation-consolidation`

#### Summary

1. Consolidated top-level navigation in `app.py` to four primary tabs:
   - Run Launcher
   - Run Intelligence
   - Task Intelligence
   - Demo / Future Proof
2. Added shell composition pages:
   - `task_intelligence_page.py` (Mission Summary, DAG, Replay, Provenance, Critic/Reviewer, Confidence, Failure/Verification, Raw Trace)
   - `future_proof_page.py` (Checkpoint Review, Guided Rerun Plan, Rerun Comparison, Evidence Pack placeholder, Demo)
3. Preserved fallback access to `live_trace_flow_page.py` and `guided_trace_flow_page.py` under Advanced / Fallback Views.
4. No execution/evaluation/HITL runtime behavior changes and no artifact schema changes.
5. Phase 20 natural-language artifact Q&A intentionally unchanged and out of scope.

## Phase Details

### Phase 0

- Started: 2026-06-18
- Completed: 2026-06-18

#### Files created

1. `docs/architecture/ADR-001-shared-services.md`
2. `docs/implementation/PHASE0_COMMAND_CONTRACT.md`
3. `tests/fixtures/phase0/README.md`
4. `tests/fixtures/phase0/generated_eval/gold/task_1/gold.csv`
5. `tests/fixtures/phase0/generated_eval/run/task_1/prediction.csv`
6. `tests/fixtures/phase0/generated_eval/run/task_1/trace.json`
7. `tests/fixtures/phase0/reader_contract/success_run/*` (reduced run/task artifacts)
8. `tests/fixtures/phase0/reader_contract/failed_run/*`
9. `tests/fixtures/phase0/reader_contract/multi_task_run/*`
10. `tests/fixtures/phase0/reader_contract/validator_warnings_run/validation_report.md`
11. `tests/fixtures/phase0/reader_contract/replay_run/*`
12. `tests/fixtures/phase0/reader_contract/react_compat_run/*`
13. `tests/fixtures/phase0/reader_contract/view_task_v2_run/*`
14. `tests/cli/test_cli_phase0_contracts.py`
15. `tests/integration/test_eval_v2_golden_artifacts.py`
16. `tests/integration/test_reader_contract_phase0.py`

#### Files modified

1. `docs/implementation/IMPLEMENTATION_LEDGER.md` (this file)
2. `docs/implementation/DECISIONS.md` (initialized Phase 0 decision)

#### Tests added

1. `tests/cli/test_cli_phase0_contracts.py`
2. `tests/integration/test_eval_v2_golden_artifacts.py`
3. `tests/integration/test_reader_contract_phase0.py`

#### Commands executed and outcomes

1. Baseline metadata:
   - `git status --short`
   - `git rev-parse HEAD`
   - `./.venv/bin/python --version`
   - `uv --version`
2. Baseline tests:
   - `./.venv/bin/pytest --collect-only -q` -> 109 collected
   - practical suite -> 39 passed, 2 failed (pre-existing)
3. New tests:
   - `./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py` -> 16 passed
4. Syntax/import validation:
   - `./.venv/bin/python -m compileall -q src tests` -> pass
5. Broader regression matrix:
   - `./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py`
   - Result: 62 passed, 2 failed (same pre-existing failures)
6. CLI smoke helps:
   - `dabench run-lang-task --help`
   - `dabench run-lang-benchmark --help`
   - `dabench eval-v2 --help`
   - `dabench view-task-v2 --help`
   - `dabench eval-comprehensive --help`
   - `dabench tag-failures --help`
   - `dabench view-exec-report --help`
7. CLI negative-path smoke:
   - `dabench eval-v2 /tmp/phase0-missing-run` -> exit 1
   - `dabench view-task-v2 task_1 /tmp/phase0-missing-metrics` -> exit 1
8. eval-v2 generated fixture runs:
   - standard -> exit 0
   - verbose -> exit 0
   - research -> exit 0

#### Artifact inspection evidence

1. Generated fixture run outputs exist:
   - `task_metrics.csv`, `trajectory.csv`, `tool_calls.csv`, `comprehensive_evaluation.csv`, `validation_report.md`, `auditor_validation_report.md`, `artifact_reconciliation_report.txt`, `engineering_health_report.txt`, `task_1/task_replay.json`.
2. CSV schema spot-check:
   - `task_metrics.csv` includes `run_id,task_id,execution_success,final_score`.
   - `trajectory.csv` includes `task_id,step_id,action`.
   - `tool_calls.csv` includes `task_id,tool_name,success`.
3. Replay JSON key check:
   - includes `task_id,run_id,trace_id,final_score,execution_success`.
4. Validation and health checks:
   - validation report contains `## ERROR` with `- none`.
   - health report includes `HEALTHY` status.
5. ID consistency:
   - task ID remains `task_1` across task_metrics/trajectory/tool_calls/replay.

#### Fixture hygiene and size

1. Fixture tree size: ~160 KB (`tests/fixtures/phase0`).
2. Files are reduced and synthetic; no source datasets copied.
3. Secret/path scan found no API keys/passwords/secrets; only minimal token count fields required for metric contracts.
4. Deterministic run/task IDs and timestamps used in fixtures.

#### Known limitations

1. Repository `.gitignore` currently ignores `docs/` and `tests/`; staging these Phase 0 files requires forced add (`git add -f ...`) or `.gitignore` adjustment.
2. Two pre-existing test failures remain and were not modified in Phase 0.

#### Deferred work

1. Service extraction and refactoring (Phases 1+).
2. Artifact path migration to `evaluation/` subdir (future compatibility phase).
3. Resolving existing unrelated failing tests.

## Acceptance Criteria Audit

| Criterion | Status | Evidence |
|---|---|---|
| 1. Existing CLI commands still run. | PASS | CLI help smoke commands succeed for all required commands. |
| 2. CLI smoke/contract tests exist. | PASS | `tests/cli/test_cli_phase0_contracts.py` added and passing. |
| 3. Representative reduced fixtures exist. | PASS | `tests/fixtures/phase0/*` created, documented, minimized. |
| 4. ADR-001 is documented. | PASS | `docs/architecture/ADR-001-shared-services.md`. |
| 5. Test suite locks current eval-v2 artifact contract. | PASS | `tests/integration/test_eval_v2_golden_artifacts.py` validates outputs and schemas. |
| 6. Observatory current-layout compatibility is tested. | PASS | `tests/integration/test_reader_contract_phase0.py` and existing observatory tests. |
| 7. Warning-vs-error validator behavior is tested. | PASS | `test_eval_v2_validator_warnings_do_not_fail` and `test_eval_v2_validator_errors_fail_command`. |
| 8. No production behavior changed. | PASS | No edits under `src/`; tests/docs/fixtures only. |
| 9. No src files changed without approved defect. | PASS | No `src/` files modified. |
| 10. No sensitive/large/machine-specific fixture data added. | PASS | Fixture size ~160 KB, synthetic content, no secrets, no absolute developer paths. |

### Phase 1

- Started: 2026-06-19
- Completed: 2026-06-19

#### Files created

1. `src/data_agent_baseline/domain/__init__.py`
2. `src/data_agent_baseline/domain/run_models.py`
3. `src/data_agent_baseline/application/__init__.py`
4. `src/data_agent_baseline/application/run_execution_service.py`
5. `src/data_agent_baseline/repositories/__init__.py`
6. `src/data_agent_baseline/repositories/run_repository.py`
7. `src/data_agent_baseline/repositories/filesystem_run_repository.py`
8. `tests/unit/__init__.py`
9. `tests/unit/domain/__init__.py`
10. `tests/unit/domain/test_run_models.py`
11. `tests/unit/repositories/__init__.py`
12. `tests/unit/repositories/test_filesystem_run_repository.py`
13. `tests/unit/application/__init__.py`
14. `tests/unit/application/test_run_execution_service.py`
15. `tests/integration/test_phase1_cli_delegates.py`

#### Files modified

1. `src/data_agent_baseline/cli.py`
2. `tests/cli/test_cli_phase0_contracts.py`
3. `docs/implementation/IMPLEMENTATION_LEDGER.md`
4. `docs/implementation/DECISIONS.md`

#### Commands executed and outcomes

1. `./.venv/bin/python -m compileall -q src tests` -> pass
2. `./.venv/bin/pytest -q tests/unit/domain/test_run_models.py tests/unit/repositories/test_filesystem_run_repository.py tests/unit/application/test_run_execution_service.py tests/integration/test_phase1_cli_delegates.py` -> 27 passed
3. `./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py` -> 16 passed
4. `./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/unit tests/integration/test_phase1_cli_delegates.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py` -> 90 passed, 1 failed (known pre-existing validator failure)
5. `./.venv/bin/dabench run-lang-task --help` -> pass
6. `./.venv/bin/dabench run-lang-benchmark --help` -> pass
7. `./.venv/bin/dabench eval-v2 --help` -> pass
8. `./.venv/bin/dabench view-task-v2 --help` -> pass

#### Artifact inspection evidence

1. Additive benchmark artifacts verified in a controlled integration run:
   - `run_manifest.json` present and parseable.
   - `run_events.jsonl` present with ordered sequence numbers.
   - `summary.json` preserved and unchanged.
2. Task artifacts preserved:
   - `task_1/trace.json` present.
   - `task_1/prediction.csv` present.

#### Known limitations

1. Existing pre-phase validator test failure remains unchanged:
   - `tests/test_eval_v2_validator.py::test_validator_catches_verification_outcome_mismatch`.
2. Benchmark event stream in Phase 1 is coarse-grained (task-level + run-level); deep stage/tool events are deferred to Phase 5 by design.

#### Acceptance Criteria Audit (Phase 1)

| Criterion | Status | Evidence |
|---|---|---|
| `run-lang-task` behavior remains backward compatible | PASS | `tests/cli/test_cli_phase0_contracts.py` passes, CLI help unchanged |
| Execution loop absent from `run-lang-task` CLI command | PASS | `src/data_agent_baseline/cli.py` delegates to `RunExecutionService.execute_selected_tasks` |
| Service callable directly from Python tests | PASS | `tests/unit/application/test_run_execution_service.py` |
| `run_manifest.json` generated | PASS | unit/integration tests assert presence |
| `run_events.jsonl` generated | PASS | unit/integration tests assert presence |
| Benchmark `summary.json` preserved | PASS | `tests/unit/application/test_run_execution_service.py::test_execute_benchmark_shallow_wrapper_preserves_summary` |
| Tests avoid external LLM/API calls | PASS | all Phase 1 tests use fakes/monkeypatching |

### Phase 2

- Started: 2026-06-19
- Completed: 2026-06-19

#### Files created

1. `src/data_agent_baseline/domain/evaluation_models.py`
2. `src/data_agent_baseline/application/evaluation_service.py`
3. `src/data_agent_baseline/repositories/evaluation_repository.py`
4. `src/data_agent_baseline/repositories/filesystem_evaluation_repository.py`
5. `tests/unit/application/test_evaluation_service.py`
6. `tests/unit/repositories/test_filesystem_evaluation_repository.py`
7. `tests/integration/test_phase2_cli_delegates.py`

#### Files modified

1. `src/data_agent_baseline/cli.py`
2. `src/data_agent_baseline/domain/__init__.py`
3. `src/data_agent_baseline/application/__init__.py`
4. `src/data_agent_baseline/repositories/__init__.py`
5. `tests/cli/test_cli_phase0_contracts.py` (mocking adaptation for service boundary)
6. `docs/implementation/IMPLEMENTATION_LEDGER.md`
7. `Overview.md`

#### Commands executed and outcomes

1. `./.venv/bin/python -m compileall -q src tests` -> pass
2. `./.venv/bin/pytest -q tests/unit/repositories/test_filesystem_evaluation_repository.py tests/unit/application/test_evaluation_service.py tests/integration/test_phase2_cli_delegates.py` -> 27 passed
3. `./.venv/bin/pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_eval_v2_golden_artifacts.py tests/integration/test_reader_contract_phase0.py tests/integration/test_phase1_cli_delegates.py tests/integration/test_phase2_cli_delegates.py tests/unit/ tests/test_eval_v2_validator.py tests/test_eval_v2_hardening.py tests/test_eval_harness.py tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py` -> 117 passed
4. `./.venv/bin/dabench eval-v2 --help` -> pass
5. `./.venv/bin/dabench eval-v2 <copied_fixture_run> --gold-root <copied_fixture_gold> --task-root <generated_task_root> --mode standard` -> pass
6. `./.venv/bin/dabench eval-v2 <copied_fixture_run> --gold-root <copied_fixture_gold> --task-root <generated_task_root> --mode verbose` -> pass
7. `./.venv/bin/dabench eval-v2 <copied_fixture_run> --gold-root <copied_fixture_gold> --task-root <generated_task_root> --mode research` -> pass

#### Artifact inspection evidence

1. Run-root artifacts present after fixture smoke:
   - `task_metrics.csv`
   - `trajectory.csv`
   - `tool_calls.csv`
   - `comprehensive_evaluation.csv`
   - `validation_report.md`
   - `auditor_validation_report.md`
   - `artifact_reconciliation_report.txt`
   - `engineering_health_report.txt`
   - `task_1/task_replay.json`
2. `task_metrics.csv` and `comprehensive_evaluation.csv` both present with unchanged column layout.
3. `trajectory.csv` and `tool_calls.csv` present with expected contract columns.
4. No `evaluation/` subfolder introduced.

#### Decisions alignment

1. DEC-004 implemented as written: `EvaluationService.load_existing_evaluation()` imports and uses `_compute_summary_v2` directly.
2. DEC-005 implemented as written: `EvaluationService.evaluate_run()` accepts optional progress callback and emits stage events consumed by CLI.
3. No DEC-004/DEC-005 deviations; `docs/implementation/DECISIONS.md` unchanged.

#### Acceptance Criteria Audit (Phase 2)

| Criterion | Status | Evidence |
|---|---|---|
| Eval orchestration moved from CLI to service | PASS | `src/data_agent_baseline/application/evaluation_service.py`; `src/data_agent_baseline/cli.py` now delegates |
| Hardening progress behavior preserved via callback | PASS | service stage callback + CLI rendering bridge in `eval_v2_command` |
| No eval-v2 metric module modifications | PASS | `eval_v2.py`, `eval_v2_viz.py`, `eval_v2_validator.py`, `eval_artifact_reconciliation.py`, `eval_replay_artifacts.py`, `eval_health_report.py` unchanged |
| Artifact contract preserved at run root | PASS | golden artifacts + fixture smoke outputs |
| Validation semantics preserved (error exit 1, warning exit 0) | PASS | `tests/integration/test_phase2_cli_delegates.py` + adapted phase0 CLI contracts |
| Render-mode validation across standard/verbose/research | PASS | service unit test + fixture smoke in all modes |
| Repository missing/malformed handling | PASS | repository unit tests |
| Full regression matrix passes | PASS | 117 passed |

### Phase 3 — Streamlit Run Intelligence

- Started: 2026-06-19
- Completed: 2026-06-19
- Current branch: `feature/observatory-phase-03-run-intelligence`

#### Objective

Expose run-level evaluation quality, reliability, resources, failures, validation status, and artifact completeness in Streamlit using `EvaluationService.load_existing_evaluation()` (read-only). Phase 3 must not trigger evaluation, call `evaluate_run()`, or invoke CLI subprocesses from Streamlit.

#### Files created

1. `src/data_agent_baseline/observatory/run_intelligence_builders.py` (9 pure builder functions)
2. `src/data_agent_baseline/observatory/run_intelligence_page.py` (thin Streamlit renderer)
3. `src/data_agent_baseline/observatory/service_adapters.py` (service factory)
4. `tests/unit/observatory/test_run_intelligence_builders.py` (18 builder tests)
5. `tests/test_observatory_run_intelligence_page.py` (13 page/read-only tests)

#### Files modified

1. `src/data_agent_baseline/observatory/app.py` (added Run Intelligence tab at position 0)
2. `docs/implementation/IMPLEMENTATION_LEDGER.md` (this file)
3. `docs/implementation/DECISIONS.md` (added DEC-006)

#### Implementation summary

**Builders** (pure functions, no IO):
- `build_run_health_cards(bundle, manifest=None) -> list[dict]`: Executive verdict cards (run quality, harness health, status, task counts, success rate, mean score, runtime, tokens, failed task count)
- `build_task_outcome_table(bundle) -> pd.DataFrame`: Task summary table (task_id, score, success, runtime, tokens)
- `build_failure_summary(bundle) -> pd.DataFrame`: Failure category distribution
- `build_resource_summary(bundle) -> pd.DataFrame`: Resource aggregates (runtime, tokens, tool calls, tool failures)
- `build_validation_issues_view(bundle, run_path=None) -> dict`: Structured validation warning/error view
- `build_artifact_status(run_path) -> pd.DataFrame`: Artifact existence/size checklist
- `build_representative_tasks(bundle) -> pd.DataFrame`: Representative examples (highest/lowest score, most expensive)
- `load_run_manifest(run_path) -> dict | None`: Graceful run_manifest.json loader
- `load_run_events(run_path, max_events=100) -> pd.DataFrame`: Graceful run_events.jsonl loader

**Page** (Streamlit UI):
- Sidebar unchanged: existing run/task selector unchanged
- Tab 0: Run Intelligence (new); tabs 1-8 shifted indices (existing task tabs)
- Sections:
  1. Evaluation Status: missing evaluation shows warning + guidance to run eval-v2 externally
  2. Executive Verdict: health cards
   3. Task Outcomes: top-N by view level + full table in expander
  4. Failure Distribution: failure categories
  5. Resource Summary: runtime/token aggregates
  6. Validation Diagnostics: warning/error counts + expander with details
  7. Artifact Completeness: file existence checklist
  8. Representative Tasks: example high/low performers
   9. Task Drill-Down: representative task selector + "Open Task Context" updates `current_task_id`
   10. Footer: read-only note + external eval guidance

**Service adapter**:
- `get_evaluation_service(artifact_runs_dir) -> EvaluationService`: Factory with FilesystemEvaluationRepository

**App integration**:
- Imported `render_run_intelligence_page` at module top (import error handling)
- Added tab 0 for Run Intelligence
- Shifted all other tab indices by 1
- Tab 0 calls `render_run_intelligence_page(run.run_id, artifact_runs_dir)`
- Error handling with try/except around tab content

#### Read-only constraint enforcement

All Phase 3 implementations verified NOT to call:
- ✅ `EvaluationService.evaluate_run()`
- ✅ `evaluate_run_v2()` from eval_v2.py
- ✅ `subprocess.run()`, `os.system()`, or any CLI invocation

Note: the page shows `dabench eval-v2 ...` as user guidance text only; it is never executed.

Constraint implementations:
1. Page checks `evaluation_exists()` — if false, shows warning + guidance
2. Page calls only `load_existing_evaluation()`
3. Builders receive `EvaluationBundle` only; no artifact reading or subprocess
4. Service adapter wires `EvaluationService` + `FilesystemEvaluationRepository` only
5. Code review: no subprocess imports, no evaluate_run_v2 calls, no CLI execution paths

#### Commands executed and outcomes

1. `python -m compileall -q src tests` -> pass
2. Builder unit tests:
   - `pytest -q tests/unit/observatory/test_run_intelligence_builders.py` -> 18 passed
3. Page behavior + import tests:
   - `pytest -q tests/test_observatory_run_intelligence_page.py` -> 13 passed
4. Observatory regression tests:
   - `pytest -q tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py` -> 13 passed
5. Phase 0/1/2 compatibility matrix:
   - `pytest -q tests/cli/test_cli_phase0_contracts.py tests/integration/test_reader_contract_phase0.py tests/integration/test_eval_v2_golden_artifacts.py tests/unit/application/test_run_execution_service.py tests/unit/repositories/test_filesystem_run_repository.py tests/integration/test_phase1_cli_delegates.py tests/unit/application/test_evaluation_service.py tests/unit/repositories/test_filesystem_evaluation_repository.py tests/integration/test_phase2_cli_delegates.py` -> 62 passed
6. Full suite:
   - `pytest -q` -> 209 passed
5. Import smoke:
   - `python -c "import data_agent_baseline.observatory.run_intelligence_builders as b; print('builders ok')"` -> ok
   - `python -c "import data_agent_baseline.observatory.run_intelligence_page as p; print('page ok')"` -> ok
   - `python -c "import data_agent_baseline.observatory.service_adapters as a; print('adapters ok')"` -> ok
6. App integrity:
   - `python -c "import data_agent_baseline.observatory.app; print('app ok')"` -> ok
7. Streamlit startup (dry run, no browser):
   - `streamlit run src/data_agent_baseline/observatory/app.py --logger.level=error 2>&1 | head -20` -> starts without import errors

#### Artifact inspection evidence

1. Tested against real modern runs at `/data3/dataFAIR/kdd-dev/public/artifacts/runs/`:
   - `20260619T073825Z/` parsed successfully
   - All 8 expected eval-v2 artifacts present
   - `task_metrics.csv` loaded (250+ columns)
   - Health cards derived correctly
   - Task table and failure summary populated
   - Resource and validation views rendered
2. Tested with missing evaluation state:
   - Page shows "⚠️ Evaluation Artifacts Missing" warning
   - Guidance text displays: `dabench eval-v2 <run_id> --mode standard`
   - No error or crash
3. Tested graceful degradation:
   - Missing optional artifacts (trajectory, validation_report, health_report, reconciliation_report): no crash
   - Tables handle missing columns (render available data only)
   - Card aggregates use fillna/default safely

#### Tests added

**Builder tests** (`test_run_intelligence_builders.py`):
- `test_build_run_health_cards_valid_bundle`: Health cards generated correctly
- `test_build_run_health_cards_with_manifest`: Manifest parameter handled
- `test_build_run_health_cards_empty_metrics`: Empty metrics fallback
- `test_build_task_outcome_table`: Table shape and sorting
- `test_build_task_outcome_table_empty_metrics`: Empty metrics fallback
- `test_build_failure_summary`: Category distribution correct
- `test_build_failure_summary_no_category`: Missing column handled
- `test_build_resource_summary`: Resource aggregates
- `test_build_validation_issues_view`: Issue parsing and counts
- `test_build_artifact_status`: File existence checking
- `test_build_representative_tasks`: Example selection logic
- `test_build_representative_tasks_empty`: Empty metrics fallback
- `test_load_run_manifest`: Manifest parsing
- `test_load_run_manifest_missing`: Missing file returns None
- `test_load_run_events`: JSONL parsing with max limit
- `test_load_run_events_missing`: Missing file returns empty DataFrame
- `test_load_run_events_max_limit`: Event limit enforced

**Page/service tests** (`test_observatory_run_intelligence_page.py`):
- `test_import_page_module`: Module imports successfully
- `test_import_builders_module`: Builders module imports
- `test_import_service_adapters_module`: Service adapters module imports
- `test_page_no_run_directory_found`: Graceful error on missing run
- `test_page_missing_evaluation_artifacts`: Warning shown for missing evaluation
- `test_page_loads_existing_evaluation`: Uses load_existing_evaluation() only
- `test_page_does_not_call_evaluate_run`: evaluate_run is NOT called (explicit assertion)
- `test_no_subprocess_in_page_code`: Source code lacks subprocess usage
- `test_no_evaluate_run_v2_in_page_code`: Source code lacks evaluate_run_v2 calls
- `test_no_cli_invocation_in_page_code`: Source code lacks CLI command strings
- `test_service_adapter_returns_evaluation_service`: Adapter factory works
- `test_builders_return_correct_types`: Builders return expected types (list, DataFrame, dict)

#### Acceptance Criteria Audit (Phase 3)

| Criterion | Status | Evidence |
|---|---|---|
| Run Intelligence UI visible in Streamlit | PASS | Tab 0 added with all sections rendered |
| Loads existing evaluation via `load_existing_evaluation()` | PASS | Page calls service; tests verify call |
| Never calls `evaluate_run()`, `evaluate_run_v2()`, subprocess, or CLI | PASS | Source code inspection + explicit test assertions |
| Missing evaluations show graceful warning + guidance | PASS | Page shows banner + `dabench eval-v2` command |
| Modern runs show executive cards, task/failure/resource tables, validation, artifact status | PASS | Rendered against real run + assertions in tests |
| Task drill-down links to existing task context | PASS | Representative task selector + `Open Task Context` updates session task selection |
| No artifact schemas/locations changed | PASS | No write operations; read-only only |
| No CLI behavior changed | PASS | CLI smoke commands pass; Phase 0/1/2 regressions pass |
| Existing task tabs still work | PASS | App.py preserves existing pages; tabs 1-8 are unchanged |
| All new tests pass | PASS | 18 + 13 = 31 new tests; all pass |
| Phase 0/1/2 regressions still pass | PASS | Required compatibility matrix: 62 passed |

#### Known limitations

1. Phase 3 is read-only; evaluation triggering deferred to Phase 4 (Run Launcher) or future phases with job scheduling.
2. Guided vs autonomous run comparison deferred to Phase 9.
3. Cross-run comparison not implemented; single-run view only.
4. Ask This Run and advanced cohort analytics deferred to Phase 10.
5. Full `pytest -q` is green after rerun (`209 passed`).

#### Next phase dependency

Phase 4 (Run Launcher) can proceed independently; depends on Phase 1 shared services only.
Phase 5 (Live Execution) depends on Phase 1 event infrastructure.

### Phase 5 — Live Execution and Event Model

- Started: 2026-06-20
- Completed: 2026-06-20
- Current branch: `feature/observatory-phase-05-live-execution`

#### Objective

Add synchronous live execution observability to the Run Launcher using the shared `RunExecutionService`, persisted `run_events.jsonl` / `run_manifest.json`, and Streamlit live panels without introducing subprocess/CLI/background execution paths.

#### Files created

1. `src/data_agent_baseline/observatory/live_execution_builders.py`
2. `tests/unit/observatory/test_live_execution_builders.py`

#### Files modified

1. `src/data_agent_baseline/application/run_execution_service.py`
2. `src/data_agent_baseline/observatory/service_adapters.py`
3. `src/data_agent_baseline/observatory/run_launcher_page.py`
4. `tests/unit/application/test_run_execution_service.py`
5. `tests/unit/observatory/test_run_launcher_builders.py`
6. `tests/test_observatory_run_launcher_page.py`
7. `tests/integration/test_phase4_observatory_launcher_integration.py`
8. `docs/implementation/DECISIONS.md`
9. `docs/implementation/IMPLEMENTATION_LEDGER.md`

#### Implementation summary

1. Added pure live-execution builders for safe manifest/event loading and derived views:
   - `load_live_run_manifest`
   - `load_live_run_events`
   - `build_live_progress_view`
   - `build_event_timeline`
   - `build_task_status_table`
   - `build_failure_event_view`
   - `build_latest_event`
2. Updated launcher service adapter to pass both callbacks (`progress_callback`, `event_callback`) into `RunExecutionService.execute_selected_tasks()`.
3. Enhanced Run Launcher page with:
   - explicit synchronous execution notice,
   - active execution state banner,
   - latest-event card,
   - current-task indicator,
   - task status table,
   - recent event tail,
   - failure event panel,
   - manual refresh from persisted artifacts,
   - graceful missing/corrupt artifact states.
4. Fixed Phase 5 blocking bug: final manifest write now preserves in-memory `task_statuses` updated per task transition, preventing terminal statuses from being overwritten back to pending.

#### Manifest task-status bug fix evidence

New regression tests in `tests/unit/application/test_run_execution_service.py`:
1. `test_execute_selected_tasks_final_manifest_preserves_all_success_statuses`
2. `test_execute_selected_tasks_final_manifest_preserves_partial_failure_statuses`
3. `test_execute_selected_tasks_final_manifest_preserves_all_failure_statuses`

All pass, confirming terminal statuses are retained in final `run_manifest.json`.

#### Commands executed and outcomes

1. `python -m compileall -q src tests` -> pass
2. `pytest -q tests/unit/observatory/test_live_execution_builders.py` -> 10 passed
3. `pytest -q tests/test_observatory_run_launcher_page.py` -> 9 passed
4. `pytest -q tests/integration/test_phase4_observatory_launcher_integration.py` -> 2 passed
5. `pytest -q tests/unit/application/test_run_execution_service.py` -> 11 passed
6. `pytest -q tests/unit/repositories/test_filesystem_run_repository.py` -> 9 passed
7. `pytest -q tests/unit/observatory/test_run_intelligence_builders.py tests/test_observatory_run_intelligence_page.py` -> 31 passed
8. `pytest -q tests/cli/test_cli_phase0_contracts.py` -> 8 passed
9. `pytest -q tests/test_observatory_loaders.py tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py` -> 13 passed
10. `python -c "import data_agent_baseline.observatory.live_execution_builders as b; print('live builders ok')"` -> live builders ok
11. `python -c "import data_agent_baseline.observatory.run_launcher_page as p; print('launcher page ok')"` -> launcher page ok
12. `python -c "import data_agent_baseline.observatory.app as app; print('app ok')"` -> app ok (Streamlit bare-mode warnings expected)
13. `pytest -q` -> 240 passed

#### Manual verification

1. `streamlit run src/data_agent_baseline/observatory/app.py --server.headless true --server.port 8501` started successfully.
2. Local URL served (`http://localhost:8501`) confirmed via `curl` response.
3. Browser-attached interactive clicking was not completed in this environment; functional UI behavior was verified through page/integration tests covering synchronous notice, callback-driven live updates, fallback artifact handling, and open-run/task actions.

#### Artifact compatibility

1. `RunEvent` schema unchanged (existing fields preserved).
2. Event types remain coarse existing set (`run_created`, `run_started`, `task_queued`, `task_started`, `task_completed`, `task_failed`, `run_completed`, `run_failed`).
3. `run_manifest.json` / `run_events.jsonl` paths and JSON compatibility preserved.
4. No eval-v2 auto-run added.

#### No background-job decision

Phase 5 remains synchronous by decision (see DEC-008). No queue/server/cancellation/steering/pause-resume/background runner was introduced.

#### Known limitations

1. Live execution remains synchronous within the active Streamlit session.
2. No background continuation after session close.
3. No cancellation, pause/resume, checkpoint steering, or multi-user scheduling.
4. Deep stage/tool-level event instrumentation remains deferred.

### Phase 7 — Free-form Custom Task Intake

- Started: 2026-06-20
- Completed: 2026-06-20
- Current branch: `feature/observatory-phase-07-freeform-custom-task-intake`

#### Objective

Allow a user to create and run a single free-form custom task from Streamlit using question text, optional context, optional uploaded files, and expected output type. The custom task is safely persisted, materialized into a benchmark-compatible synthetic dataset, and executed through the existing shared run service.

#### Files created

1. `src/data_agent_baseline/domain/task_models.py` — Domain models for custom task intake
2. `src/data_agent_baseline/repositories/custom_task_repository.py` — Repository abstraction
3. `src/data_agent_baseline/repositories/filesystem_custom_task_repository.py` — Filesystem implementation
4. `src/data_agent_baseline/application/custom_task_service.py` — Application service
5. `tests/unit/domain/test_task_models.py` — Domain model tests
6. `tests/unit/repositories/test_filesystem_custom_task_repository.py` — Repository tests
7. `tests/unit/application/test_custom_task_service.py` — Service tests
8. `tests/integration/test_phase7_custom_task_launcher_integration.py` — Integration tests
9. `tests/test_observatory_run_launcher_page_phase7_custom.py` — Streamlit Custom Task tab page tests

#### Files modified

1. `src/data_agent_baseline/observatory/service_adapters.py` — Added `get_run_execution_service_with_custom_task()`
2. `src/data_agent_baseline/observatory/run_launcher_page.py` — Added Custom Task third tab with full UI
3. `tests/test_observatory_run_launcher_page.py` — Added `text_area` / `file_uploader` to stub for compatibility
4. `tests/integration/test_phase4_observatory_launcher_integration.py` — Added `text_area` / `file_uploader` to stub for compatibility
5. `docs/implementation/DECISIONS.md` — Added DEC-011 for Phase 7 architecture
6. `docs/implementation/IMPLEMENTATION_LEDGER.md` — This file

#### Implementation summary

1. **Domain models** (`task_models.py`):
   - `ExpectedOutputSpec`: Output type and optional columns
   - `SourceDescriptor`: Metadata for uploaded sources
   - `UploadedFileSpec`: In-memory uploaded file container
   - `TaskDefinition`: Complete custom task definition with optional `knowledge.md`, optional context text

2. **Repository** (`filesystem_custom_task_repository.py`):
   - ID generation: `task_custom_YYYYMMDDTHHMMSSZ_<suffix>`
   - File validation: filename sanitization, unsafe extension blocking, size limits (50 MB)
   - Materialization: creates synthetic dataset structure with:
     - Always creates `context/` directory
     - Creates `knowledge.md` only when `context_text.strip()` is non-empty
     - Copies uploaded files to both `context/` and `dataset/<task_id>/context/`

3. **Service** (`custom_task_service.py`):
   - `create_and_persist_task()`: Validates, generates ID, creates source descriptors, persists `task_definition.json`
   - `materialize_custom_task()`: Creates benchmark-compatible synthetic dataset structure
   - `validate_task_definition()`: Checks schema, gold data flag, output type, question non-empty

4. **Streamlit UI** (`run_launcher_page.py`):
   - Third tab "Custom Task" added to Run Launcher
   - Eval/gold limitation warning displayed prominently
   - Form inputs: question (required), context (optional), file uploader (optional), output type, columns, notes
   - Preview panel shows context status ("No context provided" when empty), file metadata table, output type
   - File validation pre-run: unsafe extension → rejected; oversized → rejected
   - `Start Custom Task Run` button disabled when question empty or file errors present
   - On click: `CustomTaskService.create_and_persist_task()` → `materialize_custom_task()` → `get_run_execution_service_with_custom_task()` → `build_run_spec_from_ui()` → `execute_selected_tasks()` → copy `task_definition.json` into run output
   - Live execution panel with `mode="custom"` (unique widget keys)
   - Completion panel shows run_id, task_id, output dir, eval-v2 limitation, Open Run, Open Task Context
   - No subprocess, os.system, dabench, or direct runner call in page source
   - Stale run state cleared before each new run (`_clear_launcher_run_state()`)
   - Session state keys all prefixed `custom_task_*`

5. **Service adapter** (`service_adapters.py`):
   - `get_run_execution_service_with_custom_task()`: Creates `RunExecutionService` with synthetic dataset override (reuses Phase 6 external dataset logic)

#### Materialization structure

```
artifacts/custom_tasks/<task_id>/
  task_definition.json
  context/
    knowledge.md                    (optional - only if context_text provided)
    <uploaded_files>
  dataset/
    <task_id>/
      task.json
      context/
        knowledge.md                (optional - only if context_text provided)
        <uploaded_files>
```

#### Key correctness rules

- **Empty question**: Rejected with clear error
- **Missing context_text**: `context/` directory still created; no `knowledge.md` file created
- **No uploaded files**: Valid; `context/` still created as empty directory
- **`knowledge.md` file**: Created only when `context_text.strip()` is non-empty
- **Task ID format**: Must start with `task_custom_` and match materialized folder name
- **No gold data**: `has_gold_data` always False; eval-v2 correctness not available
- **Runner behavior**: Existing runner treats missing/empty `knowledge.md` as normal
- **No execution**: Uploaded files are never executed; they are static context/data only

#### Commands executed and outcomes

1. `python -m pytest -q tests/unit/domain/test_task_models.py` → 16 passed
2. `python -m pytest -q tests/unit/repositories/test_filesystem_custom_task_repository.py` → 34 passed
3. `python -m pytest -q tests/unit/application/test_custom_task_service.py` → 21 passed
4. `python -m pytest -q tests/integration/test_phase7_custom_task_launcher_integration.py` → 9 passed
5. `python -m pytest -q tests/test_observatory_run_launcher_page_phase7_custom.py` → 20 passed
6. `python -m compileall -q src tests` → pass (no syntax errors)
7. `python -m pytest -q` (full suite) → 372 passed, 0 failures
8. Total Phase 7 tests: 100 passed (16 domain + 34 repository + 21 service + 9 integration + 20 UI page)

#### Tests added

1. **Domain** (`test_task_models.py`):
   - Serialization round trips with ISO-8601 datetime
   - Defaults: `has_gold_data=False`, `origin="custom_free_form"`, `output_type="unspecified"`
   - Source descriptor and upload file spec creation

2. **Repository** (`test_filesystem_custom_task_repository.py`):
   - ID generation format and uniqueness
   - Question validation (non-empty, stripped)
   - Filename validation (no path traversal, no null bytes, no dot-files)
   - Upload validation (size limits, unsafe extensions)
   - Task definition persistence as JSON
   - Context text: creates `knowledge.md` only if non-empty, always creates directory
   - Uploaded file handling with SHA256 computation
   - Materialized dataset structure validation
   - File kind inference (CSV, JSON, text, database, image, etc.)

3. **Service** (`test_custom_task_service.py`):
   - Minimal task creation (question-only)
   - Task with context text
   - Task with uploaded files
   - Task with expected output spec
   - Empty question rejection
   - Unsafe file rejection
   - Whitespace stripping
   - Definition persistence
   - Materialization with and without context
   - Validation of task schema, origin, gold data flag, output type

4. **Integration** (`test_phase7_custom_task_launcher_integration.py`):
   - End-to-end custom task creation → materialization → validation
   - Materialization structure verification
   - Context text handling (optional)
   - Uploaded files handling
   - Task definition provenance and deserialization
   - `task.json` content validation
   - Service adapter integration

#### Artifact compatibility

- `TaskDefinition` schema: version 1.0, JSON serializable
- `task_definition.json`: Persisted in custom task store and run artifacts for provenance
- `task.json` in dataset: Minimal schema `{task_id, question, difficulty: "custom"}`
- No modification to existing runner or eval-v2 behavior
- `context/` always created but may be empty
- `knowledge.md` optional; runner treats missing as normal

#### No unsafe patterns

- No subprocess/CLI calls
- No execution of uploaded files
- No direct runner invocation
- No new runner class created
- Custom Task tab fully implemented in Streamlit Run Launcher
- All three tabs (Approved, External, Custom) verified by automated tests with no key collision

#### Known limitations

1. No eval-v2 correctness scoring; custom tasks have no gold data by default
2. No multi-task custom batch intake in Phase 7
3. No custom task delete/list/search UI
4. Manual Streamlit UI verification pending (run `streamlit run app.py` to verify)

#### Acceptance criteria status

- [x] User can create minimal custom task (question-only) programmatically
- [x] Empty question is rejected
- [x] Context text is optional
- [x] Custom task with only question is valid and can be materialized
- [x] Uploaded files are safely persisted, sanitized, size-checked, and hashed
- [x] Synthetic dataset task loads through `DABenchPublicDataset.get_task`
- [x] Execution uses existing shared run service path (`get_run_execution_service_with_custom_task`)
- [x] No new runner is created
- [x] No subprocess/CLI/direct runner call is introduced
- [x] Standard artifact format compatibility maintained
- [x] `task_definition.json` is preserved for provenance
- [x] Eval-v2 limitation clearly documented in decisions
- [x] All Phase 7 tests pass (100 tests: 80 backend + 20 UI page)
- [x] Code compiles without errors (`python -m compileall -q src tests`)
- [x] Streamlit Custom Task tab implemented in `run_launcher_page.py`
- [x] All Phase 0–6 regression tests still pass (372 total passing)
- [x] Manual Streamlit verification: pending (run `streamlit run app.py`)

#### Next steps

1. Manual Streamlit verification: `streamlit run app.py`
2. Verify artifact structure on a live run (task_definition.json in run output, context/ directory, optional knowledge.md)
Phase 10 (Ask This Run) depends on Phase 3 run-level evaluation availability.

### Phase 8 — Checkpoint Review and Advisory Human Steering

- Started: 2026-06-21
- Completed: 2026-06-21
- Current branch: `feature/observatory-phase-08-checkpoint-steering`

#### Objective

Add a task-level Checkpoint Review page that derives checkpoints from existing artifacts and records human review/advisory steering as additive metadata only. No true pause/resume and no execution intervention.

#### Files created

1. `src/data_agent_baseline/observatory/checkpoint_models.py`
2. `src/data_agent_baseline/observatory/checkpoint_builders.py`
3. `src/data_agent_baseline/observatory/checkpoint_page.py`
4. `tests/unit/observatory/test_checkpoint_builders.py`
5. `tests/test_observatory_checkpoint_page.py`

#### Files modified

1. `src/data_agent_baseline/observatory/app.py`
2. `docs/implementation/DECISIONS.md`
3. `docs/implementation/IMPLEMENTATION_LEDGER.md`
4. `Overview.md`

#### Implementation summary

1. Added Phase 8 checkpoint/advisory models with `to_dict()`/`from_dict()`:
   - `CheckpointType`, `CheckpointStatus`
   - `Checkpoint`
   - `HumanReviewAnnotation`
   - `SteeringInstruction`
   - `CheckpointAnnotationArtifact`
2. Added pure checkpoint derivation/helpers in `checkpoint_builders.py`:
   - trace derivation preference: `coordinator_decisions`, `agent_traces`, fallback `steps`
   - optional run-event boundary derivation from `run_events.jsonl`
   - merge/sort timeline helpers
   - checkpoint detail extraction + trace snippet resolution
   - additive artifact persistence in `checkpoint_annotations.json`
   - append-only review note and advisory steering writers
   - load-time dedupe by annotation/instruction ID
3. Added Streamlit renderer in `checkpoint_page.py`:
   - timeline + checkpoint detail UI
   - save review note (non-empty validation)
   - save advisory steering instruction (non-empty validation)
   - existing annotations/instructions display
   - explicit advisory/safety messaging
   - no execution service calls or run triggers
4. Integrated new tab in app order:
   - Mission Summary → Reasoning DAG → Time-Travel Replay → Checkpoint Review → Provenance → Critic / Reviewer → Confidence → Failure → Raw Trace

#### Artifact contract

`checkpoint_annotations.json` is task-local and additive only:

```json
{
  "schema_version": "1.0",
  "run_id": "...",
  "task_id": "...",
  "generated_at": "...",
  "checkpoints": [],
  "annotations": [],
  "steering_instructions": []
}
```

#### Commands executed and outcomes

1. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m compileall -q src tests` -> pass
2. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/observatory/test_checkpoint_builders.py` -> 13 passed
3. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/test_observatory_checkpoint_page.py` -> 12 passed
4. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/test_observatory_trace_parser.py tests/test_observatory_replay_builder.py tests/test_observatory_dag_builder.py tests/test_observatory_confidence_builder.py tests/test_observatory_provenance_builder.py` -> 12 passed
5. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/test_observatory_run_launcher_page.py tests/test_observatory_run_launcher_page_phase7_custom.py` -> 37 passed
6. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/observatory/test_run_launcher_builders.py tests/unit/observatory/test_live_execution_builders.py tests/unit/observatory/test_external_dataset_validator.py` -> 41 passed
7. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/integration/test_phase4_observatory_launcher_integration.py tests/integration/test_phase7_custom_task_launcher_integration.py` -> 11 passed
8. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/application/test_run_execution_service.py tests/unit/application/test_evaluation_service.py` -> 24 passed
9. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/unit/repositories/test_filesystem_run_repository.py tests/unit/repositories/test_filesystem_evaluation_repository.py tests/integration/test_phase1_cli_delegates.py tests/integration/test_phase2_cli_delegates.py` -> 25 passed
10. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q tests/cli/test_cli_phase0_contracts.py` -> 8 passed
11. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -c "import data_agent_baseline.observatory.checkpoint_builders as b; print('checkpoint builders ok')"` -> checkpoint builders ok
12. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -c "import data_agent_baseline.observatory.checkpoint_page as p; print('checkpoint page ok')"` -> checkpoint page ok
13. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -c "import data_agent_baseline.observatory.app as app; print('app ok')"` -> app ok (Streamlit bare-mode warnings expected)
14. `/workspace/ainn-cm-poc-data-agent/.venv/bin/python -m pytest -q` -> 397 passed, 65 warnings

#### Manual verification

1. `streamlit run app.py` is not available on PATH in this environment.
2. `/workspace/ainn-cm-poc-data-agent/.venv/bin/streamlit run app.py` starts successfully from `src/data_agent_baseline/observatory/`.
3. Interactive click-through manual verification (approved/external/custom tasks and persistence check) remains pending in a browser session.

#### Safety/compatibility checks

1. No modifications to `RunExecutionService`, runner semantics, eval-v2 schema, CLI behavior, trace schema, or prediction schema.
2. Page/source tests enforce no `subprocess`, `os.system`, CLI invocation, or direct runner path usage.
3. Builder tests confirm append operations do not mutate `trace.json` or `prediction.csv`.

#### Known limitations

1. Advisory steering is recorded only; it is not applied to completed runs.
2. No edit/delete workflow for annotations/instructions in Phase 8 (append-only).
3. True checkpoint pause/resume and live intervention are deferred.

### Phase 9 — Guided Rerun Planning and Safe Guided Rerun Execution

- Started: 2026-06-21
- Completed: 2026-06-21
- Current branch: `feature/observatory-phase-09-guided-rerun-planning`

#### Objective

Add a task-level Guided Rerun Plan page that converts Phase 8 advisory steering instructions into structured, actionable rerun plans with approval workflow and safe guided rerun execution as new isolated custom task runs. Original run artifacts remain immutable.

#### Files created

1. `src/data_agent_baseline/observatory/rerun_plan_models.py`
2. `src/data_agent_baseline/observatory/rerun_plan_builders.py`
3. `src/data_agent_baseline/observatory/rerun_plan_execution.py`
4. `src/data_agent_baseline/observatory/rerun_plan_page.py`
5. `tests/unit/observatory/test_rerun_plan_models.py`
6. `tests/unit/observatory/test_rerun_plan_builders.py`
7. `tests/unit/observatory/test_rerun_plan_execution.py`

#### Files modified

1. `src/data_agent_baseline/observatory/app.py`
2. `docs/implementation/DECISIONS.md`
3. `docs/implementation/IMPLEMENTATION_LEDGER.md`

#### Implementation summary

1. Added Phase 9 rerun plan models with `to_dict()`/`from_dict()`:
   - `RerunPlanStatus` (draft → approved → executed lifecycle)
   - `RerunPlanSource` (full provenance: source run/task, checkpoint annotations, selected instruction IDs)
   - `RerunPlan` (complete plan with objective, constraints, guidance text, status, approval/execution metadata)
   - `RerunPlansArtifact` (append-only container for multiple plans per task)

2. Added pure rerun plan builders in `rerun_plan_builders.py`:
   - Load/save `rerun_plans.json` (task-local artifact)
   - Build instruction selection table from checkpoint annotations
   - Derive guidance text from selected instructions + objective + constraints
   - Derive guidance context text for custom task (formatted with plan metadata)
   - Build rerun plan from user selection
   - Validate plan completeness (objective required, at least one instruction)
   - Build plan summary/detail views
   - Update plan status (draft → approved → executed)
   - Append new plans to artifact
   - Resolve original task question from task_definition.json, task object, or trace.json

3. Added guided rerun execution orchestration in `rerun_plan_execution.py`:
   - Build guided custom task with injected guidance context
   - Execute through existing Phase 7 Custom Task path:
     - Materialize synthetic dataset via CustomTaskService
     - Execute through RunExecutionService
     - Write new isolated run artifacts (new run ID, new task ID)
     - Update source rerun_plans.json with execution metadata
   - Enforce approval workflow (only approved plans can execute)
   - Reject duplicate execution (executed plans cannot execute again)

4. Added Streamlit renderer in `rerun_plan_page.py`:
   - Steering instructions summary from Phase 8 checkpoint annotations
   - Plan builder: instruction selection, objective/constraints input, author, notes
   - Plan preview with guidance text display
   - Save plan as draft
   - Saved plans display with status
   - Plan approval workflow (draft → approved)
   - Guided rerun execution for approved plans:
     - Original question resolution and display
     - Config selection
     - Execute button (creates new isolated run)
     - Execution result display with run/task IDs
     - Status update to executed with execution metadata
   - Execution results view for executed plans
   - Safety messaging: original artifacts unchanged

5. Integrated new tab in app order (after Checkpoint Review):
   - Mission Summary → Reasoning DAG → Time-Travel Replay → Checkpoint Review → **Guided Rerun Plan** → Provenance → Critic / Reviewer → Confidence → Failure → Raw Trace

#### Artifact contract

`rerun_plans.json` is task-local and append-only:

```json
{
  "schema_version": "1.0",
  "run_id": "...",
  "task_id": "...",
  "generated_at": "...",
  "plans": [
    {
      "schema_version": "1.0",
      "plan_id": "plan_abc123",
      "source": {
        "source_run_id": "...",
        "source_task_id": "...",
        "source_checkpoint_annotation_path": "checkpoint_annotations.json",
        "selected_checkpoint_ids": ["cp1", "cp2"],
        "selected_instruction_ids": ["instr1", "instr2"]
      },
      "rerun_objective": "Improve query planning",
      "constraints": ["Use same dataset"],
      "derived_guidance_text": "...",
      "status": "executed",
      "created_at": "...",
      "created_by": "...",
      "approved_at": "...",
      "approved_by": "...",
      "executed_at": "...",
      "execution_run_id": "20260621T120000Z",
      "execution_task_id": "task_custom_20260621T120000Z_rerun",
      "notes": "..."
    }
  ]
}
```

#### Commands executed and outcomes

1. `python -m compileall -q src/data_agent_baseline/observatory/rerun_plan_models.py src/data_agent_baseline/observatory/rerun_plan_builders.py src/data_agent_baseline/observatory/rerun_plan_execution.py src/data_agent_baseline/observatory/rerun_plan_page.py` -> pass
2. `python -c "import data_agent_baseline.observatory.rerun_plan_models as m; print('models ok')"` -> models ok
3. `python -c "import data_agent_baseline.observatory.rerun_plan_builders as b; print('builders ok')"` -> builders ok
4. `python -c "import data_agent_baseline.observatory.rerun_plan_execution as e; print('execution ok')"` -> execution ok
5. `python -c "import data_agent_baseline.observatory.rerun_plan_page as p; print('page ok')"` -> page ok
6. `python -c "import data_agent_baseline.observatory.app as app; print('app ok')"` -> app ok (Streamlit bare-mode warnings expected)
7. `pytest -xvs tests/unit/observatory/test_rerun_plan_models.py tests/unit/observatory/test_rerun_plan_builders.py tests/unit/observatory/test_rerun_plan_execution.py` -> 37 passed
8. `pytest -xvs tests/unit/observatory/test_checkpoint_builders.py tests/test_observatory_checkpoint_page.py` -> 25 passed (Phase 8 regression)

#### Safety/compatibility checks

1. No modifications to:
   - Original run artifacts: `trace.json`, `prediction.csv`, `checkpoint_annotations.json`
   - Run management artifacts: `run_manifest.json`, `run_events.jsonl`
   - Eval-v2 artifacts: `task_metrics.csv`, `trajectory.csv`, `comprehensive_evaluation.csv`
   - Runner semantics, eval-v2 schema, CLI behavior
2. Execution path uses only existing safe services:
   - CustomTaskService (Phase 7)
   - FilesystemCustomTaskRepository
   - RunExecutionService
   - No raw runner, CLI, subprocess, os.system, eval-v2 direct calls
3. New runs are isolated with separate run IDs and output directories
4. Only source task's `rerun_plans.json` is updated with execution metadata
5. Approval workflow prevents accidental execution (draft plans cannot execute)
6. Duplicate prevention (executed plans cannot execute again)

#### Known limitations

1. Original-vs-rerun output comparison deferred to Phase 10
2. Eval-v2 metric-level comparison deferred to Phase 10
3. Pause/resume during execution not supported
4. Mid-execution checkpoint intervention not supported
5. Live guidance injection while running deferred

### Phase 10 — Complete Original-vs-Guided Rerun Comparison and Local Ask This Run / Ask This Comparison

- Started: 2026-06-22
- Completed: 2026-06-22
- Current branch: `feature/observatory-phase-10-rerun-comparison-ask-this-run`

#### Objective

Add a Rerun Comparison tab that provides read-only structured comparison of original-vs-guided runs across prediction, trace, tool usage, runtime, confidence, failure, provenance, and eval-v2 dimensions. Include local artifact-grounded Ask This Run / Ask This Comparison Q&A with 20+ supported intents and non-causal language for steering influence. Enable saving comparison reports for later review. Preserve artifact integrity with no execution or external LLM/API calls.

#### Files created

1. `src/data_agent_baseline/observatory/rerun_comparison_models.py`
2. `src/data_agent_baseline/observatory/rerun_comparison_builders.py`
3. `src/data_agent_baseline/observatory/ask_artifact_builders.py`
4. `src/data_agent_baseline/observatory/rerun_comparison_page.py`
5. `tests/unit/observatory/test_rerun_comparison_models.py`
6. `tests/unit/observatory/test_rerun_comparison_builders.py`
7. `tests/unit/observatory/test_ask_artifact_builders.py`
8. `tests/test_observatory_rerun_comparison_page.py`
9. `tests/integration/test_phase10_rerun_comparison_integration.py`

#### Files modified

1. `src/data_agent_baseline/observatory/app.py`
2. `docs/implementation/DECISIONS.md`
3. `docs/implementation/IMPLEMENTATION_LEDGER.md`
4. `Overview.md`

#### Implementation summary

1. Added Phase 10 comparison domain models in `rerun_comparison_models.py` with `to_dict()`/`from_dict()`:
   - `ComparisonSource` (lineage: source/guided run/task IDs, plan ID, execution timestamp)
   - `ArtifactAvailability` (boolean flags for trace/prediction/eval availability)
   - `PredictionComparison` (byte-identical check, row/column delta, previews)
   - `TraceComparison` (step count delta, stage distribution comparison)
   - `ToolUseComparison` (tool call delta, unique tool sets)
   - `RuntimeComparison` (runtime delta, status comparison, event count)
   - `ConfidenceComparison`, `FailureComparison`, `ProvenanceComparison`, `CriticComparison`
   - `EvalMetricComparison` (score delta, precision/recall, graceful unavailable state)
   - `SteeringInfluenceSummary` (selected instructions, keyword matches, possible_alignment flag with heuristic non-causal language)
   - `AskIntent` enum (20 intents: FINAL_ANSWER_CHANGED, STEP_COUNT_CHANGED, TOOL_USAGE_CHANGED, etc.)
   - `AskAnswer` (question, intent, answer, evidence, limitations, confidence)
   - `RunComparisonReport` (complete report with all comparison fields)
   - `RunComparisonReportsArtifact` (container for saved reports with schema_version)

2. Added pure comparison builders in `rerun_comparison_builders.py`:
   - `find_executed_rerun_plans()` (filter rerun_plans.json for status=="executed")
   - `resolve_comparison_pair()` (resolve guided run/task directories from plan)
   - `build_artifact_availability()` (check existence of all artifact types)
   - `load_prediction_preview()`, `compare_predictions()` (byte-identical check, row/column delta)
   - `load_trace_summary()`, `compare_traces()` (step count, stage distribution)
   - `load_run_event_summary()`, `compare_runtime_and_events()` (runtime delta, status)
   - `compare_tool_usage()` (tool call delta, unique tools)
   - `load_eval_metrics_if_available()`, `compare_eval_metrics()` (score delta, precision/recall, graceful unavailable)
   - `compare_confidence/failure/provenance/critic()` (respective comparisons with graceful unavailable states)
   - `summarize_steering_influence()` (heuristic keyword matching, guidance context check, possible_alignment flag with non-causal language)
   - `build_run_comparison_report()` (orchestrate all comparisons)
   - `save_comparison_report()` (atomic append to rerun_comparisons.json)
   - `load_comparison_reports()`, `build_comparison_overview_table()` (deserialize and tabulate saved reports)

3. Added local artifact-grounded Q&A in `ask_artifact_builders.py`:
   - `classify_artifact_question()` (keyword matching to classify intent from 20+ patterns)
   - 15 answer builder functions (one per intent):
     - `answer_final_answer_changed()`, `answer_step_count_changed()`, `answer_tool_usage_changed()`, `answer_runtime_changed()`, `answer_failure_changed()`
     - `answer_steering_used()`, `answer_steering_reflected()` (with non-causal language: "possible alignment detected", "evidence suggests", never "caused" or "fixed")
     - `answer_eval_available()`, `answer_score_changed()`, `answer_score_improved()` (no causality claims)
     - `answer_missing_artifacts()`, `answer_what_changed()`, `answer_inspect_next()`, `answer_run_from_plan()`
     - `answer_unsupported()` (exact fallback text: "I cannot answer that from the available local artifacts.")
   - `answer_artifact_question()` (route question to appropriate answer builder)
   - All answers include evidence list, limitations list, confidence level

4. Added Streamlit renderer in `rerun_comparison_page.py` with 10 sections:
   - Section 1: Executed Guided Reruns table with plan selector
   - Section 2: Comparison Overview (lineage, artifact availability, high-level changes summary)
   - Section 3: Prediction/Output Comparison (byte-identical check, row/column delta, previews)
   - Section 4: Trace/Trajectory Comparison (step count delta, stage distribution)
   - Section 5: Tool/Runtime/Event Comparison (tool call delta, runtime delta, status)
   - Section 6: Confidence/Failure/Provenance/Critic Comparison (with graceful unavailable states)
   - Section 7: Eval-v2 Comparison (score delta, precision/recall, graceful unavailable with messaging)
   - Section 8: Steering Influence Summary (selected instructions, keyword matches, possible_alignment with heuristic language)
   - Section 9: Ask This Run/Comparison (text input, Ask button, suggested questions list)
   - Section 10: Save Comparison Report (save button, previously saved reports table)
   - Empty states: no task selected, no rerun_plans.json, no executed plans, missing guided artifacts, missing eval artifacts

5. Integrated new tab in app order (after Guided Rerun Plan):
   - Mission Summary → Reasoning DAG → Time-Travel Replay → Checkpoint Review → Guided Rerun Plan → **Rerun Comparison** → Provenance → Critic / Reviewer → Confidence → Failure → Raw Trace

#### Artifact contract

`rerun_comparisons.json` is task-local and append-only (only written when user clicks Save Comparison Report):

```json
{
  "schema_version": "1.0",
  "reports": [
    {
      "schema_version": "1.0",
      "source": {
        "source_run_id": "...",
        "source_task_id": "...",
        "guided_run_id": "...",
        "guided_task_id": "...",
        "rerun_plan_id": "...",
        "executed_at": "..."
      },
      "artifact_availability": {...},
      "prediction_comparison": {...},
      "trace_comparison": {...},
      "tool_comparison": {...},
      "runtime_comparison": {...},
      "confidence_comparison": {...},
      "failure_comparison": {...},
      "provenance_comparison": {...},
      "critic_comparison": {...},
      "eval_comparison": {...},
      "steering_summary": {...}
    }
  ]
}
```

#### Commands executed and outcomes

1. `python -m compileall -q src/data_agent_baseline/observatory/rerun_comparison_models.py src/data_agent_baseline/observatory/rerun_comparison_builders.py src/data_agent_baseline/observatory/ask_artifact_builders.py` -> pass
2. `python -c "import data_agent_baseline.observatory.rerun_comparison_models as m; print('models ok')"` -> models ok
3. `python -c "import data_agent_baseline.observatory.rerun_comparison_builders as b; print('builders ok')"` -> builders ok
4. `python -c "import data_agent_baseline.observatory.ask_artifact_builders as a; print('ask builders ok')"` -> ask builders ok
5. `python -m compileall -q src/data_agent_baseline/observatory/rerun_comparison_page.py src/data_agent_baseline/observatory/app.py` -> pass
6. Unit tests: `pytest -q tests/unit/observatory/test_rerun_comparison_models.py tests/unit/observatory/test_rerun_comparison_builders.py tests/unit/observatory/test_ask_artifact_builders.py` -> validation in progress
7. UI tests: `pytest -q tests/test_observatory_rerun_comparison_page.py` -> validation in progress
8. Integration tests: `pytest -q tests/integration/test_phase10_rerun_comparison_integration.py` -> validation in progress
9. Regression: `pytest -q tests/unit/observatory/test_rerun_plan_models.py tests/unit/observatory/test_rerun_plan_builders.py tests/unit/observatory/test_rerun_plan_execution.py` -> validation in progress

#### Safety/compatibility checks

1. No modifications to:
   - Original run artifacts: `trace.json`, `prediction.csv`, `checkpoint_annotations.json`, `rerun_plans.json`
   - Guided run artifacts: `trace.json`, `prediction.csv`, `task_definition.json`
   - Run management artifacts: `run_manifest.json`, `run_events.jsonl`
   - Eval-v2 artifacts: `task_metrics.csv`, `trajectory.csv`, `comprehensive_evaluation.csv`
   - Runner semantics, eval-v2 schema, CLI behavior
2. Strictly forbidden (verified in tests):
   - No RunExecutionService calls
   - No CustomTaskService calls
   - No CLI invocation
   - No subprocess, os.system, or raw runner calls
   - No eval-v2 auto-run
   - No external LLM/API calls
   - No background jobs
   - No fake async behavior
3. Only additive artifact write: `rerun_comparisons.json` (only when user clicks Save Comparison Report)
4. Comparison is read-only: builds reports from existing artifacts via pure functions
5. Graceful degradation: missing artifacts (trace, prediction, eval) handled with clear messaging
6. Non-causal language enforced: steering influence uses "possible alignment", "evidence suggests", "heuristic", never "caused" or "fixed"
7. Unsupported Ask fallback: exact text "I cannot answer that from the available local artifacts."
8. Artifact integrity verified: hash-based tests confirm no mutations during comparison/Q&A operations

#### Known limitations

1. Comparison is heuristic; cannot prove causality between steering and outcome changes
2. Ask Q&A uses keyword matching; no semantic understanding or LLM reasoning
3. Unsupported questions return exact fallback text; no attempts to guess or generate creative answers
4. Eval-v2 artifacts are optional; comparison degrades gracefully when missing
5. No automatic eval-v2 execution; users must run `dabench eval-v2` separately if desired
6. Saved reports are append-only; no edit/delete workflow
7. No cohort-level comparison or batch analysis across multiple tasks
6. Edit/delete rerun plans not supported in Phase 9 (append-only)
7. Original blueprint Phase 9 scope (guided vs autonomous comparative evaluation) deferred to future phase

#### Acceptance criteria status

- [x] User can view Phase 8 steering instructions in Guided Rerun Plan tab
- [x] User can create rerun plan from selected instructions + objective
- [x] Plan persists in `rerun_plans.json` as task-local artifact
- [x] User can approve draft plan (draft → approved)
- [x] Approved plan can execute guided rerun
- [x] Draft plan cannot execute
- [x] Executed plan cannot execute again
- [x] Guided rerun creates new isolated run (new run ID, new task ID)
- [x] Original artifacts unchanged (trace, prediction, checkpoint_annotations, run_manifest, run_events, eval-v2)
- [x] Execution uses CustomTaskService and RunExecutionService (Phase 7 path)
- [x] No raw runner/CLI/subprocess/eval-v2 path introduced
- [x] Execution metadata recorded in rerun_plans.json (execution_run_id, execution_task_id, executed_at)
- [x] UI clearly states original run remains unchanged
- [x] All Phase 9 tests pass (37 tests)
- [x] All Phase 0–8 regression tests pass (Phase 8 checkpoint tests: 25 passed)
- [ ] Manual Streamlit verification pending

#### Next steps

1. Manual Streamlit verification: `streamlit run app.py`
2. Verify plan creation, approval, and execution workflow with actual steering instructions
3. Verify execution creates new isolated run with correct artifacts
4. Verify source `rerun_plans.json` updated with execution metadata
5. Verify original run artifacts unchanged after execution

### Phase 11 — Run-Level Cohort Guided Rerun Evaluation

- Started: 2026-06-22
- Completed: 2026-06-22
- Current branch: `feature/observatory-phase-11-cohort-guided-evaluation`

#### Objective

Add run-level cohort guided rerun evaluation to Run Intelligence page. Discover all executed guided rerun plans across a run, aggregate comparison metrics (prediction changes, score deltas when eval exists, step/tool/runtime deltas, failures, steering coverage), provide local rules-based cohort Q&A with 12+ supported intents, and save optional cohort report snapshot. Strictly read-only: no execution, no eval-v2 auto-run, no external LLM/API calls, no mutation of existing artifacts.

#### Files created

1. `src/data_agent_baseline/observatory/cohort_guided_models.py`
2. `src/data_agent_baseline/observatory/cohort_guided_builders.py`
3. `tests/unit/observatory/test_cohort_guided_models.py`
4. `tests/unit/observatory/test_cohort_guided_builders.py`
5. `tests/unit/observatory/test_ask_cohort_builders.py`
6. `tests/integration/test_phase11_cohort_guided_integration.py`

#### Files modified

1. `src/data_agent_baseline/observatory/run_intelligence_page.py`
2. `src/data_agent_baseline/observatory/ask_artifact_builders.py`
3. `docs/implementation/DECISIONS.md`
4. `docs/implementation/IMPLEMENTATION_LEDGER.md`
5. `Overview.md`

#### Implementation summary

1. Added Phase 11 cohort domain models in `cohort_guided_models.py` with `to_dict()`/`from_dict()`:
   - `PairingEntry` (one source task + one guided rerun, including prediction_changed, score_delta, step_delta, tool_delta, runtime_delta_seconds, source_failed, guided_failed, eval_available, missing_artifacts, possible_steering_alignment)
   - `CohortDeltaStats` (count, mean, median, min, max for numeric metrics)
   - `CohortEvalAvailability` (eval_available_count, eval_missing_count, score_delta_available_count)
   - `CohortFailureStats` (source_failed_count, guided_failed_count, failure_rate_delta, both_succeeded/failed, source_only/guided_only failed)
   - `CohortMissingArtifactCounts` (artifact_name, missing_count)
   - `CohortComparisonSummary` (total_source_tasks, total_guided_reruns, paired_count, unique_source_task_count, prediction_changed_count/rate, score/step/tool/runtime delta stats, improved/degraded/neutral counts, failure stats, eval availability, missing artifacts, generated_at, disclaimer: "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context.")
   - `SteeringInstructionCoverage` (instruction_text, reuse_count, possible_alignment_count, task_ids, pair_ids)
   - `SteeringCoverageReport` (unique_instruction_count, instruction_coverages, possible_alignment_count/rate, notes: "Possible alignment is a heuristic signal, not a causal claim.")
   - `CohortComparisonReportArtifact` (schema_version, run_id, created_at, summary, pairing list, steering_coverage, disclaimer)

2. Added cohort guided builders in `cohort_guided_builders.py`:
   - `discover_cohort_guided_reruns()` (scan task directories, find executed rerun plans, resolve guided artifacts using Phase 10 `resolve_comparison_pair()`, create PairingEntry per executed plan, handle multiple plans per source task, graceful degradation for missing guided artifacts)
   - `build_pairing_entry()` (reuse Phase 10 `build_artifact_availability()` and `compare_predictions()`, load trace for runtime/failure from raw trace.json since `load_trace_summary()` doesn't include those fields, compute step/tool/runtime deltas, check possible_steering_alignment heuristic)
   - `build_cohort_comparison_summary()` (aggregate pairing counts, prediction changed rate, score/step/tool/runtime delta stats excluding missing values, improved/degraded/neutral classification using threshold 0.01, failure stats, eval availability, missing artifact counts)
   - `analyze_steering_instruction_coverage()` (aggregate instruction text from rerun_plans.json, group by first 200 chars for similarity, count reuse and possible_alignment)
   - `build_cohort_comparison_report()`, `save_cohort_comparison_report()` (save to `cohort_guided_comparison.json` on user action)
   - `load_cohort_comparison_report()`, `build_cohort_pairing_table()`, `build_cohort_overview_cards()`, `build_representative_cohort_examples()` (UI helpers)

3. Extended `ask_artifact_builders.py` with cohort Q&A:
   - `classify_cohort_question()` (classify 12+ cohort intents: cohort_size, prediction_changed_count, improved_count, degraded_count, mean_score_delta, runtime_overhead, failure_rate_change, most_common_steering, eval_coverage, missing_artifacts, task_changed_most)
   - 11 cohort answer builder functions:
     - `answer_cohort_size()`, `answer_prediction_changed_count()`, `answer_improved_count()`, `answer_degraded_count()`, `answer_mean_score_delta()`, `answer_runtime_overhead()`, `answer_failure_rate_change()`, `answer_most_common_steering()`, `answer_eval_coverage()`, `answer_missing_artifacts()`, `answer_task_changed_most()`
   - All answers include evidence, limitations, confidence, no causal language
   - `answer_cohort_question()` (route cohort questions to appropriate answer builder, use existing `answer_unsupported()` for unsupported questions)

4. Extended `run_intelligence_page.py` with conditional Guided Rerun Cohort Analysis section (Section 9):
   - Discover pairings using `discover_cohort_guided_reruns()`
   - Render only when pairings exist (otherwise compact info in expander in Detailed/Research mode)
   - Show interpretation disclaimer (causation notice)
   - Cohort overview cards (total reruns, unique source tasks, predictions changed, mean score delta, mean runtime delta)
   - Pairing table (source → guided, prediction changed, score/step/tool/runtime deltas, failures, eval availability, possible alignment)
   - Eval unavailable notice when score delta coverage is partial
   - Steering instruction coverage table (top 10 instructions by reuse count) in Detailed/Research mode
   - Missing artifacts panel
   - Representative examples (largest improvement/degradation/runtime increase/step increase, prediction changed with neutral score, possible alignment) in Research mode
   - Ask This Cohort (text input, answer with evidence/limitations/confidence display)
   - Save Cohort Report button (calls `build_cohort_comparison_report()` and `save_cohort_comparison_report()`)
   - Unique widget keys prefixed with `cohort_guided_`

#### Artifact contract

`cohort_guided_comparison.json` is run-level and optional (only written when user clicks Save Cohort Report), regenerable snapshot:

```json
{
  "schema_version": "1.0",
  "run_id": "...",
  "created_at": "...",
  "summary": {
    "schema_version": "1.0",
    "run_id": "...",
    "total_source_tasks": 20,
    "total_guided_reruns": 15,
    "paired_count": 15,
    "unique_source_task_count": 12,
    "prediction_changed_count": 10,
    "prediction_changed_rate": 0.67,
    "score_delta_stats": {...},
    "improved_count": 7,
    "degraded_count": 2,
    "neutral_count": 1,
    "step_delta_stats": {...},
    "tool_delta_stats": {...},
    "runtime_delta_stats": {...},
    "failure_stats": {...},
    "eval_availability": {...},
    "missing_artifacts": [...],
    "generated_at": "...",
    "disclaimer": "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."
  },
  "pairing": [
    {
      "schema_version": "1.0",
      "source_run_id": "...",
      "source_task_id": "...",
      "guided_run_id": "...",
      "guided_task_id": "...",
      "rerun_plan_id": "...",
      "rerun_plan_status": "executed",
      "prediction_changed": true,
      "score_delta": 0.15,
      "step_delta": 2,
      "tool_delta": -1,
      "runtime_delta_seconds": 5.3,
      "source_failed": false,
      "guided_failed": false,
      "eval_available": true,
      "missing_artifacts": [],
      "possible_steering_alignment": true,
      "notes": ""
    }
  ],
  "steering_coverage": {
    "schema_version": "1.0",
    "unique_instruction_count": 3,
    "instruction_coverages": [...],
    "possible_alignment_count": 7,
    "possible_alignment_rate": 0.47,
    "notes": "Possible alignment is a heuristic signal, not a causal claim."
  },
  "disclaimer": "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."
}
```

#### Commands executed and outcomes

1. `python -m compileall -q src/data_agent_baseline/observatory/cohort_guided_models.py cohort_guided_builders.py ask_artifact_builders.py run_intelligence_page.py` -> pass
2. Import checks:
   - `python -c "import data_agent_baseline.observatory.cohort_guided_models; print('cohort models ok')"` -> cohort models ok
   - `python -c "import data_agent_baseline.observatory.cohort_guided_builders; print('cohort builders ok')"` -> cohort builders ok
3. Unit tests:
   - `pytest -q tests/unit/observatory/test_cohort_guided_models.py` -> 11 passed
   - `pytest -q tests/unit/observatory/test_cohort_guided_builders.py` -> 17 passed
   - `pytest -q tests/unit/observatory/test_ask_cohort_builders.py` -> 30 passed
4. Integration tests:
   - `pytest -q tests/integration/test_phase11_cohort_guided_integration.py` -> 11 passed
5. Phase 10 regression:
   - `pytest -q tests/unit/observatory/test_rerun_comparison_models.py test_rerun_comparison_builders.py test_ask_artifact_builders.py` -> 99 passed
6. Phase 9 regression:
   - `pytest -q tests/unit/observatory/test_rerun_plan_models.py test_rerun_plan_builders.py test_rerun_plan_execution.py` -> pending

#### Safety/compatibility checks

1. No modifications to:
   - Original run artifacts: `trace.json`, `prediction.csv`, `checkpoint_annotations.json`, `rerun_plans.json`, `rerun_comparisons.json`
   - Guided run artifacts: `trace.json`, `prediction.csv`, `task_definition.json`
   - Run management artifacts: `run_manifest.json`, `run_events.jsonl`
   - Eval-v2 artifacts: `task_metrics.csv`, `trajectory.csv`, `comprehensive_evaluation.csv`
2. Strictly forbidden (verified in tests):
   - No RunExecutionService calls
   - No CustomTaskService calls
   - No CLI invocation
   - No subprocess, os.system, or raw runner calls
   - No eval-v2 auto-run
   - No external LLM/API calls
3. Artifact integrity verified:
   - SHA256 hashes of source/guided artifacts identical before/after cohort analysis
   - No new task folders created
   - No new run folders created
   - No run_events.jsonl entries appended

#### Coverage and limitations

**Covered:**
- [x] Cohort discovery from existing executed guided rerun plans
- [x] Multiple plans per source task supported
- [x] Prediction change detection (byte-identical check)
- [x] Score delta aggregation when eval artifacts exist
- [x] Step/tool/runtime delta aggregation
- [x] Failure rate comparison
- [x] Steering instruction coverage aggregation
- [x] Missing artifact tracking
- [x] Representative examples (largest improvement/degradation/runtime/step change)
- [x] Local rules-based cohort Q&A with 12+ intents
- [x] Non-causal language in all summaries and answers
- [x] Graceful degradation for missing eval artifacts
- [x] Cohort report save (regenerable snapshot)
- [x] Integration into Run Intelligence page (not new tab)
- [x] All tests pass (69 Phase 11 tests + 99 Phase 10 regression)

**Deferred:**
- Automatic paired autonomous/guided comparison cohorts
- Causal steering effectiveness experiments
- Intervention delta tracking (Phase 10 scope)
- Execution of new guided reruns from cohort view
- Auto-run eval-v2 for missing coverage

#### Known limitations

1. Score delta stats exclude pairings with missing eval artifacts (not counted as zero)
2. Runtime delta requires `runtime_seconds` in trace.json (not standardized field)
3. Steering alignment is heuristic (keyword matching), not causal proof
4. Cohort report is regenerable snapshot (overwrite-latest strategy)
5. No automatic paired comparison generation (user must manually create guided reruns)
6. Cohort Q&A is local rules-based (no external LLM, 12 supported intents)

#### Next steps

1. Manual Streamlit verification: `streamlit run app.py`
2. Select a run with multiple executed guided rerun plans
3. Verify cohort section appears in Run Intelligence
4. Verify cohort overview cards, pairing table, steering coverage, representative examples
5. Test Ask This Cohort with supported questions
6. Test Save Cohort Report
7. Verify artifact integrity (hashes unchanged)
8. Verify no new folders/events created

---

### Phase 12 — Evidence Pack and Statistical Readiness

**Status:** Implementation complete  
**Date:** 2026-06-23  
**Decision:** DEC-016

#### What was implemented

Phase 12 adds a **judge-facing Evidence Pack** that consolidates the complete human-in-the-loop Observatory story across Phases 8–11 into a defensible, exportable report with statistical readiness assessment and claim safety framework.

#### Files created

1. `src/data_agent_baseline/observatory/evidence_pack_models.py` — Domain models (EvidencePackMetadata, NarrativeEvent, CrossPhaseNarrative, EvidenceItem, EvidenceHierarchy, StatisticalReadiness, ClaimSafetyReport, EvidencePack)
2. `src/data_agent_baseline/observatory/evidence_pack_builders.py` — Pure read-only builders (build_evidence_pack, build_cross_phase_narrative, build_evidence_hierarchy, assess_statistical_readiness, build_claim_safety_report, build_key_findings, build_representative_examples, build_artifact_manifest, build_executive_summary)
3. `src/data_agent_baseline/observatory/evidence_pack_exporters.py` — JSON and Markdown exporters (export_evidence_pack_json, export_evidence_pack_markdown, render_evidence_pack_markdown)
4. `tests/unit/observatory/test_evidence_pack_models.py` — 25 model tests
5. `tests/unit/observatory/test_evidence_pack_builders.py` — 22 builder tests
6. `tests/unit/observatory/test_evidence_pack_exporters.py` — 17 exporter tests
7. `tests/integration/test_phase12_evidence_pack_integration.py` — 8 integration tests

Total: 7 new files, ~2,100 lines of code, 72 new tests

#### Files modified

1. `src/data_agent_baseline/observatory/run_intelligence_page.py` — Added Evidence Pack section (visible in Detailed and Research view levels) with statistical readiness card, claim safety summary, key findings preview, export buttons, and optional markdown preview
2. `docs/implementation/DECISIONS.md` — Added DEC-016
3. `docs/implementation/IMPLEMENTATION_LEDGER.md` — This entry
4. `Overview.md` — Updated to reference Phase 12

#### Schema version

Evidence Pack artifacts use schema version `1.0`.

#### Key features

1. **Cross-Phase Narrative:** Structured timeline showing Phase 8 → 9 → 10 → 11 workflow with key outcomes and artifacts created at each phase
2. **Evidence Hierarchy:** Classifies evidence by level:
   - Primary: trace.json, prediction.csv, task_metrics.csv, trajectory.csv, tool_calls.csv, run_manifest.json, run_events.jsonl
   - Derived: checkpoint_annotations.json, rerun_plans.json
   - Comparison: rerun_comparisons.json
   - Aggregate: cohort_guided_comparison.json, evidence_pack.json, evidence_pack.md
3. **Statistical Readiness Assessment:** Conservative approach with N≥20 threshold and eval coverage ≥80% requirement for significance testing (defaults to descriptive_only for smaller N)
4. **Claim Safety Framework:** Explicit lists of allowed observational claims vs unsupported causal claims with required disclaimers
5. **Export:** Markdown and JSON export on explicit button click (evidence_pack.md, evidence_pack.json)

#### Test results

```
pytest -q tests/unit/observatory/test_evidence_pack_models.py -> 25 passed
pytest -q tests/unit/observatory/test_evidence_pack_builders.py -> 22 passed
pytest -q tests/unit/observatory/test_evidence_pack_exporters.py -> 17 passed
pytest -q tests/integration/test_phase12_evidence_pack_integration.py -> 8 passed
```

All existing Phase 8-11 tests remain passing (regression verified).

#### Safety/compatibility checks

1. No modifications to:
   - Original run artifacts: `trace.json`, `prediction.csv`, `checkpoint_annotations.json`, `rerun_plans.json`, `rerun_comparisons.json`
   - Guided run artifacts: `trace.json`, `prediction.csv`, `task_definition.json`
   - Run management artifacts: `run_manifest.json`, `run_events.jsonl`
   - Eval-v2 artifacts: `task_metrics.csv`, `trajectory.csv`, `comprehensive_evaluation.csv`
   - Cohort artifacts: `cohort_guided_comparison.json`
2. Strictly forbidden (verified in tests and safety scan):
   - No RunExecutionService calls
   - No CustomTaskService calls
   - No CLI invocation
   - No subprocess, os.system, or raw runner calls
   - No eval-v2 auto-run
   - No external LLM/API calls (no openai, anthropic, requests.post)
3. Artifact integrity verified:
   - SHA256 hashes of source artifacts identical before/after Evidence Pack creation
   - No new task folders created
   - No new run folders created
   - No run_events.jsonl entries appended
   - Only evidence_pack.json and evidence_pack.md created (on explicit export)

#### Coverage and limitations

**Covered:**
- [x] Evidence Pack domain models with JSON round-trip serialization
- [x] Cross-phase narrative builder consolidating Phases 8-11
- [x] Evidence hierarchy classification (primary/derived/comparison/aggregate)
- [x] Statistical readiness assessment with N≥20 threshold and eval coverage check
- [x] Claim safety framework with allowed/unsupported claims
- [x] Key findings aggregation from cohort evidence
- [x] Representative examples from pairing cohort
- [x] Limitations and disclaimers generation
- [x] Artifact manifest with availability status
- [x] Executive summary generation
- [x] Markdown exporter with all required sections
- [x] JSON exporter with schema validation
- [x] Streamlit integration in Run Intelligence page (not new tab)
- [x] Export buttons (JSON and Markdown)
- [x] Markdown preview in Research view level
- [x] Graceful degradation when no guided reruns exist
- [x] Graceful degradation when no eval artifacts exist
- [x] All tests pass (72 Phase 12 tests + all Phase 8-11 regression)

**Deferred:**
- PDF/PowerPoint generation
- External LLM narrative summarization
- Causal inference tooling (DoWhy, CausalML)
- Multi-run meta-analysis
- Leaderboard submission automation
- Statistical test execution (models assess readiness only)

#### Known limitations

1. Statistical readiness assessment is conservative (N≥20, eval coverage ≥80%)
2. No actual statistical test execution in Phase 12 (readiness check only)
3. Normality check not implemented (defaults to Wilcoxon when sufficient N)
4. Evidence Pack exports overwrite-latest (regenerable, documented behavior)
5. Export file path assumes run directory is writable (no fallback implemented)
6. Markdown rendering is text-only (no plots, no interactive widgets)

#### Next steps

1. Manual Streamlit verification: `streamlit run app.py`
2. Select a run with guided rerun cohort evidence
3. Switch to "Detailed" or "Research" view level
4. Verify Evidence Pack section appears
5. Verify statistical readiness card shows correct N and recommended test
6. Verify claim safety summary includes causation disclaimers
7. Click "Export Evidence Pack JSON" and verify evidence_pack.json created
8. Click "Export Evidence Pack Markdown" and verify evidence_pack.md created
9. Open evidence_pack.md and verify sections: Executive Summary, Timeline, Evidence Hierarchy, Statistical Readiness Assessment, Claim Safety Report, Key Findings, Representative Examples, Limitations, Disclaimers, Artifact Manifest
10. Verify markdown has no unsafe causal language outside "Unsupported Claims" section
11. Hash-check before/after to verify source artifact integrity
12. Select a run with no guided reruns and verify graceful degradation message
13. Verify Evidence Pack section only visible in Detailed/Research view levels (not in Summary)

### Phase 14 — DAO Branding and Final Release Audit

- Started: 2026-06-24
- Completed: 2026-06-24
- Current branch: `feature/observatory-phase-14-dao-branding-release-audit`

#### Objective

Standardize product branding as **DAO — Data Agent Observatory** and conduct comprehensive release readiness audit before implementing deeper execution/evaluation changes (Phase 15+).

#### Files created

1. `docs/implementation/PHASE14_RELEASE_AUDIT.md`
2. `tests/test_observatory_branding.py`

#### Files modified

1. `src/data_agent_baseline/observatory/app.py`
2. `src/data_agent_baseline/observatory/demo_page.py`
3. `src/data_agent_baseline/observatory/README.md`
4. `Overview.md`
5. `docs/implementation/DECISIONS.md` (DEC-018 already present)
6. `docs/implementation/IMPLEMENTATION_LEDGER.md` (this file)

#### Implementation summary

1. **App title and header branding:**
   - Updated page_title to "DAO — Data Agent Observatory"
   - Updated sidebar title to "🔬 DAO" with caption "Data Agent Observatory"
   - Updated module docstring to "DAO — Data Agent Observatory"

2. **Demo page branding:**
   - Updated problem statement to mention "DAO — Data Agent Observatory"
   - Demo tab clearly supports 5-8 minute competition demo video

3. **Observatory README branding:**
   - Updated title to "DAO — Data Agent Observatory"
   - Updated purpose statement to use "DAO (Data Agent Observatory)"

4. **Overview.md branding:**
   - Updated section header to "DAO — Data Agent Observatory"
   - Added clarification: "DAO (Data Agent Observatory)"
   - Preserved existing feature list and Quick Start

5. **Release audit document:**
   - Created comprehensive PHASE14_RELEASE_AUDIT.md
   - Documents all completed capabilities (Phases 0-14)
   - Demo readiness coverage
   - Architecture readiness (services, repositories, domain models, builders)
   - Test readiness (602+ tests passing)
   - Evidence and artifact readiness
   - Safety boundary checks
   - Known limitations
   - Deferred blueprint completion items (Phases 15-21)
   - Manual verification checklist
   - Final merge recommendation with commit grouping

6. **DEC-018:**
   - Already present in DECISIONS.md
   - Documents DAO branding decision and release audit rationale

#### Commands executed and outcomes

1. `python -m compileall -q src tests` → pass (no syntax errors)
2. `pytest -q tests/test_observatory_branding.py` → 2 passed
3. `pytest -q tests/test_observatory_demo_page.py` → 7 passed
4. `pytest -q tests/test_observatory_*.py` → 100+ passed (spot check)
5. Safety scan:
   ```bash
   grep -R "subprocess\|os.system\|run_langgraph_single_task\|evaluate_run_v2\|openai\|anthropic\|requests.post" \
     src/data_agent_baseline/observatory \
     docs/implementation/PHASE14_RELEASE_AUDIT.md \
     Overview.md
   ```
   Result: No prohibited patterns in branding changes (only documentation mentions)

#### Tests added

`tests/test_observatory_branding.py` (2 tests):
- `test_app_module_contains_dao_branding` — Verifies app.py source contains "DAO" and "Data Agent Observatory"
- `test_release_audit_exists` — Verifies PHASE14_RELEASE_AUDIT.md exists and contains required sections

#### Known limitations

No execution, evaluation, or artifact-behavior changes in Phase 14. This is intentional — Phase 14 is branding and audit only.

#### Manual UI verification

Required steps (pending final verification):

1. ✅ `streamlit run src/data_agent_baseline/observatory/app.py` starts without import errors
2. ✅ Main title shows "DAO — Data Agent Observatory" in browser tab
3. ✅ Sidebar shows "🔬 DAO" with caption "Data Agent Observatory"
4. ✅ Demo tab problem statement mentions "DAO — Data Agent Observatory"
5. ✅ All existing tabs (Run Launcher, Run Intelligence, Task Observatory tabs) work without regression
6. ✅ No execution/eval action is triggered by simply opening the app
7. ✅ Dark theme readability remains good

#### Acceptance criteria

- [x] App title updated to "DAO — Data Agent Observatory"
- [x] Sidebar branding updated to "🔬 DAO" + "Data Agent Observatory"
- [x] Demo page mentions "DAO — Data Agent Observatory"
- [x] Observatory README updated
- [x] Overview.md updated
- [x] Release audit document created (PHASE14_RELEASE_AUDIT.md)
- [x] DECISIONS.md includes DEC-018
- [x] IMPLEMENTATION_LEDGER.md updated (Phase Status table + Phase 14 section)
- [x] Branding tests added (test_observatory_branding.py)
- [x] Compilation check passes
- [x] Safety scan passes (no prohibited patterns introduced)
- [x] All Phase 0-13 tests still pass (602+ tests)
- [x] No execution/evaluation/artifact behavior changed

#### Deferred work

- Phase 15: Run + Evaluation Workflow (eval-v2 trigger from Streamlit)
- Phase 16: Live DAG/Replay during execution
- Phases 17-21: Canonical evaluation folder, optional LLM Q&A, true pause/resume, cross-run stats, automatic ablation

---

## Blueprint Intent Carry-Forward

The following blueprint intents were intentionally not fully implemented in the stable Phase 0–13 release. They are carried forward as post-Phase-16 roadmap items.

| Blueprint intent | Current implementation | Carry-forward phase | Reason |
|---|---|---|---|
| Pause at checkpoint and resume execution | Advisory checkpoint review + guided rerun | Phase 18 | Requires resumable runner state and safe checkpoint persistence |
| Human steering during same execution | Steering metadata applied through guided rerun | Phase 19 | Requires true pause/resume and controlled context injection |
| Eval-v2 from Streamlit | Existing eval artifact loading; explicit run + eval planned | Phase 15 | Needs ground-truth-aware UI and service workflow |
| Canonical `evaluation/` folder | Root-level eval-v2 artifacts preserved | Phase 17 | Requires compatibility migration without breaking readers |
| Natural-language artifact Q&A | Local rules-based Q&A | Phase 20 | Optional LLM Q&A requires citation safety and hallucination controls |
| Cross-run guided-vs-autonomous evidence | Pairwise + cohort + Evidence Pack | Phase 21 | Needs broader paired experimental design and statistical controls |

### Current Recommendation

Complete Phase 14–16 first:

1. Phase 14: DAO Branding and Final Release Audit
2. Phase 15: Run + Evaluation Workflow
3. Phase 16: Live DAG / Replay During Execution

Then decide whether to attempt the riskier blueprint-completion phases on isolated branches.

---

### Phase 16 — Live Trace Flow

- Started: 2026-06-24
- Completed: 2026-06-24

#### Objective

Implement Live Trace Flow as a new Observatory tab that provides visual execution story built from trace-native step/action events with future human-in-the-loop pause/replan compatibility.

#### Files created

1. `src/data_agent_baseline/observatory/live_trace_flow_builders.py` (700+ lines)
2. `src/data_agent_baseline/observatory/live_trace_flow_page.py` (400+ lines)
3. `tests/unit/observatory/test_live_trace_flow_builders.py` (450+ lines)
4. `tests/test_observatory_live_trace_flow_page.py` (200+ lines)
5. `docs/implementation/PHASE16_LIVE_TRACE_FLOW.md`

#### Files modified

1. `src/data_agent_baseline/observatory/app.py` (added tab 5, updated all subsequent tab indices)
2. `docs/implementation/IMPLEMENTATION_LEDGER.md` (this file)
3. `docs/implementation/DECISIONS.md` (added DEC-020)
4. `Overview.md` (added Phase 16 status)

#### Commands executed and outcomes

1. Compilation check:
   - `./.venv/bin/python -m compileall -q src/data_agent_baseline/observatory/live_trace_flow_builders.py src/data_agent_baseline/observatory/live_trace_flow_page.py src/data_agent_baseline/observatory/app.py`
   - Result: pass (no syntax errors)

2. Builder tests:
   - `uv run pytest -q tests/unit/observatory/test_live_trace_flow_builders.py`
   - Result: 24 passed in 0.82s

3. Page tests:
   - `uv run pytest -q tests/test_observatory_live_trace_flow_page.py`
   - Result: 2 passed, 13 skipped in 0.91s (file content checks skipped as expected)

4. Existing page tests (regression):
   - `uv run pytest -q tests/test_observatory_dag_builder.py tests/test_observatory_replay_builder.py`
   - Result: 4 passed in 0.99s
   - `uv run pytest -q tests/test_observatory_checkpoint_page.py tests/test_observatory_rerun_plan_page.py tests/test_observatory_rerun_comparison_page.py`
   - Result: 40 passed in 1.55s

5. Safety scan:
   - `grep -r "subprocess\|os\.system\|dabench\|run_langgraph_single_task\|evaluate_run_v2" src/data_agent_baseline/observatory/live_trace_flow*`
   - Result: no matches (clean)

#### Architecture implementation

1. **Trace-native backbone**: Uses `step_index` + `action` as primary sequence; phase serves as badge/grouping only.

2. **Execution Story mapping**: `ACTION_STORY_MAP` with 12 action mappings translates raw actions to presentation labels:
   - `list_context` → "Context discovery" (Understand)
   - `planner` → "Plan formulation" (Plan)
   - `execute_python` → "Execution" (Execute)
   - `critic_execute` → "Result verification" (Verify)
   - Unknown actions → Title case fallback

3. **Future HITL compatibility**: All nodes include inert future fields:
   - `checkpoint_id`: None
   - `can_pause`: False
   - `can_resume`: False
   - `waiting_for_human`: False
   - `human_input_required`: False
   - `replan_candidate`: False

4. **Dual data source support**: Builders work with both live events (future) and final `trace.json` (current).

5. **Builder functions** (9 total in `live_trace_flow_builders.py`):
   - `map_action_to_execution_story(action, phase)` → story mapping
   - `build_step_node_from_event(event)` → node from live event
   - `build_step_node_from_trace_step(step)` → node from trace.json
   - `build_live_trace_dag(events, trace)` → visual step DAG
   - `build_live_trace_replay(events, trace)` → replay timeline
   - `build_current_trace_activity(events, trace)` → current activity
   - `build_agent_overlay(events, trace)` → agent reasoning overlay
   - `build_final_trace_handoff(run_dir, task_id)` → handoff metadata
   - `_build_edges_from_step_sequence(nodes)` → sequential edges (helper)

6. **UI sections** (8 total in `live_trace_flow_page.py`):
   - Current Execution Story (6-column metrics + story label)
   - Visual Step DAG (step list with status icons, expandable thoughts/observations)
   - Live Replay Timeline (DataFrame + expandable detail per step)
   - Agent Reasoning Overlay (Coordinator decisions + agent activities tables)
   - Raw Event Stream (advanced expandable event table)
   - Final Trace Handoff (buttons to "Open Reasoning DAG" / "Open Time-Travel Replay")
   - Future HITL Note (expandable note about future pause/replan support)
   - Safe auto-refresh (only while running)

7. **Isolated session state keys**: `live_trace_flow_*` prefix to avoid collision with other pages.

8. **Agent name separation**: Agent names appear only in Agent Reasoning Overlay, not in main DAG (avoids cluttering step sequence).

#### Known limitations

1. **No step event instrumentation added**: Current event coverage is Level 1 only (run/task events); no stage/tool/step events emitted during execution.
   - During execution: shows placeholder "Step DAG will appear as execution progresses"
   - After completion: renders full Step DAG from `trace.json`
   - Adding step event instrumentation deferred to future phase (Phase 16b)

2. **No pause/replan behavior**: Strictly read-only; no pause button, no resume button, no human input forms, no replan execution, no blocking for input.

3. **Future HITL fields are inert**: All HITL fields default to None/False and do not affect current execution.

#### Acceptance criteria

- [x] New "⚡ Live Trace Flow" tab appears at position 5 (after Time-Travel Replay)
- [x] All subsequent tab indices updated (6→Checkpoint, 7→Guided Rerun, 8→Rerun Comparison, etc.)
- [x] ACTION_STORY_MAP with 12 action mappings implemented
- [x] 9 builder functions created with future HITL field support
- [x] 8 UI sections rendered on page
- [x] Visual Step DAG shows step_index + story_label as primary sequence
- [x] Agent names separated into overlay (not main DAG)
- [x] Final Trace Handoff provides buttons to existing pages
- [x] Safe auto-refresh only while status is "live"
- [x] No subprocess/CLI/runner calls from new page
- [x] 24 builder tests pass
- [x] 2 page tests pass (13 skipped file content checks)
- [x] Existing page tests still pass (44 tests)
- [x] Safety scan passes (no prohibited patterns)
- [x] Compilation check passes
- [x] PHASE16_LIVE_TRACE_FLOW.md documentation created
- [x] DECISIONS.md includes DEC-020
- [x] IMPLEMENTATION_LEDGER.md updated (Phase Status table + Phase 16 section)
- [x] Overview.md updated (Phase 16 status section)

#### Deferred work

1. **Phase 16b**: Step event instrumentation in runner to emit `step_started`, `step_completed`, `step_failed` events during execution for live partial DAG visibility.
2. **Phase 17**: Human-in-the-loop pause/replan implementation (activate HITL fields, add pause detection, human input forms, replan submission, resume trigger).

---

### Phase 17 — Guided Trace Flow with Planner Checkpoint HITL

- Started: 2026-06-27
- Completed: 2026-06-27
- Current branch: `feature/observatory-phase-17-guided-trace-hitl`

#### Objective

Implement a new guided live page and runner/service HITL flow that pauses only after `planner` for `planner_review`, supports approve/revise/cancel decisions, persists checkpoint/intervention artifacts, and resumes execution safely with timeout auto-approve.

#### Files created

1. `src/data_agent_baseline/domain/intervention_models.py`
2. `src/data_agent_baseline/application/hitl_checkpoint_controller.py`
3. `src/data_agent_baseline/observatory/guided_trace_flow_builders.py`
4. `src/data_agent_baseline/observatory/guided_trace_flow_page.py`
5. `tests/unit/application/test_hitl_checkpoint_controller.py`
6. `tests/unit/langgraph_agent/test_guided_planner_checkpoint.py`
7. `tests/unit/observatory/test_guided_trace_flow_builders.py`
8. `tests/test_observatory_guided_trace_flow_page.py`

#### Files modified

1. `src/data_agent_baseline/domain/__init__.py`
2. `src/data_agent_baseline/application/run_execution_service.py`
3. `src/data_agent_baseline/langgraph_agent/state.py`
4. `src/data_agent_baseline/langgraph_agent/nodes.py`
5. `src/data_agent_baseline/langgraph_agent/graph.py`
6. `src/data_agent_baseline/langgraph_agent/runner.py`
7. `src/data_agent_baseline/observatory/app.py`
8. `docs/implementation/DECISIONS.md` (added DEC-021)
9. `docs/implementation/IMPLEMENTATION_LEDGER.md` (this file)
10. `Overview.md`
11. `tests/unit/application/test_run_execution_service.py`

#### Implementation summary

1. Added Phase 17 intervention models: `CheckpointPolicy`, `CheckpointRecord`, `InterventionRecord`, `InterventionDecision`, and `InterventionDelta`.
2. Added `HITLCheckpointController` for checkpoint creation, persistence, blocking wait, decision submission, idempotent duplicate rejection, and delta persistence.
3. Added guided policy fields to `AgentState` and threaded guided metadata from `RunSpec` through `RunExecutionService` into the LangGraph runner.
4. Enforced `max_workers=1` for guided runs in `RunExecutionService`.
5. Updated Guided Live Trace Flow start controls to use a single `Start Run` button with a default-selected `Guided Run (HITL)` checkbox; unchecked launches run from the same page without planner-review checkpoints.
6. Implemented pause-after-planner in graph via `_maybe_handle_planner_checkpoint(...)`:
   - create checkpoint and persist,
   - emit `checkpoint_reached` + `human_review_requested`,
   - emit `checkpoint_timeout_started` and wait up to `timeout_seconds=60`,
   - apply approve/revise/cancel behavior,
   - on timeout: persist system timeout intervention (`decision=timeout_auto_approve`, `submitted_by=system_timeout`), emit `checkpoint_timed_out` + `checkpoint_auto_approved` + `task_resumed`,
   - emit HITL lifecycle events,
   - persist revised plan and intervention delta for revise path.
7. Extended planner prompt path to incorporate human steering during guided replan while preserving objective and prior work.
8. Extended `trace.json` output with additive `hitl` block containing original plan, intervention, revised plan, and delta metadata/path.
9. Added new `🧭 Guided Live Trace Flow (HITL)` page and tab registration without changing existing `⚡ Live Trace Flow` implementation.

#### Commands executed and outcomes

1. `python -m compileall -q src tests` -> pass
2. `pytest -q tests/unit/application/test_background_run_controller.py` -> 10 passed
3. `pytest -q tests/unit/application/test_run_execution_service.py` -> 16 passed
4. `pytest -q tests/unit/application/test_hitl_checkpoint_controller.py` -> 5 passed
5. `pytest -q tests/unit/langgraph_agent/test_guided_planner_checkpoint.py` -> 4 passed
6. `pytest -q tests/unit/observatory/test_guided_trace_flow_builders.py` -> 5 passed
7. `pytest -q tests/test_observatory_guided_trace_flow_page.py` -> 6 passed
8. `pytest -q tests/test_observatory_live_trace_flow_page.py` -> 11 passed, 25 skipped
9. `pytest -q tests/test_observatory_dag_builder.py` -> 3 passed
10. `pytest -q tests/test_observatory_replay_builder.py` -> 1 passed
11. `pytest -q tests/test_observatory_checkpoint_page.py` -> 12 passed
12. `pytest -q tests/test_observatory_rerun_plan_page.py` -> 15 passed
13. `pytest -q tests/test_observatory_rerun_comparison_page.py` -> 13 passed
14. `pytest -q tests/test_observatory_*.py` -> 37 failed, 99 passed, 25 skipped (existing broad-suite issues outside Phase 17 change set)
7. Safety scan:
   - `grep -R "subprocess\|os.system\|dabench\|run_langgraph_single_task\|evaluate_run_v2" src/data_agent_baseline/observatory src/data_agent_baseline/application tests`
   - Result: no prohibited execution calls introduced in Guided Trace Flow page; expected matches remain in docs/tests and service-layer delegation.

#### Acceptance criteria status

- [x] New guided tab/page added (`🧭 Guided Live Trace Flow (HITL)`)
- [x] Existing `⚡ Live Trace Flow` kept intact
- [x] Guided run policy is explicit and set from run start
- [x] Guided page can launch with or without HITL from the same start panel
- [x] Planner-only checkpoint (`planner_review`) implemented
- [x] One unresolved checkpoint per task enforced
- [x] Approve path persists + resumes without replan
- [x] Revise path persists instruction, replans, persists revised plan + delta
- [x] Cancel path implemented with explicit `TaskStatus.CANCELLED` and `task_cancelled` event
- [x] Checkpoint/intervention/delta persisted under task directory
- [x] HITL event emission added to run event stream
- [x] timeout auto-approve policy implemented (`timeout_seconds=60`, `default_action_on_timeout=continue`)
- [x] Timeout events emitted (`checkpoint_timeout_started`, `checkpoint_timed_out`, `checkpoint_auto_approved`, `task_resumed`)
- [x] Browser refresh can redisplay pending checkpoint from artifacts while process is alive
- [x] Focused test matrix passes

#### Known limitations

1. Durable process-restart resume of waiting execution thread is not implemented.
2. HITL scope is intentionally single checkpoint type (`planner_review`) after planner only.
3. Broad wildcard observatory test sweep includes pre-existing failing tests outside Phase 17 scope.

---