docs/implementation/DECISIONS.md

Architecture and Implementation Decisions

DEC-001

• Date: 2026-06-18
• Phase: Phase 0
• Decision: Freeze runtime behavior and capture executable contracts through documentation, reduced fixtures, and tests before refactoring.
• Reason: Later architecture extraction must preserve current CLI/eval/observatory behavior.
• Alternatives: Start service extraction immediately.
• Consequences: Phase 0 adds only docs/fixtures/tests and evidence; no src/ refactor.
• Blueprint impact: Aligns with Phase 0 baseline protection objective.

DEC-002

• Date: 2026-06-18
• Phase: Phase 0
• Decision: Keep existing .gitignore behavior unchanged during Phase 0; stage new docs/tests/fixtures with force-add in commit prep.
• Reason: Phase 0 scope is regression protection without repository policy refactoring.
• Alternatives: Change .gitignore to unignore docs/ and tests/ paths.
• Consequences: Commit instructions must explicitly include git add -f for Phase 0 artifacts.
• Blueprint impact: No runtime impact; preserves Phase 0 non-refactor constraint.

DEC-003

• Date: 2026-06-19
• Phase: Phase 1
• Decision: Update one Phase 0 CLI contract test (test_run_lang_task_delegates_once_per_task_preserves_order_shared_run_dir) to monkeypatch RunExecutionService.execute_selected_tasks instead of patching cli.py internals.
• Reason: Phase 1 intentionally removes the selected-task orchestration loop from CLI and moves it to a shared service boundary.
• Alternatives: Keep patching run_langgraph_single_task directly in CLI tests.
• Consequences: External command contract checks are preserved while aligning tests with the new adapter boundary.
• Blueprint impact: No behavior change; this is a test harness adaptation required by the approved Phase 1 architecture.

DEC-004

• Date: 2026-06-19
• Phase: Phase 2
• Decision: Allow EvaluationService.load_existing_evaluation() to use the existing internal _compute_summary_v2 helper from eval_v2.py when reconstructing an EvaluationBundle from already-generated run-root artifacts.
• Reason: Reusing the existing summary computation avoids duplicating metric logic inside the service and reduces the risk of drift from the current eval-v2 behavior.
• Alternatives: Duplicate summary reconstruction inside EvaluationService; expose a new public compute_run_summary_v2() wrapper in eval_v2.py; return loaded artifacts without summary fields.
• Consequences: Phase 2 introduces a documented internal dependency on a private eval-v2 helper. This is acceptable as a temporary compatibility choice; a public wrapper can be added later if needed.
• Blueprint impact: Preserves the Phase 2 constraint of not rewriting eval-v2 metrics, not renaming columns, and not changing artifact schemas.

DEC-005

• Date: 2026-06-19
• Phase: Phase 2
• Decision: Add an optional progress callback to EvaluationService.evaluate_run() so the CLI can preserve hardening/evaluation progress messages while the service owns eval-v2 orchestration.
• Reason: Moving evaluation orchestration into the service should not remove useful CLI progress behavior, and the service must remain reusable by CLI, Streamlit, and future APIs.
• Alternatives: Print progress directly inside EvaluationService; keep hardening orchestration inside cli.py; remove live progress and show only a final summary.
• Consequences: The service emits coarse progress events, while CLI decides how to render them. This keeps the service free of Typer/Streamlit dependencies and gives Streamlit a future hook for progress display.
• Blueprint impact: Aligns with the shared-service architecture and preserves current CLI behavior without adding Streamlit or live-execution scope in Phase 2.

DEC-006

• Date: 2026-06-19
• Phase: Phase 3
• Decision: Make the Run Intelligence Streamlit page read-only for evaluation artifacts; do not trigger eval-v2 recomputation from Streamlit in Phase 3.
• Reason: Phase 3 should consume existing artifacts through EvaluationService.load_existing_evaluation() and avoid long-running evaluation, CLI subprocess calls, or business logic inside Streamlit.
• Alternatives: Add a Streamlit button to run eval-v2 when artifacts are missing; invoke the CLI from Streamlit; call EvaluationService.evaluate_run() directly from the page.
• Consequences: Runs without eval-v2 artifacts show a clear missing-evaluation state and guidance to run evaluation externally. Evaluation triggering can be revisited in a later phase with proper job/progress handling.
• Blueprint impact: Narrows Phase 3 to read-only Run Intelligence and defers evaluation triggering to a later execution/control phase.

DEC-007

• Date: 2026-06-20
• Phase: Phase 4
• Decision: Implement the Streamlit Run Launcher as a synchronous predefined-task launcher using RunExecutionService.execute_selected_tasks().
• Reason: The repository currently has a reliable shared selected-task execution service but no background job framework, cancellation model, or multi-user execution queue. Synchronous execution is the smallest safe implementation for Phase 4.
• Alternatives: Invoke the CLI through subprocess; call the LangGraph runner directly from Streamlit; introduce a background job server or queue; use benchmark parallel execution for selected tasks.
• Consequences: The page blocks while selected tasks run and shows progress through the service callback. Background execution, cancellation, pause/resume, and parallel selected-task execution are deferred.
• Blueprint impact: Satisfies Phase 4 run-launcher capability while deferring Phase 5+ live/background execution infrastructure.

DEC-008

• Date: 2026-06-20
• Phase: Phase 5
• Decision: Keep live execution observability synchronous and event/manifest-driven rather than introducing a background job framework.
• Reason: The repository currently has reliable synchronous selected-task execution and persisted run events/manifests, but no lifecycle-safe background executor, cancellation model, or multi-user queue.
• Alternatives: Add a background thread manager; introduce a queue/server; call CLI commands asynchronously; defer all live execution UI.
• Consequences: The Streamlit page shows live progress during the current session using callbacks and persisted artifacts, but true background execution, cancellation, pause/resume, and multi-user scheduling remain deferred.
• Blueprint impact: Satisfies Phase 5 observability goals while preserving the shared-service architecture and avoiding unsafe execution infrastructure.

DEC-009

• Date: 2026-06-20
• Phase: Phase 6
• Decision: Make difficulty optional in task.json (defaults to "unknown") in both benchmark/dataset.py and external_dataset_validator.py.
• Reason: External benchmark datasets sourced from third parties or custom task intake may not include a difficulty field. Requiring it blocked all 60 tasks from loading during live testing. The field is metadata-only and does not affect execution.
• Alternatives: Reject tasks without difficulty; require callers to supply a default; keep difficulty mandatory and document the requirement.
• Consequences: DABenchPublicDataset.get_task() and the external validator both treat missing difficulty as "unknown". The change is backward-compatible — existing datasets with difficulty are unaffected.
• Blueprint impact: Narrows the dataset contract breaking-change risk; keeps the approved-config path unchanged.

DEC-010

• Date: 2026-06-20
• Phase: Phase 6
• Decision: Apply external dataset root as a temporary in-memory config override (dataclasses.replace()) rather than writing a new config file to disk.
• Reason: Writing a config file would mutate the workspace and require cleanup. An in-memory override is reversible, leaves no side-effects, and is invisible to other processes.
• Alternatives: Write a temp config file; pass dataset root directly to RunExecutionService (would require a new service API); create a new runner class.
• Consequences: get_run_execution_service_with_external_dataset() in service_adapters.py creates a patched config, constructs the service, and discards the override after the call returns. The approved output directory is preserved from the approved config.

DEC-011

• Date: 2026-06-20
• Phase: Phase 7
• Decision: Execute free-form custom tasks by materializing them into a benchmark-compatible synthetic dataset folder and routing execution through the existing shared run service using an in-memory config override.
• Reason: The current runner expects dataset-backed tasks. Materialization preserves existing execution, tracing, artifact writing, and observability without introducing a second runner. The knowledge.md file is optional — only the context/ directory is required.
• Alternatives: Build a new custom runner; invoke the CLI; call the runner directly from Streamlit; modify eval-v2 to score custom tasks; require users to manually create full benchmark datasets.
• Consequences: Custom tasks can be run and inspected with standard artifacts, but correctness evaluation remains unsupported unless compatible gold data is added later. The approach reuses all Phase 4–6 infrastructure without modification to RunExecutionService or its callers.
• Blueprint impact: Completes the deferred free-form custom task intake capability while preserving Phase 6 external dataset support and existing artifact contracts.
• Blueprint impact: Satisfies Phase 6 without introducing new execution paths, file I/O side-effects, or breaking changes to the service API.

DEC-012

• Date: 2026-06-20
• Phase: Phase 8
• Decision: Implement checkpoint review and human steering as advisory, additive annotations derived from existing trace and event artifacts, without true pause/resume execution.
• Reason: The current execution stack is synchronous and does not expose a safe resumable checkpoint protocol. Additive annotations provide human-in-the-loop review value without destabilizing the runner or mutating execution history.
• Alternatives: Add true pause/resume; add a background job server; modify runner state; alter trace schema; defer all steering functionality.
• Consequences: Users can inspect checkpoints, save review notes, and record steering instructions for future reruns, but completed executions are not altered and steering is not applied automatically.
• Blueprint impact: Delivers judge-visible checkpoint and human review capability while deferring executable intervention infrastructure to a later phase.

DEC-013

• Date: 2026-06-21
• Phase: Phase 9
• Decision: Implement Guided Rerun Planning and safe guided rerun execution as new isolated runs using the existing Custom Task execution path (Phase 7). Plans follow approval workflow: draft → approved → executed. Execution materializes guided custom tasks with injected guidance context and executes through RunExecutionService without modifying original run artifacts or introducing pause/resume capabilities.
• Reason: Phase 8 creates advisory steering instructions but does not apply them. Phase 9 closes the review-to-improvement loop by allowing an approved plan to execute as a new guided custom task while preserving the original run. This leverages the proven Phase 7 Custom Task infrastructure for safe, isolated execution without requiring runner modifications or background job infrastructure.
• Alternatives: Defer execution entirely and provide only manual handoff; add pause/resume to existing runner; modify runner to accept guidance directly; implement automatic prompt rewriting with LLM; build background execution infrastructure first; defer to Phase 10+.
• Consequences: Users can create, approve, and execute guided rerun plans. Original artifacts remain unchanged (trace.json, prediction.csv, checkpoint_annotations.json, run_manifest.json, run_events.jsonl, eval-v2 artifacts). New runs are isolated and linked back to the source plan via execution metadata recorded in rerun_plans.json. Output comparison and eval-v2 metric-level diff are deferred to Phase 10. Original blueprint Phase 9 scope (guided vs autonomous comparative evaluation) is deferred to a future phase.
• Blueprint impact: Refines the original Phase 9 comparative-evaluation scope into a workflow automation phase. Provides immediate value by enabling actionable reruns from human review. Guided-vs-autonomous comparative analysis and cohort metrics remain valuable but are deferred to preserve focus on proven safe execution paths.

DEC-014

• Date: 2026-06-22
• Phase: Phase 10
• Decision: Implement full original-vs-guided rerun comparison and local artifact-grounded Ask This Run / Ask This Comparison without executing tasks or calling external LLM/API. Comparison is read-only, uses pure builder functions, performs graceful degradation for missing artifacts, and preserves artifact integrity. Q&A uses local rules-based intent matching with 20+ supported intents and non-causal language for steering influence summaries. Saved comparison reports are stored additively in rerun_comparisons.json.
• Reason: Phase 9 creates guided reruns. Phase 10 closes the loop by explaining what changed, using existing artifacts only, while preserving artifact integrity. This enables structured comparison and interactive Q&A without execution risks or external dependencies.
• Alternatives: Auto-run eval-v2 on every comparison; call external LLM for Q&A; implement causality analysis; defer comparison entirely; provide only manual diff viewing; require eval-v2 artifacts to exist.
• Consequences: Users can compare original and guided runs across prediction/trace/tool/runtime/eval dimensions, ask natural language questions about changes, view steering influence summaries with heuristic (non-causal) language, and save comparison reports for later review. Phase 10 does not execute tasks, does not mutate artifacts (except additive rerun_comparisons.json), does not call external APIs, and does not claim causality for steering effects. Comparison degrades gracefully when artifacts are missing (e.g., no eval-v2 artifacts).
• Blueprint impact: Completes the human-in-the-loop improvement cycle (Phase 8 checkpoints → Phase 9 guided reruns → Phase 10 comparison). Provides structured comparison and Q&A capabilities while maintaining strict read-only constraints and artifact integrity.

DEC-015

• Date: 2026-06-22
• Phase: Phase 11
• Decision: Implement run-level cohort guided rerun evaluation as a Run Intelligence extension using existing Phase 10 comparison artifacts and local rules-based cohort Q&A.
• Reason: Phase 10 explains one original-vs-guided pair. Phase 11 aggregates existing guided reruns across a run to show observed cohort-level changes in predictions, score deltas when eval artifacts exist, runtime, tool usage, failures, and steering coverage.
• Alternatives: Add a new top-level Guided Evaluation tab; extend task-level Rerun Comparison; execute new guided reruns; auto-run eval-v2; perform causal analysis; require paired autonomous/guided runs for all tasks.
• Consequences: Users get benchmark/run-level evidence without creating new executions or causal claims. The system remains artifact-grounded, read-only except for an optional regenerable cohort report snapshot (cohort_guided_comparison.json). Cohort analysis discovers existing executed guided rerun plans from rerun_plans.json, reuses Phase 10 pairing and comparison logic, aggregates metrics with graceful degradation for missing eval artifacts, provides local rules-based cohort Q&A with 12+ supported intents (cohort size, prediction changes, score deltas, runtime overhead, failure rate, steering coverage, missing artifacts), and avoids all causal language. Disclaimer: "Observed associations do not establish causation. Guided reruns differ in timing, model state, and execution context."
• Blueprint impact: Moves the Observatory from single-rerun explanation toward evidence-backed guided-vs-autonomous evaluation while preserving the service/persistence boundary. Cohort analysis is a read-only aggregation layer that does not execute tasks, does not mutate existing artifacts (trace.json, prediction.csv, rerun_plans.json, rerun_comparisons.json, checkpoint_annotations.json, run_manifest.json, run_events.jsonl, eval-v2 artifacts), and does not call external APIs. Integration into Run Intelligence page (not a new tab) maintains run-level focus. Future work can add paired-comparison cohorts, causal steering experiments, or intervention delta tracking.

DEC-016

• Date: 2026-06-23
• Phase: Phase 12
• Decision: Implement a judge-facing Evidence Pack with cross-phase narrative, evidence hierarchy, statistical readiness assessment, claim safety framework, and markdown/JSON exports.
• Reason: Phases 8–11 provide checkpoint steering, guided rerun execution, pairwise comparison, and cohort evaluation, but judges need a consolidated, defensible artifact explaining the workflow, evidence, limitations, and statistical readiness.
• Alternatives: Add a standalone Evidence Pack tab; generate PDF/PowerPoint; use external LLM summarization; perform causal inference; defer evidence packaging.
• Consequences: The Observatory can export a reproducible judge-facing report while remaining artifact-grounded, read-only except for explicit export, and conservative about causal/statistical claims. Evidence Pack consolidates Phases 8–11 into structured narrative timeline (Phase 8 Checkpoint Review → Phase 9 Guided Rerun Planning → Phase 10 Original-vs-Guided Comparison → Phase 11 Run-Level Cohort Evaluation), classifies evidence by level (primary: trace.json, prediction.csv, task_metrics.csv, trajectory.csv, tool_calls.csv, run_manifest.json, run_events.jsonl; derived: checkpoint_annotations.json, rerun_plans.json; comparison: rerun_comparisons.json; aggregate: cohort_guided_comparison.json, evidence_pack.json, evidence_pack.md), assesses statistical readiness with N≥20 threshold and eval coverage ≥80% requirement for significance testing (defaults to descriptive_only for smaller N), provides claim safety framework distinguishing allowed observational claims from unsupported causal claims with explicit disclaimers, exports to evidence_pack.json and evidence_pack.md only on explicit button click, integrates into Run Intelligence page (not a new tab) visible in Detailed and Research view levels, performs no execution (no subprocess, no RunExecutionService, no CustomTaskService, no eval-v2 auto-run, no external LLM/API calls), and mutates no source artifacts except creating evidence_pack.json and evidence_pack.md exports.
• Blueprint impact: Converts the Observatory from an operational analysis UI into a presentation-ready evidence system with statistical rigor and claim safety, while maintaining the service/persistence boundary and read-only constraints.

DEC-017

• Date: 2026-06-23
• Phase: Phase 13
• Decision: Add a Demo tab with Recorded Evidence Mode (default), optional Live Mini Demo Mode, demo health scoring, competition requirement checklist, guided navigation, and demo bundle export.
• Reason: The Observatory has strong Phase 8-12 capabilities, but the competition video requires a concise 5-8 minute story covering the problem, real end-to-end demonstration, agent trajectory, non-trivial case, quantitative evidence, and limitations.
• Alternatives: Keep demo guidance outside the app; only add buttons across existing pages; make Run Intelligence the demo entry point; add auto-execution.
• Consequences: The app gains a clear demo entry point without disturbing the existing architecture. Live actions remain explicit and service-backed; the default recorded evidence path remains stable for video recording. No subprocess, CLI, or raw execution calls.
• Blueprint impact: Converts the Observatory from evidence-ready to demo-ready for competition submission.

DEC-018

• Date: 2026-06-24
• Phase: Phase 14
• Decision: Rebrand the Observatory as Data Agent Observatory (DAO) and conduct comprehensive release readiness audit covering naming, documentation, imports, demo flow, competition readiness, and backward compatibility.
• Reason: The product has matured from a debugging tool into a judge-facing execution, observability, and evidence system. The "Observatory" name was always a working title and should be replaced with Data Agent Observatory to reflect mission-control branding, scope, and purpose.
• Alternatives: Keep "Observatory" as the product name; introduce "DAO" as an alias only; defer branding until after competition.
• Consequences: All user-facing text, documentation, and CLI guidance now refer to "Data Agent Observatory" or "DAO". Internal module paths remain unchanged (src/data_agent_baseline/observatory/) for backward compatibility. Comprehensive audit checklist ensures no breaking changes and validates all Phase 0-13 capabilities remain intact.
• Blueprint impact: Completes the transition from development-phase internal tooling to release-ready branded product suitable for judge evaluation and external presentation.

DEC-019

• Date: 2026-06-24
• Phase: Post-Phase-13 Roadmap
• Decision: Carry forward unfinished high-risk blueprint intents into post-Phase-16 roadmap phases rather than forcing them into the stable demo release.
• Reason: The stable Observatory now supports execution, observability, checkpoint review, guided reruns, comparison, cohort evidence, Evidence Pack export, and Demo flow. Some original blueprint intents — true pause/resume, in-execution steering, canonical evaluation folder migration, optional LLM Q&A, and cross-run statistical studies — require deeper runtime or evaluation changes and should be attempted only after release stabilization.
• Alternatives: Implement all blueprint intents before branding/release audit; mark deferred intents as out of scope permanently; merge risky runtime changes into Phase 13 Demo.
• Consequences: The release remains stable and demo-ready, while the documentation honestly preserves the original ambitions and provides a clear continuation roadmap. Riskier capabilities can be developed behind feature flags or isolated branches and rolled back if they destabilize the system.
• Blueprint impact: Distinguishes stable realized architecture from future blueprint-completion work.
• Blueprint impact: Converts the Observatory from evidence-ready to demo-ready for competition submission.

DEC-020

• Date: 2026-06-24
• Phase: Phase 16
• Decision: Implement Live Trace Flow as a trace-native step/action backbone visualization with future HITL-compatible node schema (inert fields) but no actual pause/replan/execution control.
• Reason: Current event coverage is Level 1 only (run/task events); no stage/tool/step events are emitted during execution. The trace.json becomes available only after task completion. Building Phase 16 around trace-native step_index + action provides a solid foundation that works with final trace.json today and will gracefully accept live step events in a future instrumentation phase.
• Alternatives: Add step event instrumentation to runner first; build fake/simulated live execution UI; defer all live trace flow to post-release; build separate live vs final trace pages.
• Consequences: Phase 16 delivers Visual Step DAG, Live Replay Timeline, Agent Reasoning Overlay, and Final Trace Handoff as read-only observability. During execution: shows placeholder message. After completion: renders full Step DAG from trace.json. All nodes include future HITL fields (checkpoint_id, can_pause, can_resume, waiting_for_human, human_input_required, replan_candidate) but these default to None/False and do not affect current execution. Step event instrumentation is explicitly deferred to a future phase. No pause/resume buttons, no human input forms, no replan execution, no blocking for input. Execution Story provides presentation layer (story_label, story_group, description) separate from data layer (raw_action, step_index, phase). Agent names separated into overlay (not main DAG) to avoid cluttering step sequence.
• Blueprint impact: Provides trace-native live observability foundation without requiring runner instrumentation changes or introducing execution control risk. Future HITL design pattern established: add fields early (inert) then activate in later implementation phase.

DEC-021

• Date: 2026-06-27
• Phase: Phase 17
• Decision: Implement live human-in-the-loop intervention as a single guided checkpoint type (planner_review) triggered only after planner, with run-level guided policy set at launch, max_workers=1 enforced for guided runs, explicit cancellation semantics, and timeout auto-approve.
• Reason: The project needs real in-execution intervention without risking arbitrary pause points, multi-checkpoint race conditions, or multi-task human coordination complexity.
• Alternatives: Add arbitrary pause at any step; add multiple checkpoint types in Phase 17; keep only post-hoc advisory checkpoints (Phase 8/9 pattern).
• Consequences: HITL-enabled guided runs pause at exactly one planner checkpoint per task, persist checkpoint/intervention/delta artifacts under task directories, emit canonical HITL events in run_events.jsonl, and resume via approve/revise/cancel/timeout auto-approve. The Guided Live Trace Flow start panel also allows the same task path to be launched without HITL by clearing the Guided Run (HITL) checkbox; in that mode, no planner checkpoint is created. Timeout policy defaults are timeout_seconds=60 and default_action_on_timeout=continue. Human cancel maps to TaskStatus.CANCELLED with reason=cancelled_by_human and emits task_cancelled (not task_failed). Process-restart durable resume is not claimed; browser refresh recovery is supported while process remains alive.
• Blueprint impact: Activates HITL execution behavior in a constrained, service/runner-owned manner while preserving existing Live Trace Flow and post-hoc Phase 8/9 workflows.

DEC-022

• Date: 2026-07-01
• Phase: Phase 19
• Decision: Consolidate Observatory top-level navigation to four primary pages while keeping execution/evaluation behavior unchanged and preserving Live Trace Flow pages as fallback access only.
• Reason: The prior many-tab layout increased maintenance overhead and user cognitive load. Consolidation improves usability and keeps existing renderer logic reusable through shell composition.
• Alternatives: Keep current many-tab top-level layout; refactor runtime pages and merge Live Trace internals into Run Launcher; move Evidence Pack logic out of Run Intelligence immediately.
• Consequences: Primary tabs are now Run Launcher, Run Intelligence, Task Intelligence, and Demo / Future Proof. Task-level pages are grouped under Task Intelligence internal tabs. Checkpoint/rerun/comparison/demo workflows are grouped under Demo / Future Proof internal tabs. Live Trace Flow and Guided Live Trace Flow remain available through Advanced / Fallback Views with no internals refactored.
• Blueprint impact: Delivers Phase 19 navigation consolidation as a UI-shell change only; execution services, evaluation services, HITL behavior, and artifact schema remain unchanged. Phase 20 natural-language artifact Q&A remains out of scope.

DEC-023

• Date: 2026-07-01
• Phase: Phase 19
• Decision: Harden Run Intelligence missing-evaluation behavior into a fully read-only no-eval fallback and prohibit in-page eval execution (no subprocess/CLI/os.system/background job paths).
• Reason: Runs commonly appear before eval-v2 artifacts exist. The page must remain useful without mutating artifacts or launching execution from Streamlit.
• Alternatives: Keep a Run Evaluation button using subprocess CLI invocation; auto-trigger eval-v2 on page load; add background job execution in Streamlit.
• Consequences: Missing-eval runs now render execution summary, task status, artifact coverage, event timeline summary, and explicit eval-v2 guidance. Existing eval-present behavior is unchanged. Evaluation remains opt-in outside this page.
• Blueprint impact: Strengthens service-boundary and read-only observability guarantees while improving operator guidance for incomplete runs.