docs/implementation/PHASE15_RUN_EVALUATION_WORKFLOW.md

Phase 15: Run + Evaluation Workflow

Overview

Phase 15 adds explicit support for running evaluation (eval-v2) alongside task execution in the Data Agent Observatory. This enables a streamlined workflow where users can optionally evaluate runs immediately after execution, or manually trigger evaluation from the Run Intelligence page.

Key Capabilities

1. Run Only (Default)

Execute selected tasks without evaluation. This remains the default behavior.

2. Run + Eval-v2 (Optional)

When gold_root is configured and the user checks "Run eval-v2 after execution", the system will:

• Execute the selected tasks
• Automatically run eval-v2 if:
  • Run completed successfully
  • gold_root exists
  • Evaluation is enabled in config

3. Load Existing Eval-v2 Artifacts

Run Intelligence page continues to load and display existing evaluation artifacts.

4. Explicit Evaluation Trigger

When eval artifacts are missing and gold_root is available, Run Intelligence shows a "Run Evaluation Now" button.

Architecture

Streamlit UI → Application Services → Existing eval-v2 Engine

Hard Boundaries (Preserved)

• No subprocess calls
• No CLI invocation from Streamlit
• No duplicate evaluator implementation
• No duplicate execution loop
• No automatic evaluation on page load
• No mutation of trace.json or prediction.csv
• No new metric formulas

Service Layers

1. EvaluationService: Added evaluate_run_if_configured() method
2. Service Adapters: Added execute_with_optional_evaluation() orchestration
3. UI Pages: Updated to expose evaluation controls

Configuration

Add an evaluation section to your config YAML:

evaluation:
  enabled: true
  gold_root: /data3/dataFAIR/kdd-dev/public/output
  run_after_execution: false
  fail_if_ground_truth_missing: false
  lambda_penalty: 0.1
  mode: standard  # standard | verbose | research

Configuration Fields

• enabled: Enable evaluation support (default: false)
• gold_root: Path to ground truth data directory (default: None)
• run_after_execution: Auto-run eval after execution (default: false)
• fail_if_ground_truth_missing: Raise error if gold_root missing (default: false)
• lambda_penalty: Penalty for eval-v2 scoring (default: 0.1)
• mode: Evaluation mode (default: "standard")

User Workflows

Workflow 1: Run Only

1. Select tasks in Run Launcher
2. Leave "Run eval-v2 after execution" unchecked
3. Click "Start Run"
4. Review run artifacts

Workflow 2: Run + Evaluate

1. Configure evaluation.gold_root in config
2. Select tasks in Run Launcher
3. Check "Run eval-v2 after execution"
4. Click "Start Run"
5. System executes tasks and then runs evaluation
6. Review both run and evaluation artifacts

Workflow 3: Manual Evaluation Trigger

1. Open Run Intelligence for a completed run
2. If eval artifacts missing and gold_root configured:
   • Click "Run Evaluation Now"
   • Wait for evaluation to complete
   • Page automatically refreshes

UI Changes

Run Launcher Page

Added Evaluation section showing:

• Ground truth configuration status
• Gold root path (if configured)
• Evaluation mode and lambda penalty
• Checkbox: "Run eval-v2 after execution"
  • Disabled if gold_root not configured or not found
  • Default value from config.evaluation.run_after_execution

Run Intelligence Page

When eval artifacts missing:

• Shows "Evaluation Artifacts Missing" warning
• Shows eval command guidance: dabench eval-v2 <run_id>
• Shows a "Run Evaluation" button
• Button runs dabench eval-v2 <run_id> and auto-refreshes on success
• View level selector is hidden in missing-evaluation state and appears only after evaluation artifacts are available

Demo Page

Added read-only evaluation status badge:

• ✅ Evaluated
• ⚠️ Not evaluated
• ❓ Evaluation unavailable

Artifact Layout

Phase 15 preserves the current root-level eval-v2 artifact layout:

run_dir/
  task_metrics.csv
  trajectory.csv
  tool_calls.csv
  comprehensive_evaluation.csv
  validation_report.md
  auditor_validation_report.md
  artifact_reconciliation_report.txt
  engineering_health_report.txt

No migration to evaluation/ folder in Phase 15. This is deferred to Phase 17.

Failure Handling

Custom Tasks Without Gold

• Evaluation checkbox disabled
• Status: "Evaluation unavailable"
• Not treated as an error

Missing Gold Root

• Graceful skip (default): Evaluation returns None
• Strict mode (fail_if_ground_truth_missing=true): Raises FileNotFoundError

Run Failed

• Evaluation automatically skipped
• No error raised

Evaluation States

User-facing states displayed in UI:

• Evaluation ready: Config valid, gold_root exists
• Evaluation unavailable: No gold_root or missing
• Evaluation exists: Artifacts already present
• Evaluation running: In progress (with spinner)
• Evaluation completed: Success
• Evaluation failed: Error occurred
• Evaluation skipped: Conditions not met

Implementation Details

Config Schema

Added EvaluationConfig dataclass:

@dataclass(frozen=True, slots=True)
class EvaluationConfig:
    enabled: bool = False
    gold_root: Path | None = None
    run_after_execution: bool = False
    fail_if_ground_truth_missing: bool = False
    lambda_penalty: float = 0.1
    mode: str = "standard"

Service Method

Added to EvaluationService:

def evaluate_run_if_configured(
    self,
    run_result: RunExecutionResult,
    eval_config: EvaluationConfig,
    *,
    progress_callback: ProgressCallback | None = None,
) -> EvaluationBundle | None:
    ...

Orchestration

Added to service_adapters.py:

def execute_with_optional_evaluation(
    run_service: RunExecutionService,
    eval_service: EvaluationService,
    spec: RunSpec,
    eval_config: EvaluationConfig,
    ...
) -> tuple[RunExecutionResult, EvaluationBundle | None]:
    ...

Testing

Unit Tests

• tests/unit/test_config.py: Config schema and parsing
• tests/unit/application/test_evaluation_service.py: Conditional evaluation logic
• tests/unit/observatory/test_service_adapters_phase15.py: Orchestration

Integration Tests

• tests/integration/test_phase15_run_evaluation_workflow.py: End-to-end workflow

UI Tests

• tests/test_observatory_run_launcher_page.py: Run Launcher evaluation controls
• tests/test_observatory_run_intelligence_page.py: Manual evaluation trigger
• tests/test_observatory_demo_page.py: Evaluation status badge

Known Limitations

1. No background evaluation: Evaluation runs synchronously during execution
2. No evaluation progress UI: Progress callbacks not yet wired to UI
3. No evaluation retry: Failed evaluations require manual re-trigger
4. No canonical evaluation/ folder: Deferred to Phase 17
5. Custom tasks: Evaluation unavailable (by design)

Safety

Phase 15 maintains architectural boundaries:

• ✅ No subprocess/CLI from Streamlit
• ✅ No duplicate eval-v2 implementation
• ✅ No mutation of existing artifacts
• ✅ No automatic evaluation on page load
• ✅ Service layer properly isolated

Migration Path

Existing configs without evaluation section:

• Work without modification
• Evaluation disabled by default
• No breaking changes

To enable evaluation:

1. Add evaluation section to config
2. Set enabled: true and configure gold_root
3. Optionally set run_after_execution: true for auto-eval

Future Work (Phase 17+)

• Migrate to canonical evaluation/ folder structure
• Add evaluation progress UI
• Support background/async evaluation
• Add evaluation retry mechanism
• Support custom gold data for custom tasks