Blame for docs/implementation/PHASE15_RUN_EVALUATION_WORKFLOW.md

# Phase 15: Run + Evaluation Workflow

## Overview

Phase 15 adds explicit support for running evaluation (eval-v2) alongside task execution in the Data Agent Observatory. This enables a streamlined workflow where users can optionally evaluate runs immediately after execution, or manually trigger evaluation from the Run Intelligence page.

## Key Capabilities

### 1. Run Only (Default)
Execute selected tasks without evaluation. This remains the default behavior.

### 2. Run + Eval-v2 (Optional)
When `gold_root` is configured and the user checks "Run eval-v2 after execution", the system will:
- Execute the selected tasks
- Automatically run eval-v2 if:
  - Run completed successfully
  - `gold_root` exists
  - Evaluation is enabled in config

### 3. Load Existing Eval-v2 Artifacts
Run Intelligence page continues to load and display existing evaluation artifacts.

### 4. Explicit Evaluation Trigger
When eval artifacts are missing and `gold_root` is available, Run Intelligence shows a "Run Evaluation Now" button.

## Architecture

```
Streamlit UI → Application Services → Existing eval-v2 Engine
```

### Hard Boundaries (Preserved)
- No subprocess calls
- No CLI invocation from Streamlit
- No duplicate evaluator implementation
- No duplicate execution loop
- No automatic evaluation on page load
- No mutation of trace.json or prediction.csv
- No new metric formulas

### Service Layers
1. **EvaluationService**: Added `evaluate_run_if_configured()` method
2. **Service Adapters**: Added `execute_with_optional_evaluation()` orchestration
3. **UI Pages**: Updated to expose evaluation controls

## Configuration

Add an `evaluation` section to your config YAML:

```yaml
evaluation:
  enabled: true
  gold_root: /data3/dataFAIR/kdd-dev/public/output
  run_after_execution: false
  fail_if_ground_truth_missing: false
  lambda_penalty: 0.1
  mode: standard  # standard | verbose | research
```

### Configuration Fields

- **enabled**: Enable evaluation support (default: false)
- **gold_root**: Path to ground truth data directory (default: None)
- **run_after_execution**: Auto-run eval after execution (default: false)
- **fail_if_ground_truth_missing**: Raise error if gold_root missing (default: false)
- **lambda_penalty**: Penalty for eval-v2 scoring (default: 0.1)
- **mode**: Evaluation mode (default: "standard")

## User Workflows

### Workflow 1: Run Only
1. Select tasks in Run Launcher
2. Leave "Run eval-v2 after execution" unchecked
3. Click "Start Run"
4. Review run artifacts

### Workflow 2: Run + Evaluate
1. Configure `evaluation.gold_root` in config
2. Select tasks in Run Launcher
3. Check "Run eval-v2 after execution"
4. Click "Start Run"
5. System executes tasks and then runs evaluation
6. Review both run and evaluation artifacts

### Workflow 3: Manual Evaluation Trigger
1. Open Run Intelligence for a completed run
2. If eval artifacts missing and `gold_root` configured:
   - Click "Run Evaluation Now"
   - Wait for evaluation to complete
   - Page automatically refreshes

## UI Changes

### Run Launcher Page
Added **Evaluation** section showing:
- Ground truth configuration status
- Gold root path (if configured)
- Evaluation mode and lambda penalty
- Checkbox: "Run eval-v2 after execution"
  - Disabled if gold_root not configured or not found
  - Default value from `config.evaluation.run_after_execution`

### Run Intelligence Page
When eval artifacts missing:
- Shows "Evaluation Artifacts Missing" warning
- Shows eval command guidance: `dabench eval-v2 <run_id>`
- Shows a "Run Evaluation" button
- Button runs `dabench eval-v2 <run_id>` and auto-refreshes on success
- `View level` selector is hidden in missing-evaluation state and appears only after evaluation artifacts are available

### Demo Page
Added read-only evaluation status badge:
- ✅ Evaluated
- ⚠️ Not evaluated
- ❓ Evaluation unavailable

## Artifact Layout

Phase 15 preserves the current root-level eval-v2 artifact layout:

```
run_dir/
  task_metrics.csv
  trajectory.csv
  tool_calls.csv
  comprehensive_evaluation.csv
  validation_report.md
  auditor_validation_report.md
  artifact_reconciliation_report.txt
  engineering_health_report.txt
```

**No migration to `evaluation/` folder in Phase 15.** This is deferred to Phase 17.

## Failure Handling

### Custom Tasks Without Gold
- Evaluation checkbox disabled
- Status: "Evaluation unavailable"
- Not treated as an error

### Missing Gold Root
- **Graceful skip** (default): Evaluation returns None
- **Strict mode** (`fail_if_ground_truth_missing=true`): Raises FileNotFoundError

### Run Failed
- Evaluation automatically skipped
- No error raised

## Evaluation States

User-facing states displayed in UI:

- **Evaluation ready**: Config valid, gold_root exists
- **Evaluation unavailable**: No gold_root or missing
- **Evaluation exists**: Artifacts already present
- **Evaluation running**: In progress (with spinner)
- **Evaluation completed**: Success
- **Evaluation failed**: Error occurred
- **Evaluation skipped**: Conditions not met

## Implementation Details

### Config Schema
Added `EvaluationConfig` dataclass:
```python
@dataclass(frozen=True, slots=True)
class EvaluationConfig:
    enabled: bool = False
    gold_root: Path | None = None
    run_after_execution: bool = False
    fail_if_ground_truth_missing: bool = False
    lambda_penalty: float = 0.1
    mode: str = "standard"
```

### Service Method
Added to `EvaluationService`:
```python
def evaluate_run_if_configured(
    self,
    run_result: RunExecutionResult,
    eval_config: EvaluationConfig,
    *,
    progress_callback: ProgressCallback | None = None,
) -> EvaluationBundle | None:
    ...
```

### Orchestration
Added to `service_adapters.py`:
```python
def execute_with_optional_evaluation(
    run_service: RunExecutionService,
    eval_service: EvaluationService,
    spec: RunSpec,
    eval_config: EvaluationConfig,
    ...
) -> tuple[RunExecutionResult, EvaluationBundle | None]:
    ...
```

## Testing

### Unit Tests
- `tests/unit/test_config.py`: Config schema and parsing
- `tests/unit/application/test_evaluation_service.py`: Conditional evaluation logic
- `tests/unit/observatory/test_service_adapters_phase15.py`: Orchestration

### Integration Tests
- `tests/integration/test_phase15_run_evaluation_workflow.py`: End-to-end workflow

### UI Tests
- `tests/test_observatory_run_launcher_page.py`: Run Launcher evaluation controls
- `tests/test_observatory_run_intelligence_page.py`: Manual evaluation trigger
- `tests/test_observatory_demo_page.py`: Evaluation status badge

## Known Limitations

1. **No background evaluation**: Evaluation runs synchronously during execution
2. **No evaluation progress UI**: Progress callbacks not yet wired to UI
3. **No evaluation retry**: Failed evaluations require manual re-trigger
4. **No canonical evaluation/ folder**: Deferred to Phase 17
5. **Custom tasks**: Evaluation unavailable (by design)

## Safety

Phase 15 maintains architectural boundaries:
- ✅ No subprocess/CLI from Streamlit
- ✅ No duplicate eval-v2 implementation
- ✅ No mutation of existing artifacts
- ✅ No automatic evaluation on page load
- ✅ Service layer properly isolated

## Migration Path

Existing configs without `evaluation` section:
- Work without modification
- Evaluation disabled by default
- No breaking changes

To enable evaluation:
1. Add `evaluation` section to config
2. Set `enabled: true` and configure `gold_root`
3. Optionally set `run_after_execution: true` for auto-eval

## Future Work (Phase 17+)

- Migrate to canonical `evaluation/` folder structure
- Add evaluation progress UI
- Support background/async evaluation
- Add evaluation retry mechanism
- Support custom gold data for custom tasks