docs/overview/PHASE1_EVALUATION_GUIDE.md

# Phase 1 Baseline Evaluation Harness - Usage Guide

## Overview

This Phase 1 integration enables evaluation of baseline ReAct agent runs using a standardized evaluation pipeline. The system adapts baseline traces to a canonical schema and computes metrics compatible with future multi-agent systems.

## Quick Start

### Run Baseline Evaluation

```bash
cd /data3/dataFAIR/kdd-dev/public

# Evaluate a baseline run
dabench eval-baseline <run_id>

# Example:
dabench eval-baseline 20260613T114457Z
```

### Output

The evaluation generates three files in `<run_dir>/baseline_evaluation/`:

```
baseline_evaluation/
├── task_results.csv          # Per-task metrics
├── summary_metrics.json      # Aggregated statistics
└── evaluation_report.md      # Scientific analysis
```

## Command Options

```bash
dabench eval-baseline <run_id> [OPTIONS]

Arguments:
  run_id              Run ID (directory name under artifacts/runs/)

Options:
  --task-root PATH    Root directory containing task metadata
                      Default: /data3/dataFAIR/kdd-dev/public/input_full
  
  --gold-root PATH    Root directory containing gold answer files
                      Default: /data3/dataFAIR/kdd-dev/public/output
  
  --output-dir PATH   Output directory for evaluation results
                      Default: <run_dir>/baseline_evaluation
```

## Architecture

### Components

1. **Canonical Evaluation Schema** (`src/data_agent_baseline/evaluation/__init__.py`)
   - Unified trace representation for all agent types
   - Compatible with baseline and LangGraph traces
   - Extensible for future agent architectures

2. **Baseline Trace Adapter** (`src/data_agent_baseline/evaluation/baseline_adapter.py`)
   - Converts baseline traces to canonical schema
   - Enriches traces with task metadata
   - Computes derivable metrics from steps

3. **Phase 1 Evaluator** (`src/data_agent_baseline/evaluation/phase1_evaluator.py`)
   - Computes baseline-compatible metrics
   - Scores predictions against gold answers
   - Classifies failures and buckets

4. **Report Generator** (`src/data_agent_baseline/evaluation/report_generator.py`)
   - Generates CSV, JSON, and Markdown reports
   - Computes summary statistics
   - Creates scientific evaluation reports

### Data Flow

```
Baseline Trace (trace.json)
    ↓
BaselineTraceAdapter.normalize()
    ↓
CanonicalTrace (normalized schema)
    ↓
Phase1Evaluator.evaluate_task()
    ↓
EvaluationResult (computed metrics)
    ↓
Phase1ReportGenerator.generate_all_reports()
    ↓
Output Files (CSV, JSON, MD)
```

## Metrics

### Phase 1 Metrics (Available)

✅ **Accuracy Metrics:**
- Overall accuracy
- Per-difficulty accuracy (Easy/Medium/Hard/Extreme)
- Answer precision, recall, F1
- Column precision, recall, F1
- Row precision, recall, F1

✅ **Efficiency Metrics:**
- Average steps per task
- Average tool calls per task
- Average runtime per task
- Tool diversity (unique tools / total calls)
- Tool efficiency (1 - failures / total)

✅ **Reliability Metrics:**
- Success rate
- Execution failure rate
- Timeout rate
- Tool error rate

✅ **Failure Classification:**
- Failure categories (timeout, planning, execution, wrong_answer)
- Failure stages (execution, answer_generation)
- Root causes (timeout, insufficient_steps, filter_logic, etc.)
- Recoverability assessment

✅ **Bucket Distribution:**
- perfect (score >= 0.999)
- wrong_row_count
- wrong_col_count
- partial_correct (score > 0.5)
- mostly_wrong (score > 0.0)
- completely_wrong
- no_gold (no gold answer available)

### Phase 1 Limitations (Not Available)

❌ **Token Metrics:**
- Total tokens
- Prompt tokens
- Completion tokens
- Estimated cost

❌ **Phase-Specific Metrics:**
- Explore phase timing
- Planner phase timing
- Execute phase timing
- Critic phase timing

❌ **Advanced Metrics:**
- Confidence scores
- Confidence calibration
- Replan counts
- Recovery attempts
- Self-correction loops

**Why?** Baseline traces don't include:
- Token usage tracking
- Phase labels on steps
- Confidence scores
- Recovery/replan signals

These will be available in future phases with enhanced agents.

## Output Format

### task_results.csv

Per-task metrics in CSV format:

```csv
run_id,task_id,agent_type,difficulty,execution_success,execution_time,final_score,
answer_f1,column_f1,row_f1,trajectory_length,tool_calls,tool_failures,
tool_efficiency,unique_tools_used,tool_diversity,total_tokens,llm_calls,
timeout_occurred,failure_category,failure_stage,root_cause,bucket,
pred_rows,pred_cols,gold_rows,gold_cols
```

### summary_metrics.json

Aggregated statistics:

```json
{
  "overall": {
    "total_tasks": 10,
    "successful_tasks": 9,
    "success_rate": 0.9,
    "perfect_rate": 0.7,
    "average_score": 0.85,
    "average_trajectory_length": 8.5,
    "average_tool_calls": 6.2,
    "average_execution_time": 15.3,
    "tool_error_rate": 0.05,
    "timeout_rate": 0.0,
    "execution_failure_rate": 0.1
  },
  "by_difficulty": {
    "Easy": {...},
    "Medium": {...},
    "Hard": {...}
  },
  "failure_analysis": {
    "failure_categories": {...},
    "failure_stages": {...},
    "root_causes": {...}
  },
  "bucket_distribution": {
    "perfect": 7,
    "wrong_row_count": 2,
    "no_gold": 1
  }
}
```

### evaluation_report.md

Scientific evaluation report with:

- Experimental setup
- Overall results table
- Performance by difficulty
- Reliability analysis
- Failure analysis (top categories and causes)
- Efficiency analysis
- Phase 1 limitations
- Recommendations for improvement

## Comparing Baseline vs Multi-Agent

The canonical schema enables direct comparison:

```python
from data_agent_baseline.evaluation.baseline_adapter import BaselineTraceAdapter
from data_agent_baseline.evaluation.phase1_evaluator import Phase1Evaluator

# Evaluate baseline run
baseline_adapter = BaselineTraceAdapter(task_root=task_root)
baseline_traces = baseline_adapter.normalize_run(baseline_run_path, run_id="baseline")

# Evaluate LangGraph run (future)
langgraph_adapter = LangGraphTraceAdapter(task_root=task_root)
langgraph_traces = langgraph_adapter.normalize_run(langgraph_run_path, run_id="langgraph")

# Compare using same evaluator
evaluator = Phase1Evaluator(gold_root=gold_root)
baseline_results = evaluator.evaluate_run(baseline_traces)
langgraph_results = evaluator.evaluate_run(langgraph_traces)

# Generate comparison report
compare_agents(baseline_results, langgraph_results)
```

## Extending to New Agent Types

To add a new agent type:

1. **Create an adapter** implementing:
   ```python
   class NewAgentAdapter:
       def normalize(self, trace_path: Path, run_id: str) -> CanonicalTrace:
           # Convert agent-specific trace to canonical schema
           pass
   ```

2. **Register agent type** in canonical schema:
   ```python
   AgentType = Literal["baseline_react", "langgraph_agent", "new_agent"]
   ```

3. **Evaluate using existing pipeline**:
   ```python
   adapter = NewAgentAdapter(task_root=task_root)
   traces = adapter.normalize_run(run_path, run_id)
   evaluator = Phase1Evaluator(gold_root=gold_root)
   results = evaluator.evaluate_run(traces)
   ```

## Programmatic Usage

```python
from pathlib import Path
from data_agent_baseline.evaluation.baseline_adapter import BaselineTraceAdapter
from data_agent_baseline.evaluation.phase1_evaluator import Phase1Evaluator
from data_agent_baseline.evaluation.report_generator import Phase1ReportGenerator

# Setup paths
run_path = Path("/data3/dataFAIR/kdd-dev/public/artifacts/runs/20260613T114457Z")
task_root = Path("/data3/dataFAIR/kdd-dev/public/input_full")
gold_root = Path("/data3/dataFAIR/kdd-dev/public/output")
output_dir = run_path / "baseline_evaluation"

# Normalize traces
adapter = BaselineTraceAdapter(task_root=task_root)
canonical_traces = adapter.normalize_run(run_path, run_id="20260613T114457Z")

# Evaluate
evaluator = Phase1Evaluator(gold_root=gold_root)
results = evaluator.evaluate_run(canonical_traces)

# Generate reports
generator = Phase1ReportGenerator(output_dir=output_dir)
outputs = generator.generate_all_reports(results, run_id="20260613T114457Z")

print(f"Task results: {outputs['task_results']}")
print(f"Summary: {outputs['summary_metrics']}")
print(f"Report: {outputs['evaluation_report']}")
```

## Troubleshooting

### "Trace file not found"

Ensure the run directory contains task subdirectories with `trace.json` files:
```
<run_id>/
├── task_11/
│   └── trace.json
├── task_22/
│   └── trace.json
└── ...
```

### "No gold answer" / bucket="no_gold"

The gold answer file is missing. Ensure `<gold_root>/<task_id>/gold.csv` exists.

This is not an error - tasks without gold answers are still evaluated for execution metrics.

### "Error normalizing traces"

Check that trace.json is valid JSON with expected fields:
- task_id
- answer (optional)
- steps (array)
- succeeded
- e2e_elapsed_seconds

### Score is 0.0 but execution succeeded

This usually means:
1. No gold answer available (bucket="no_gold"), OR
2. Prediction doesn't match gold at all (bucket="completely_wrong")

Check the bucket classification in task_results.csv.

## Future Enhancements

### Phase 2: Multi-Agent Support
- Add LangGraph trace adapter
- Compare baseline vs multi-agent performance
- Measure planning efficiency

### Phase 3: Token Tracking
- Instrument baseline with token counting
- Compute cost metrics
- Optimize for cost/performance tradeoff

### Phase 4: Confidence Calibration
- Add confidence scores to baseline
- Measure calibration accuracy
- Implement uncertainty quantification

### Phase 5: Human-in-the-Loop
- Track human interventions
- Measure autonomy score
- Analyze when help is needed

## References

- **Baseline architecture**: `BASELINE_ARCHITECTURE.md`
- **Canonical schema**: `src/data_agent_baseline/evaluation/__init__.py`
- **Existing evaluation**: `src/data_agent_baseline/langgraph_agent/eval_v2.py`
- **Scoring function**: `src/data_agent_baseline/langgraph_agent/evaluator.py`