docs/implementation/EVAL_CLEANUP_FIXES_SUMMARY.md

Final Eval-V2 Cleanup Fixes - Implementation Summary

Date: June 14, 2026

COMPLETED FIXES

✅ FIX 1: Separate Harness Health from Run Quality

Status: COMPLETE

Changes Made:

• Updated EngineeringHealthReport dataclass to include:

  • partial_correct_count and mean_final_score fields
  • Separated assessment logic

• Modified format_report() in eval_health_report.py:

  • Added "HARNESS HEALTH ASSESSMENT" section (score 9.7/10, Status: ✓ HEALTHY)
  • Added "RUN QUALITY ASSESSMENT" section (Status: ⚠ DEGRADED, 64% accuracy)
  • Shows: Answer Accuracy, Mean Final Score, Failed/Below-threshold tasks, Partial-correct tasks

• Created _compute_harness_health_score() method:

  • Evaluates infrastructure: reconciliation, validator errors, attribution coverage
  • Deducts points for infrastructure failures only

• Updated CLI (cli.py):

  • Displays both statuses separately in hardening suite output
  • Renamed "Time gaps" → "Unaccounted Overhead Time"

Verified Output:

HARNESS HEALTH ASSESSMENT
Harness Health Score: 9.7/10
Status: ✓ HEALTHY
  Infrastructure functional, no validator errors
Metrics:
  - Reconciliation pass rate: 100.0%
  - Validator errors: 0
  - Attribution coverage: 100.0%

RUN QUALITY ASSESSMENT
Run Quality: ⚠ DEGRADED
  Moderate answer accuracy
  Answer Accuracy: 64.0%
  Mean Final Score: 0.644
  Failed / Below-threshold Tasks: 18 / 50
  Partial-correct Tasks: 1

✅ FIX 2: Populate Failure Distribution by Stage (PARTIAL)

Status: WORKING - Shows stages, but needs improvement

Changes Made:

• Modified render_mas_failure_analysis() in eval_v2_viz.py:
  • Added inference logic to map buckets/root_causes to AAT stages
  • Maps:
    • wrong_column/wrong_schema → UNDERSTAND
    • filter_logic/low_recall → EXECUTE
    • wrong_row → EXECUTE
    • replan_limit/planner_failure → PLAN
    • verification failures → VERIFY

Verified Output:

Failure Distribution by Stage

Stage        Count       %
UNDERSTAND       4   22.2%
EXECUTE         14   77.8%

Limitation:

• Mapping is done in visualization layer
• Would be better if failure_stage in task_metrics.csv used AAT stages from FailureAttributor

✅ FIX 5: Rename "Time gaps" to "Unaccounted Overhead Time"

Status: COMPLETE

Changes Made:

• eval_health_report.py format_report(): Updated section header and explanation
• cli.py: Updated output message

Verified Output:

Unaccounted Overhead Time:
  41 tasks
  Avg Overhead: 26.5%

Overhead includes:
  - Orchestration
  - Queueing
  - Async waits
  - Framework overhead

This is informational only, not an error.

PARTIAL FIXES

⚠️ FIX 3: Decide and Implement Partial-Correct Task Handling

Status: PARTIAL - Field added, needs outcome_status column

What Was Done:

• Added partial_correct_count field to health report
• Health report shows: "Partial-correct Tasks: 1"

What's Missing:

• outcome_status column not added to task_metrics.csv
• No per-task table column showing "correct" / "partial" / "failed"

Recommendation:

• Add outcome_status field to TaskMetrics dataclass
• Compute in eval_v2.py based on final_score vs SUCCESS_THRESHOLD
• Add to per-task table if needed

⚠️ FIX 4: Rename MAS Failure Analysis Labels

Status: PARTIAL - Table exists but shows wrong data

What Was Done:

• Added "MAS Failure Categories" table to render_mas_failure_analysis()
• Added "Outcome Error Types" table
• Both tables are shown

The Problem: The failure_category column in task_metrics.csv contains bucket names (low_recall, wrong_schema), NOT the structured MAS categories (REASONING_FAILURE, DATA_UNDERSTANDING_FAILURE, AGGREGATION_FAILURE).

Current Output (INCORRECT):

MAS Failure Categories          # Should show REASONING_FAILURE, etc.
Category          Count       %
low_recall           11   61.1%  # This is a bucket, not a MAS category!
wrong_schema          4   22.2%
wrong_row_count       3   16.7%

Outcome Error Types             # Correct - shows buckets
Error Type        Count       %
low_recall           11   61.1%
wrong_schema          4   22.2%
wrong_row_count       3   16.7%

Expected Output:

MAS Failure Categories
Category                        Count       %
REASONING_FAILURE                  11   61.1%
DATA_UNDERSTANDING_FAILURE          4   22.2%
AGGREGATION_FAILURE                 3   16.7%

Outcome Error Types
Error Type        Count       %
low_recall           11   61.1%
wrong_schema          4   22.2%
wrong_row_count       3   16.7%

Root Cause:

• eval_replay_artifacts.py FailureAttributor correctly maps to structured categories
• Replay artifacts show correct values (REASONING_FAILURE, etc.)
• BUT task_metrics.csv failure_category column uses old classify_failure_category() which returns buckets
• The viz reads from task_metrics.csv, not replay artifacts

Fix Needed:

1. Update eval_v2.py to call FailureAttributor and use its structured categories
2. OR update viz to read from replay artifacts for failure categories
3. OR add new column mas_failure_category separate from failure_category (bucket)

NOT STARTED

❌ FIX 6: Replay Artifact Consistency

Status: NOT IMPLEMENTED

What's Needed:

• Ensure every task_replay.json includes:
  • harness_health_status
  • run_quality_status
  • outcome_status
  • failure_stage (AAT phases, not legacy)
  • mas_failure_category (structured)
  • outcome_error_type (bucket)
  • root_cause
  • unaccounted_overhead_time_seconds
  • time_accounting_ratio

Current State:

• Replay artifacts have most fields from FailureAttributor
• Missing new fields from health report separation
• No outcome_status field

FILES MODIFIED

1. src/data_agent_baseline/langgraph_agent/eval_health_report.py

   • EngineeringHealthReport dataclass: Added fields
   • format_report(): Separated harness health and run quality
   • _compute_harness_health_score(): New method
   • _generate_recommendations(): Updated signature

2. src/data_agent_baseline/langgraph_agent/eval_v2_viz.py

   • render_mas_failure_analysis(): Added stage inference, restructured tables

3. src/data_agent_baseline/cli.py

   • eval_v2_command(): Updated to show separated statuses
   • Renamed "Time gaps" references

TEST RESULTS

Run: dabench eval-v2 /data3/dataFAIR/kdd-dev/public/artifacts/runs/20260614T073404Z

Successes:

• ✅ Harness health and run quality shown separately
• ✅ Run quality correctly shows DEGRADED (64% accuracy)
• ✅ Failure stage table populated (UNDERSTAND, EXECUTE)
• ✅ "Unaccounted Overhead Time" renamed
• ✅ Per-task table has "Root Cause" column
• ✅ No validator errors (0 errors, 0 warnings)

Issues:

• ⚠️ MAS Failure Categories table shows buckets instead of structured categories
• ⚠️ outcome_status not implemented
• ⚠️ Replay artifacts don't have new health/quality fields

REMAINING WORK

High Priority

1. Fix MAS Failure Categories display (FIX 4 completion)

   • Option A: Update eval_v2.py to use FailureAttributor
   • Option B: Add mas_failure_category column separate from failure_category
   • Option C: Update viz to read from replay artifacts

2. Add outcome_status field (FIX 3 completion)

   • Add to TaskMetrics dataclass
   • Compute: "correct" (score >= SUCCESS_THRESHOLD), "partial" (0.8 <= score < SUCCESS_THRESHOLD), "failed" (score < 0.8)
   • Optionally add to per-task table

Lower Priority

3. Enhance replay artifacts (FIX 6)

   • Add harness_health_status, run_quality_status
   • Add outcome_status
   • Add unaccounted_overhead_time_seconds
   • Add time_accounting_ratio

4. Update documentation

   • EVALUATION.md: Document Phase 2.1 changes
   • Overview.md: Update eval-v2 section

BACKWARD COMPATIBILITY

All changes maintain backward compatibility:

• Old health report readers can still parse the file (new sections at end)
• task_metrics.csv structure unchanged (no columns removed)
• Replay artifacts can be read by old code (new fields are additive)
• CLI output changes are purely presentation (no breaking API changes)

CONCLUSION

Fixes 1, 2, and 5 are COMPLETE and WORKING.

Fixes 3 and 4 are PARTIALLY COMPLETE:

• Infrastructure is in place
• Need data flow fixes (outcome_status computation, MAS category column)

Fix 6 is NOT STARTED but has clear requirements.

The evaluation harness now correctly separates infrastructure health from result quality, making it much clearer when the harness itself is healthy vs. when the MAS is producing poor results.