Spaces:
Running
Running
File size: 9,177 Bytes
d3d0e0e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 | # Phase 0 Command Contract
This document captures baseline command contracts without changing runtime behavior.
## Contract Scope
Commands covered:
1. run-lang-task
2. run-lang-benchmark
3. eval-v2
4. view-task-v2
5. eval-comprehensive
6. tag-failures
7. view-exec-report
## run-lang-task
- Command: `dabench run-lang-task <task_ids...> --config <path> [--display-mode technical|executive]`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `task_ids` required positional list.
2. `--config` required file path.
3. `--display-mode` default `technical`.
- Backend call path:
1. `run_lang_task_command` -> `create_run_output_dir`.
2. Loop task IDs in provided order.
3. Per task call `run_langgraph_single_task`.
4. Optional executive rendering via `generate_executive_report` when display mode is `executive` and `trace.json` exists.
- Artifacts produced:
1. Shared run directory under configured output dir.
2. Per task: `trace.json`, `prediction.csv` (or fallback minimal CSV), task status fields in trace.
- Error and exit semantics:
1. Invalid/duplicate run id from output dir creation surfaces as Typer bad parameter.
2. Per-task failures are represented in artifacts; command still summarizes all selected tasks.
- Compatibility constraints:
1. Tasks share a single run directory.
2. Task order is preserved.
3. Must continue to support both technical and executive modes.
## run-lang-benchmark
- Command: `dabench run-lang-benchmark --config <path> [--limit N]`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `--config` required file path.
2. `--limit` optional positive int.
- Backend call path:
1. `run_lang_benchmark_command` -> dataset task count for progress setup.
2. Delegates to `run_langgraph_benchmark(config, limit, progress_callback)`.
- Artifacts produced:
1. Run directory with per-task task folders.
2. Run-level `summary.json` from langgraph runner.
- Error and exit semantics:
1. Invalid run id/max workers bubbles as command error.
2. Task failures do not abort benchmark loop by default.
- Compatibility constraints:
1. Uses existing benchmark runner implementation.
2. Keeps progress and summary behavior.
## eval-v2
- Command: `dabench eval-v2 <run_id_or_path> [--gold-root path] [--task-root path] [--lambda-penalty float] [--mode standard|verbose|research]`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `run_id` required.
2. `--gold-root` default `/data3/dataFAIR/kdd-dev/public/output`.
3. `--task-root` optional, auto-detected from input_full/input_med/input_hard when omitted.
4. `--lambda-penalty` default `0.1`.
5. `--mode` default `standard`.
- Backend call path:
1. Resolve run path.
2. `evaluate_run_v2`.
3. `write_evaluation_v2` and `write_auditor_validation_report`.
4. Hardening: `validate_evaluation_run`, `generate_health_report`, `generate_replay_artifacts`.
5. Consistency checks: `validate_evaluation_consistency`, `write_validation_report`.
6. Render report via `render_evaluation_report`.
- Artifacts produced at run root:
1. `task_metrics.csv`
2. `trajectory.csv`
3. `tool_calls.csv`
4. `comprehensive_evaluation.csv`
5. `auditor_validation_report.md`
6. `validation_report.md`
7. `artifact_reconciliation_report.txt`
8. `engineering_health_report.txt`
9. Per-task `task_replay.json`
- Error and exit semantics:
1. Missing run path exits non-zero.
2. Validator warnings do not fail command.
3. Validator errors fail command with non-zero exit.
- Compatibility constraints:
1. Keep run-root outputs for compatibility.
2. Keep hardening pipeline invocation order and report generation.
## view-task-v2
- Command: `dabench view-task-v2 <task_id> <run_id_or_path>`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `task_id` required.
2. `run_id` required.
- Backend call path:
1. Resolve run path and task directory.
2. Read `task_metrics.csv` (required).
3. Read optional `trajectory.csv` and `tool_calls.csv`.
4. Render via `render_verbose_task_detail`.
- Artifacts read:
1. Run-root metrics CSVs.
2. Task directory existence check.
- Error and exit semantics:
1. Missing task directory exits non-zero.
2. Missing `task_metrics.csv` exits non-zero with guidance to run eval-v2.
3. Missing row for task exits non-zero.
- Compatibility constraints:
1. Run-root layout remains required.
2. Optional trajectory/tool calls remain best-effort.
## eval-comprehensive
- Command: `dabench eval-comprehensive <run_id_or_path> [--gold-root path] [--lambda-penalty float] [--output-csv path]`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `run_id` required.
2. `--gold-root` default `/data3/dataFAIR/kdd-dev/public/output`.
3. `--lambda-penalty` default `0.1`.
4. `--output-csv` optional; defaults to `<run>/comprehensive_evaluation.csv`.
- Backend call path:
1. Resolve run path.
2. Call `evaluate_run_comprehensive`.
3. Write CSV (drop `action_counts` if present).
- Artifacts produced:
1. `comprehensive_evaluation.csv` or custom output path.
- Error and exit semantics:
1. Missing run path exits non-zero.
2. Empty evaluated task set exits non-zero.
- Compatibility constraints:
1. Preserve current CSV columns and summary rendering behavior.
## tag-failures
- Command: `dabench tag-failures <run_id_or_path> [--gold-root path] [--eval-csv path]`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `run_id` required.
2. `--gold-root` default `/data3/dataFAIR/kdd-dev/public/output`.
3. `--eval-csv` optional; defaults to `<run>/evaluation.csv` if present.
- Backend call path:
1. Resolve run path.
2. Call `tag_run`.
3. Print bucket and phase summaries.
4. Write `failure_tags.csv`.
- Artifacts produced:
1. `failure_tags.csv` at run root.
- Error and exit semantics:
1. Missing run path exits non-zero.
2. No tasks found exits non-zero.
- Compatibility constraints:
1. Preserve existing bucket taxonomy output contract.
## view-exec-report
- Command: `dabench view-exec-report <task_id> <run_id_or_path>`
- Definition: `src/data_agent_baseline/cli.py`
- Arguments and defaults:
1. `task_id` required.
2. `run_id` required.
- Backend call path:
1. Resolve run path.
2. Resolve task directory.
3. Require `trace.json`.
4. Call `generate_executive_report(trace_path, task_dir, console)`.
- Artifacts read:
1. Per-task `trace.json`.
- Error and exit semantics:
1. Missing run, task, or trace exits non-zero.
- Compatibility constraints:
1. Preserve report generation from existing trace artifact.
## Eval-v2 Artifact Schema — Current Production Names
This section documents the **current actual column names** as exported by `eval-v2` at Phase 0 baseline. The schema is frozen for Phase 0. Schema renaming or migration is **deferred and is not part of Phase 0**.
### Source constants
Canonical column lists are defined in `src/data_agent_baseline/langgraph_agent/eval_v2.py` as:
- `TASK_METRICS_COLUMNS`
- `TRAJECTORY_COLUMNS`
- `TOOL_CALLS_COLUMNS`
### task_metrics.csv / comprehensive_evaluation.csv
`comprehensive_evaluation.csv` is written by `write_evaluation_v2` as an alias for `task_metrics.csv` (same data, same columns). Key columns at Phase 0:
- Identity: `run_id`, `task_id`, `trace_id`, `difficulty`, `timestamp`, `task_type`
- Outcome: **`execution_success`**, **`final_score`**, `failure_reason`, `failure_category`, `outcome_status`
- Answer accuracy: `answer_precision`, `answer_recall`, `answer_f1`, `column_precision`, `column_recall`, `column_f1`, `row_precision`, `row_recall`, `row_f1`, `recall`
- Verification: `verification_triggered`, `verification_score`, `aat_verification_triggered`, `aat_verification_score`
- Agent behaviour: `replan_count`, `retry_count`, `total_replans`, `autonomy_score`, `tool_efficiency`, `data_understanding_score`
- Cost: `total_tokens`, `llm_calls`, `execution_time`
- Full column list: see `TASK_METRICS_COLUMNS` in `eval_v2.py` (238 columns at Phase 0 baseline).
The names `execution_success` and `final_score` are the current production names. **No renaming is performed in Phase 0.**
### trajectory.csv
Columns from `TRAJECTORY_COLUMNS` (29 columns at Phase 0 baseline):
`run_id`, `task_id`, `trace_id`, `step_id`, `parent_step_id`, `stage`, `legacy_stage`, `phase`, `agent`, `trajectory_agent`, `agent_type`, `tool`, `action`, `coordinator_checkpoint`, `confidence`, `decision`, `review_type`, `verification_status`, `specialist_selected`, `success`, `duration_seconds`, `tokens`, `retries`, `timestamp`, `elapsed_seconds`, `thought`, `action_input`, `observation`, `raw_response`, `metadata_json`
### tool_calls.csv
Columns from `TOOL_CALLS_COLUMNS` (17 columns at Phase 0 baseline):
`run_id`, `task_id`, `trace_id`, `step_id`, `call_id`, `tool_name`, `tool_category`, `success`, `latency_seconds`, `tokens`, `retry_count`, `input_size`, `output_size`, `error_type`, `error_message`, `timestamp`, `metadata_json`
### Deferred work
- Column renaming (e.g., `execution_success` → `task_success`, `final_score` → `score`) is explicitly deferred.
- No schema migration, column aliasing, or DataFrame reshaping is performed in Phase 0.
- Tests reference only the current names.
|