Spaces:
Sleeping
Sleeping
| # Baseline ReAct Agent Architecture Summary | |
| ## Execution Flow | |
| ### 1. CLI Entry Points | |
| **Located in:** `src/data_agent_baseline/cli.py` | |
| Two main commands: | |
| - `dabench run-task <task_id>` - Execute single task | |
| - `dabench run-benchmark` - Execute multiple tasks (with parallelization) | |
| ### 2. Runner Layer | |
| **Located in:** `src/data_agent_baseline/run/runner.py` | |
| **Key functions:** | |
| - `run_single_task()` - Orchestrates single task execution | |
| - `run_benchmark()` - Orchestrates benchmark execution with optional parallelization | |
| - `_run_single_task_with_timeout()` - Subprocess-based timeout handling | |
| - `_write_task_outputs()` - Generates artifacts | |
| **Execution flow:** | |
| 1. Load task from DABenchPublicDataset | |
| 2. Initialize ReActAgent with model adapter and tool registry | |
| 3. Run agent (with optional timeout in subprocess) | |
| 4. Write outputs to disk | |
| ### 3. Agent Layer | |
| **Located in:** `src/data_agent_baseline/agents/react.py` | |
| **ReActAgent class:** | |
| - Implements classic ReAct loop (Thought → Action → Observation) | |
| - Maximum steps configurable (default 16) | |
| - Uses ModelAdapter for LLM calls | |
| - Uses ToolRegistry for tool execution | |
| **Core method:** `run(task: PublicTask) -> AgentRunResult` | |
| **Loop structure:** | |
| ``` | |
| for step in range(max_steps): | |
| 1. Build message history (system + task + previous steps) | |
| 2. Call LLM to get next step | |
| 3. Parse JSON response (thought, action, action_input) | |
| 4. Execute tool via ToolRegistry | |
| 5. Record observation | |
| 6. Check if terminal (answer submitted) | |
| 7. Break if answer submitted | |
| ``` | |
| ### 4. Runtime State Management | |
| **Located in:** `src/data_agent_baseline/agents/runtime.py` | |
| **Data structures:** | |
| - `StepRecord` - Single step (thought, action, action_input, observation, ok) | |
| - `AgentRuntimeState` - Mutable state during execution | |
| - `AgentRunResult` - Final immutable result | |
| ### 5. Artifact Generation | |
| **Outputs per task:** | |
| ``` | |
| <run_output_dir>/<task_id>/ | |
| ├── trace.json # Full execution trace | |
| └── prediction.csv # Final answer (if submitted) | |
| ``` | |
| ## Baseline Trace Structure | |
| ### Baseline trace.json Format | |
| ```json | |
| { | |
| "task_id": "task_22", | |
| "answer": { | |
| "columns": ["col1", "col2"], | |
| "rows": [["val1", "val2"], ...] | |
| }, | |
| "steps": [ | |
| { | |
| "step_index": 1, | |
| "thought": "I need to...", | |
| "action": "read_csv", | |
| "action_input": {"path": "...", "max_rows": 10}, | |
| "raw_response": "```json\n{...}\n```", | |
| "observation": { | |
| "ok": true, | |
| "tool": "read_csv", | |
| "content": {...} | |
| }, | |
| "ok": true | |
| } | |
| ], | |
| "failure_reason": null, | |
| "succeeded": true, | |
| "e2e_elapsed_seconds": 8.528 | |
| } | |
| ``` | |
| **Key characteristics:** | |
| - **Simple structure:** No phases, no per-step timing | |
| - **No metadata:** Missing difficulty, question text, context info | |
| - **Minimal metrics:** Only e2e_elapsed_seconds, no token counts | |
| - **No recovery tracking:** No retry/replan information | |
| - **No confidence scores:** No self-assessment | |
| ### Current Evaluation Harness Expectations | |
| **Located in:** `src/data_agent_baseline/langgraph_agent/eval_v2.py` | |
| The newer LangGraph agent produces traces with extensive metadata: | |
| ```json | |
| { | |
| "task_id": "task_22", | |
| "difficulty": "easy", | |
| "trace_id": "/path/to/trace", | |
| "answer": {...}, | |
| "steps": [ | |
| { | |
| "step_index": 1, | |
| "thought": "...", | |
| "action": "read_csv", | |
| "action_input": {...}, | |
| "raw_response": "...", | |
| "observation": {...}, | |
| "ok": true, | |
| "phase": "explore", // ← Baseline missing | |
| "duration_seconds": 0.01, // ← Baseline missing | |
| "token_usage": { // ← Baseline missing | |
| "prompt_tokens": 100, | |
| "completion_tokens": 50, | |
| "total_tokens": 150 | |
| } | |
| } | |
| ], | |
| "comprehensive_metrics": { // ← Baseline missing | |
| "execution_success": 1, | |
| "execution_time": 8.5, | |
| "tool_calls": 5, | |
| "tool_failures": 0, | |
| "llm_calls": 5, | |
| "total_tokens": 27000, | |
| "stage_metrics": {...}, | |
| "action_counts": {...}, | |
| "per_action_metrics": {...}, | |
| "confidence_score": 0.9, | |
| ... | |
| }, | |
| "failure_reason": null, | |
| "succeeded": true, | |
| "e2e_elapsed_seconds": 8.528 | |
| } | |
| ``` | |
| ## Gap Analysis | |
| ### Missing in Baseline Traces | |
| | Feature | Baseline | Evaluation Harness | Impact | | |
| |---------|----------|-------------------|--------| | |
| | **Phase tracking** | ❌ | ✅ (explore, planner, execute, critic) | Cannot compute phase-specific metrics | | |
| | **Per-step timing** | ❌ | ✅ duration_seconds | Cannot compute per-phase time | | |
| | **Token usage** | ❌ | ✅ Per-step and total | Cannot compute cost metrics | | |
| | **Comprehensive metrics** | ❌ | ✅ Full metadata object | Must compute from steps | | |
| | **Stage metrics** | ❌ | ✅ Per-stage breakdown | Cannot compute stage efficiency | | |
| | **Action counts** | ❌ | ✅ Tool usage frequency | Must derive from steps | | |
| | **Confidence scores** | ❌ | ✅ Self-assessment | Cannot evaluate calibration | | |
| | **Recovery tracking** | ❌ | ✅ Replan/retry counts | Cannot measure autonomy | | |
| | **Task metadata** | Partial | ✅ difficulty, question | Need to fetch from task.json | | |
| | **Trace ID** | ❌ | ✅ Full path | Must construct | | |
| ### Available in Baseline Traces | |
| ✅ **Core execution data:** | |
| - task_id | |
| - answer (columns, rows) | |
| - steps (thought, action, action_input, observation, ok) | |
| - succeeded flag | |
| - failure_reason (if any) | |
| - e2e_elapsed_seconds | |
| ✅ **Derivable metrics:** | |
| - Tool call counts (from steps) | |
| - Tool failure counts (from step.ok) | |
| - Tool diversity (from unique actions) | |
| - Trajectory length (from len(steps)) | |
| - Tool efficiency (failures / total calls) | |
| - Action counts (frequency analysis) | |
| ## Evaluation Strategy | |
| ### Phase 1: Adapter-Based Approach | |
| **Do NOT modify baseline execution code.** | |
| Instead: | |
| 1. Create adapter layer to normalize baseline traces | |
| 2. Infer missing metadata where possible | |
| 3. Compute metrics from available data | |
| 4. Mark unavailable metrics as NULL | |
| 5. Generate comparable evaluation reports | |
| ### Adapter Responsibilities | |
| ```python | |
| class BaselineTraceAdapter: | |
| def load_trace(trace_path: Path) -> dict | |
| def enrich_with_task_metadata(trace: dict, task_root: Path) -> dict | |
| def compute_derivable_metrics(trace: dict) -> dict | |
| def normalize_to_canonical_schema(trace: dict) -> CanonicalTrace | |
| def export_for_evaluation(trace: CanonicalTrace) -> dict | |
| ``` | |
| ### Metrics Coverage | |
| **Phase 1 Metrics (Baseline-Compatible):** | |
| ✅ **Accuracy metrics:** | |
| - Overall accuracy (from gold comparison) | |
| - Per-difficulty accuracy | |
| - Answer precision/recall/F1 | |
| ✅ **Efficiency metrics:** | |
| - Average steps per task | |
| - Average runtime | |
| - Tool calls per task | |
| ✅ **Reliability metrics:** | |
| - Success rate | |
| - Tool error rate | |
| - Timeout rate (if configured) | |
| ✅ **Tool usage metrics:** | |
| - Tool diversity | |
| - Tool efficiency | |
| - Most common tools | |
| ❌ **Not available for baseline:** | |
| - Token costs (no token tracking) | |
| - Phase-specific timing (no phase labels) | |
| - Confidence calibration (no confidence scores) | |
| - Recovery metrics (no replan tracking) | |
| - Stage efficiency (no stage breakdown) | |
| ### Future Compatibility | |
| Design ensures: | |
| - Baseline and LangGraph traces can coexist | |
| - Same evaluation pipeline for both | |
| - Clear indication of unavailable metrics | |
| - Easy addition of new agent types | |
| - Scientific comparison across systems | |