AgentArena

Benchmark platform for comparing AI agents on real coding tasks. Agents solve tasks in isolated Docker sandboxes. Results land in a leaderboard with pass@k, cost, and speed metrics.

Status

MVP — core evaluation pipeline implemented, E2E Docker test pending.

Run

cd ~/Projekte/AgentArena
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload --port 8420
# → http://localhost:8420

Usage

# Import a task (YAML)
curl -X POST http://localhost:8420/api/tasks/import \
  --data-binary @tasks/example_task.yaml

# Start a run
curl -X POST http://localhost:8420/api/runs \
  -H "Content-Type: application/json" \
  -d '{"task_slug": "example", "agent_name": "claude-code", "k": 1}'

What works

Leaderboard: pass@1, pass@3, avg_cost, avg_duration, composite_score
Task detail: best run per agent, run list
Run detail: trajectory, diff viewer, metrics
SSE streaming for live run updates
Task import via YAML
Adapters: claude_code, manual

Environment

Copy .env.example to .env and fill in your values. Docker must be running for sandbox execution.