| # AgentEval |
|
|
| <p align="center"> |
| <strong>Measure what agents actually do — not just what they say.</strong> |
| </p> |
|
|
| <p align="center"> |
| <img src="https://img.shields.io/badge/Agent-Evaluation-2563EB?style=for-the-badge" alt="Agent Evaluation"> |
| <img src="https://img.shields.io/badge/Task-Success-14B8A6?style=for-the-badge" alt="Task Success"> |
| <img src="https://img.shields.io/badge/Tool-Use-7C3AED?style=for-the-badge" alt="Tool Use"> |
| <img src="https://img.shields.io/badge/Reliability-F59E0B?style=for-the-badge" alt="Reliability"> |
| </p> |
|
|
| --- |
|
|
| ## Evaluation for AI systems that act |
|
|
| **AgentEval** is an independent Hugging Face organization focused on evaluating AI agents as complete systems. |
|
|
| A strong agent should do more than produce a good-looking answer. |
|
|
| It should: |
|
|
| - understand the task |
| - choose the right tools |
| - use them correctly |
| - recover from failure |
| - stay within constraints |
| - complete the task |
| - do so efficiently |
| - produce a verifiable result |
|
|
| That requires a different evaluation mindset. |
|
|
| > **Outcome first. Trace second. Model score third.** |
|
|
| --- |
|
|
| # What should an agent evaluation measure? |
|
|
| A useful agent evaluation can include several layers: |
|
|
| ```text |
| TASK |
| ↓ |
| PLAN |
| ↓ |
| TOOL CHOICE |
| ↓ |
| TOOL EXECUTION |
| ↓ |
| RECOVERY |
| ↓ |
| FINAL RESULT |
| ↓ |
| VERIFICATION |
| ``` |
|
|
| AgentEval focuses on the full path. |
|
|
| --- |
|
|
| ## 01 · Task Success |
|
|
| The most important question: |
|
|
| > **Did the agent complete the task?** |
|
|
| Possible metrics: |
|
|
| - success / failure |
| - partial completion |
| - goal coverage |
| - final answer correctness |
| - constraint compliance |
| - completion consistency across repeated runs |
|
|
| A fluent answer is not enough if the task was not completed. |
|
|
| --- |
|
|
| ## 02 · Tool Use |
|
|
| Agents increasingly depend on external tools. |
|
|
| Evaluation can inspect: |
|
|
| - correct tool selection |
| - valid arguments |
| - unnecessary tool calls |
| - failed tool calls |
| - retries |
| - fallback behavior |
| - sequence of tool usage |
| - tool-result interpretation |
|
|
| --- |
|
|
| ## 03 · Traces |
|
|
| Agent behavior becomes easier to understand when the execution trace is visible. |
|
|
| Useful trace elements may include: |
|
|
| - steps |
| - tool calls |
| - model calls |
| - errors |
| - retries |
| - timestamps |
| - latency |
| - token usage |
| - cost |
| - intermediate state |
| - final result |
|
|
| A final answer can hide a bad process. |
|
|
| A trace makes the process inspectable. |
|
|
| --- |
|
|
| ## 04 · Reliability |
|
|
| One successful run does not prove reliability. |
|
|
| AgentEval is interested in repeated execution. |
|
|
| Possible metrics: |
|
|
| - pass rate |
| - variance across runs |
| - retry rate |
| - failure mode frequency |
| - deterministic vs. unstable behavior |
| - recovery success |
| - consistency under perturbation |
|
|
| --- |
|
|
| ## 05 · Efficiency |
|
|
| An agent can complete a task and still be inefficient. |
|
|
| Possible efficiency metrics: |
|
|
| - number of steps |
| - tool calls |
| - model calls |
| - latency |
| - token usage |
| - estimated cost |
| - redundant actions |
| - unnecessary retries |
|
|
| The best agent is not always the one with the highest raw capability. |
|
|
| Sometimes it is the one that completes the task with fewer resources. |
|
|
| --- |
|
|
| ## 06 · Safety & Constraint Following |
|
|
| Agent evaluations may also inspect whether a system stays within defined boundaries. |
|
|
| Examples: |
|
|
| - allowed tools only |
| - no unauthorized actions |
| - no secret leakage |
| - no unsafe command execution |
| - approval required before sensitive actions |
| - adherence to role or workflow constraints |
|
|
| --- |
|
|
| # Possible Spaces |
|
|
| ### Agent Task Evaluator |
| Compare expected outcomes with an agent's final result and trace. |
|
|
| ### Tool-Call Grader |
| Check whether an agent selected the correct tool and valid parameters. |
|
|
| ### Trace Inspector |
| Upload an agent trace and inspect steps, retries, latency, failures, and cost. |
|
|
| ### Reliability Lab |
| Run repeated synthetic evaluations and compare consistency. |
|
|
| ### Agent Efficiency Score |
| Measure task completion against steps, token usage, tool calls, and runtime. |
|
|
| ### Failure Mode Explorer |
| Analyze why an agent failed and classify recurring failure patterns. |
|
|
| ### Multi-Agent Evaluator |
| Compare collaboration quality between planner, executor, reviewer, and verifier roles. |
|
|
| ### Agent Benchmark Builder |
| Create custom task sets with expected outputs and evaluation criteria. |
|
|
| ### Tool-Use Benchmark |
| Score agents on structured tool-selection and execution tasks. |
|
|
| ### Agent Regression Check |
| Compare two agent versions on the same evaluation set. |
|
|
| --- |
|
|
| # Possible Datasets |
|
|
| AgentEval can host datasets designed specifically for evaluation. |
|
|
| Examples: |
|
|
| ```text |
| agent-eval-tasks |
| tool-use-evals |
| agent-failure-cases |
| agent-traces |
| multi-agent-tasks |
| recovery-scenarios |
| constraint-following-tests |
| agent-regression-suite |
| ``` |
|
|
| A useful evaluation dataset may include: |
|
|
| - task |
| - environment |
| - available tools |
| - expected result |
| - allowed actions |
| - prohibited actions |
| - success criteria |
| - reference trace |
| - scoring rubric |
|
|
| --- |
|
|
| # Possible Models |
|
|
| Models may support evaluation tasks such as: |
|
|
| - trace classification |
| - failure-mode detection |
| - tool-call validation |
| - task-success prediction |
| - reward modeling |
| - output grading |
| - execution-quality scoring |
| - anomaly detection in agent traces |
|
|
| --- |
|
|
| # A simple evaluation record |
|
|
| ```json |
| { |
| "task": "Find the cheapest valid route", |
| "success": true, |
| "tool_calls": 4, |
| "failed_tool_calls": 1, |
| "steps": 7, |
| "latency_seconds": 12.4, |
| "estimated_cost": 0.031, |
| "constraint_violations": 0 |
| } |
| ``` |
|
|
| One record is useful. |
|
|
| Hundreds of repeated records become a benchmark. |
|
|
| --- |
|
|
| # Core Evaluation Dimensions |
|
|
| | Dimension | Core Question | |
| |---|---| |
| | **Success** | Did the agent complete the task? | |
| | **Correctness** | Was the result right? | |
| | **Tool Use** | Were the right tools used correctly? | |
| | **Reliability** | Does it work repeatedly? | |
| | **Efficiency** | How much time, cost and work did it require? | |
| | **Recovery** | Can it recover from failure? | |
| | **Safety** | Did it respect constraints? | |
| | **Trace Quality** | Is the execution understandable and auditable? | |
|
|
| --- |
|
|
| # AgentEval Scorecard |
|
|
| A practical evaluation may combine multiple signals: |
|
|
| ```text |
| Task Success 40% |
| Tool Accuracy 20% |
| Reliability 15% |
| Efficiency 10% |
| Recovery 10% |
| Constraint Safety 5% |
| ``` |
|
|
| The exact weighting depends on the use case. |
|
|
| AgentEval does not promote one universal score. |
|
|
| Different agents require different evaluation criteria. |
|
|
| --- |
|
|
| # Why agent evaluation is different |
|
|
| Traditional model evaluation often asks: |
|
|
| > How good is the answer? |
|
|
| Agent evaluation may need to ask: |
|
|
| > Did the system complete the task correctly, efficiently, safely, and repeatably? |
|
|
| That distinction matters. |
|
|
| An agent is a system. |
|
|
| Its quality depends on more than one model response. |
|
|
| --- |
|
|
| # Principles |
|
|
| ### Measure outcomes |
| A convincing explanation does not equal successful execution. |
|
|
| ### Inspect traces |
| Execution history matters. |
|
|
| ### Repeat evaluations |
| One run is not enough. |
|
|
| ### Separate capability from reliability |
| An agent may be capable but unstable. |
|
|
| ### Include cost and latency |
| Operational quality matters. |
|
|
| ### Evaluate tool behavior |
| Tool misuse can invalidate an otherwise correct result. |
|
|
| ### Make failure visible |
| Benchmarks should help explain failure, not hide it. |
|
|
| --- |
|
|
| # Technology Directions |
|
|
| Projects may use: |
|
|
| - Hugging Face Spaces |
| - Hugging Face Datasets |
| - structured evaluation sets |
| - agent traces |
| - tool-calling logs |
| - JSON evaluation records |
| - Python |
| - JavaScript |
| - benchmark harnesses |
| - scoring pipelines |
| - LLM-as-a-judge experiments |
| - deterministic validators |
| - regression testing |
| - observability data |
|
|
| --- |
|
|
| # Who is AgentEval for? |
|
|
| AgentEval may be useful for: |
|
|
| - agent developers |
| - AI engineers |
| - eval teams |
| - platform engineers |
| - researchers |
| - QA teams |
| - MLOps teams |
| - startups |
| - enterprise AI teams |
| - tool developers |
| - open-source contributors |
|
|
| --- |
|
|
| # Important Note |
|
|
| Projects published here are primarily intended for: |
|
|
| - research |
| - development |
| - benchmarking |
| - education |
| - prototyping |
| - technical experimentation |
|
|
| Evaluation scores are not universal guarantees of: |
|
|
| - reliability |
| - safety |
| - production readiness |
| - correctness |
| - regulatory compliance |
| - suitability for high-impact use |
|
|
| Agent systems should be evaluated in the environment and context in which they are actually used. |
|
|
| --- |
|
|
| # Independent Organization |
|
|
| **AgentEval is an independent Hugging Face community organization.** |
|
|
| It is not an official benchmark authority, certification body, standards organization, model provider, or Hugging Face organization. |
|
|
| The name **AgentEval** reflects the technical focus: |
|
|
| > **evaluation for AI systems that plan, act, use tools, and complete tasks.** |
|
|
| --- |
|
|
| <p align="center"> |
|
|
| # AgentEval |
|
|
| ### **Evaluate actions. Measure outcomes. Improve agents.** |
|
|
| </p> |
|
|
