Spaces:
Sleeping
Sleeping
File size: 4,906 Bytes
116524e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 | # ACE Benchmarks
Evaluate ACE performance with scientific rigor using our comprehensive benchmark suite.
This evaluation framework tests Agentic Context Engineering (ACE) across multiple datasets with automatic metrics, train/test splits, and overfitting analysis to ensure honest performance measurements.
## Quick Start
```bash
# List available benchmarks
uv run python scripts/run_benchmark.py list
# Run ACE evaluation with train/test split (default)
uv run python scripts/run_benchmark.py finer_ord --limit 100
# Run baseline only (no ACE learning)
uv run python scripts/run_benchmark.py simple_qa --limit 50 --skip-adaptation
# Compare baseline vs ACE side-by-side
uv run python scripts/run_benchmark.py hellaswag --limit 50 --compare
```
## Available Benchmarks
| Benchmark | Description | Domain | Default Limit |
|-----------|-------------|---------|---------------|
| **finer_ord** | Financial Named Entity Recognition | Finance | 100 |
| **simple_qa** | Question Answering (SQuAD) | General | 200 |
| **simple_math** | Math Word Problems (GSM8K) | Mathematics | 100 |
| **mmlu** | Massive Multitask Language Understanding | General Knowledge | 500 |
| **hellaswag** | Commonsense Reasoning | Common Sense | 200 |
| **arc_easy** | AI2 Reasoning Challenge (Easy) | Reasoning | 200 |
| **arc_challenge** | AI2 Reasoning Challenge (Hard) | Reasoning | 200 |
## Command Options
```bash
uv run python scripts/run_benchmark.py <benchmark> [options]
```
**Key Options:**
- `--limit` - Override sample limit (always overrides config)
- `--model` - Model name (default: gpt-4o-mini)
- `--skip-adaptation` - Skip ACE learning (faster baseline)
- `--compare` - Run both baseline and ACE, then compare results
- `--epochs` - ACE adaptation epochs (default: 1)
- `--split-ratio` - Train/test split ratio (default: 0.8)
- `--online-mode` - Use continuous learning instead of offline
- `--prompt-version` - Use v1 or v2 prompts (default: v1)
- `--save-detailed` - Save per-sample results
- `--quiet` - Suppress progress output
## Examples
```bash
# Quick test with 10 samples
uv run python scripts/run_benchmark.py finer_ord --limit 10 --quiet
# Compare baseline vs ACE
uv run python scripts/run_benchmark.py simple_qa --limit 50 --compare
# Full ACE evaluation with v2 prompts
uv run python scripts/run_benchmark.py simple_qa --epochs 3 --prompt-version v2 --save-detailed
# Online learning mode
uv run python scripts/run_benchmark.py hellaswag --limit 100 --online-mode
# Custom train/test split (90/10)
uv run python scripts/run_benchmark.py mmlu --limit 100 --split-ratio 0.9
# Test all benchmarks quickly (baseline only)
for benchmark in finer_ord simple_qa hellaswag arc_easy; do
uv run python scripts/run_benchmark.py $benchmark --limit 5 --skip-adaptation --quiet
done
```
## Output
Results saved to `benchmark_results/` with format:
- **Summary**: `{benchmark}_{model}_{timestamp}_summary.json`
- **Detailed**: `{benchmark}_{model}_{timestamp}_detailed.json` (if `--save-detailed`)
## Adding Custom Benchmarks
Create `benchmarks/tasks/my_benchmark.yaml`:
```yaml
task: my_benchmark
version: "1.0"
data:
source: huggingface
dataset_path: my/dataset
split: test
limit: 100
metrics:
- name: exact_match
weight: 1.0
metadata:
description: "My custom benchmark"
domain: "my_domain"
```
## Evaluation Modes
The benchmark script supports three evaluation modes:
1. **ACE Mode (default)**: Train/test split with learning
```bash
uv run python scripts/run_benchmark.py simple_qa --limit 100
```
2. **Baseline Mode**: No learning, direct evaluation
```bash
uv run python scripts/run_benchmark.py simple_qa --limit 100 --skip-adaptation
```
3. **Comparison Mode**: Runs both baseline and ACE, shows improvement
```bash
uv run python scripts/run_benchmark.py simple_qa --limit 100 --compare
```
## Key Features
- **Overfitting Prevention**: Automatic 80/20 train/test splits ensure true generalization metrics
- **Scientific Rigor**: Comprehensive evaluation modes with honest performance analysis
- **Multiple Domains**: Finance, general knowledge, reasoning, math, and common sense benchmarks
- **Flexible Configuration**: Customizable limits, models, and evaluation parameters
- **Performance Tracking**: Detailed results with per-sample analysis options
## Notes
- **Default 80/20 train/test split** prevents overfitting and shows true generalization
- The `--limit` parameter always overrides config file limits
- ACE adaptation improves performance through iterative learning
- Use `--compare` to see baseline vs ACE improvement side-by-side
- Overfitting warnings help identify when ACE memorizes vs generalizes
- Opik tracing warnings ("Failed to log adaptation metrics") are harmless |