File size: 2,130 Bytes
fc6b03c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | # Quickstart
## 1. Install
```bash
git clone https://github.com/karnamshiva/agent-eval-harness
cd agent-eval-harness
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
```
## 2. Get a free LLM API key
The examples default to **Groq** (free tier, 30 req/min on Llama 3.1 70B).
1. Sign up at [console.groq.com](https://console.groq.com/)
2. Create an API key
3. Export it: `export GROQ_API_KEY=gsk_...`
Other supported providers (set the relevant env var):
| Provider | Env var | Free tier |
|----------|---------|-----------|
| Groq | `GROQ_API_KEY` | Yes |
| Google Gemini | `GEMINI_API_KEY` | Yes (1500 req/day) |
| OpenAI | `OPENAI_API_KEY` | No (paid only) |
| Anthropic | `ANTHROPIC_API_KEY` | $5 signup credit |
| Ollama (local) | — | 100% free, local |
## 3. Run your first eval
```bash
python examples/run_eval.py
```
Expected output (times vary):
```
n=15
accuracy=0.933
pass@0.75=0.933
latency p50=420ms p95=850ms
total cost=$0.0012
error rate=0.000
Serialized run to runs/example.json
```
## 4. Try the Gradio UI
```bash
python app.py
```
Open http://localhost:7860 in your browser.
## 5. Ship your own dataset
Create `datasets/my_cases.jsonl` (one JSON per line):
```json
{"input": "What does our product do?", "expected": "It is an eval harness for LLM agents.", "tags": ["product"], "lang": "en"}
{"input": "¿Qué hace nuestro producto?", "expected": "Es un arnés de evaluación para agentes LLM.", "tags": ["product"], "lang": "es"}
```
Then:
```python
from src import EvalHarness, LLMJudge
harness = EvalHarness(
model="groq/llama-3.1-70b-versatile",
dataset="datasets/my_cases.jsonl",
judge=LLMJudge(model="groq/llama-3.1-8b-instant"),
)
result = harness.run()
print(result.summary())
result.save("runs/my-first-real-run.json")
```
## 6. Set up a regression gate in CI
Commit a baseline run once (`runs/baseline.json`). Then in CI:
```bash
python examples/regression_gate.py \
--baseline runs/baseline.json \
--candidate runs/latest.json
```
The script exits non-zero if any tolerance is violated — perfect for a GitHub Actions check.
|