Spaces:
Sleeping
Sleeping
File size: 7,606 Bytes
116524e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 | # Opik Observability
ACE integrates with [Opik](https://github.com/comet-ml/opik) for tracing, cost tracking, and performance monitoring. All Opik tracing is **explicit opt-in** — it is never auto-enabled just because the package is installed.
Two independent tracing modes:
1. **Pipeline step** (`OpikStep`) — client-agnostic, logs one Opik trace per sample with ACE context fields.
2. **LiteLLM callback** (`register_opik_litellm_callback`) — LiteLLM-specific, tracks per-LLM-call tokens and costs.
## Installation
```bash
uv add ace-framework[observability]
```
## Quick Start
```python
from ace import ACELiteLLM
# Easiest: ACELiteLLM enables both tracing modes with one flag
ace = ACELiteLLM.from_model("gpt-4o-mini", opik=True, opik_project="my-experiment")
```
```python
from ace import (
ACE, OpikStep,
Agent, Reflector, SkillManager,
SimpleEnvironment,
)
# Manual: Add OpikStep via extra_steps
runner = ACE.from_roles(
agent=Agent("gpt-4o-mini"),
reflector=Reflector("gpt-4o-mini"),
skill_manager=SkillManager("gpt-4o-mini"),
environment=SimpleEnvironment(),
extra_steps=[OpikStep(project_name="my-experiment")],
)
```
```python
# LLM-level cost tracking only (no pipeline traces)
from ace import register_opik_litellm_callback
registered = register_opik_litellm_callback(project_name="my-experiment")
```
## Starting the Opik Server
=== "Local (Docker)"
```bash
docker run -d -p 5173:5173 --name opik ghcr.io/comet-ml/opik:latest
# View traces at http://localhost:5173
```
=== "Comet Cloud"
```bash
export COMET_API_KEY="your-api-key"
# Traces appear at https://www.comet.com/opik
```
## OpikStep
`OpikStep` is a terminal side-effect step that logs one Opik trace per sample. It reads context fields but never mutates them — safe to append to any pipeline.
### Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `project_name` | `str` | `"ace-framework"` | Opik project for organizing traces |
| `tags` | `list[str]` | `None` | Extra tags attached to every trace |
### What Gets Logged
Each trace includes:
| Field | Source |
|-------|--------|
| **Input** | Question and context from the sample |
| **Output** | Answer, reasoning, and skill IDs from `AgentOutput` |
| **Metadata** | Epoch, step index, skill count, reflection insights, operation counts |
| **Feedback scores** | Accuracy extracted from environment feedback (correct / incorrect) |
### Trace Hierarchy
```mermaid
graph TD
P["Project: my-experiment"]
P --> T["Trace: sample_run_001"]
T --> I["Input: question + context"]
T --> O["Output: answer + reasoning + skill_ids"]
T --> M["Metadata: epoch=2, skills=12, ops=3"]
T --> F["Feedback: accuracy=1.0"]
T --> L["LLM Calls (automatic)"]
L --> L1["agent_generate — 450 tokens, $0.0003"]
L --> L2["reflector_reflect — 620 tokens, $0.0004"]
L --> L3["skill_manager_update — 380 tokens, $0.0002"]
```
## LLM Cost Tracking
`OpikStep` does **not** register the LiteLLM callback — the two tracing modes are independent. To get per-LLM-call cost tracking, call `register_opik_litellm_callback()` separately:
```python
from ace import register_opik_litellm_callback
success = register_opik_litellm_callback(project_name="cost-tracking")
# Returns True if registered, False if Opik unavailable
```
Every LLM call is then automatically tracked with:
- Input / output tokens
- Model used
- Cost per call
- Latency
When using `ACELiteLLM` with `opik=True`, both modes are enabled together automatically — no need to call `register_opik_litellm_callback()` manually.
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `OPIK_PROJECT_NAME` | Project name for organizing traces | `ace-framework` |
| `OPIK_DISABLED=true` | Disable all Opik tracing | Not set |
| `OPIK_ENABLED=false` | Alternative way to disable tracing | Not set |
| `OPIK_URL_OVERRIDE` | Custom Opik server URL | `http://localhost:5173/api` |
| `OPIK_WORKSPACE` | Opik workspace name | `default` |
## Error Handling
When using `ACELiteLLM` with `opik=True`, errors are **raised immediately**:
- `ImportError` if the `opik` package is not installed
- `RuntimeError` if the Opik client fails to initialize (bad config, disabled via env vars)
This ensures you know immediately if tracing is broken, rather than discovering missing traces later.
When using `OpikStep` directly via `extra_steps`, it soft-imports Opik and silently becomes a no-op if the package is absent — useful for pipelines that should work with or without observability.
```python
from ace import OPIK_AVAILABLE
if OPIK_AVAILABLE:
print("Opik tracing is available")
```
## Troubleshooting: `~/.opik.config`
The Opik SDK stores a global config file at `~/.opik.config` (created by `opik.configure()`). This file **overrides environment variables** and can cause silent failures if it contains stale settings.
If traces aren't appearing, check:
```bash
cat ~/.opik.config
```
A correct config for Comet Cloud looks like:
```ini
[opik]
url_override = https://www.comet.com/opik/api/
workspace = your-workspace-name
```
Common issues:
- **Wrong URL**: `https://www.comet.com/api/` (missing `/opik/`) causes 404 errors
- **Wrong workspace**: `workspace = default` instead of your actual workspace name
- **Stale config**: Re-run `opik.configure()` or edit the file directly to fix
## Disabling Tracing
```bash
# In CI or tests
OPIK_DISABLED=true pytest tests/
# Or via the alternative variable
OPIK_ENABLED=false python my_script.py
```
## Full Example
=== "ACELiteLLM (easiest)"
```python
from ace import ACELiteLLM, Sample, SimpleEnvironment
ace = ACELiteLLM.from_model("gpt-4o-mini", opik=True, opik_project="ace-training")
samples = [
Sample(question="What is 2+2?", context="", ground_truth="4"),
Sample(question="Capital of France?", context="", ground_truth="Paris"),
]
results = ace.learn(samples, environment=SimpleEnvironment(), epochs=3)
ace.save("trained.json")
# View traces at http://localhost:5173 → project "ace-training"
```
=== "ACE runner (manual)"
```python
from ace import (
ACE, Agent, Reflector, SkillManager, Skillbook,
SimpleEnvironment, Sample, OpikStep,
register_opik_litellm_callback,
)
runner = ACE.from_roles(
agent=Agent("gpt-4o-mini"),
reflector=Reflector("gpt-4o-mini"),
skill_manager=SkillManager("gpt-4o-mini"),
environment=SimpleEnvironment(),
extra_steps=[OpikStep(project_name="ace-training")],
)
# Optionally add LLM-level cost tracking
register_opik_litellm_callback(project_name="ace-training")
samples = [
Sample(question="What is 2+2?", context="", ground_truth="4"),
Sample(question="Capital of France?", context="", ground_truth="Paris"),
]
results = runner.run(samples, epochs=3)
runner.save("trained.json")
```
## What to Read Next
- [Integration Pattern](../guides/integration.md) — how runners compose pipeline steps
- [Full Pipeline Guide](../guides/full-pipeline.md) — building pipelines from scratch
- [Async Learning](../guides/async-learning.md) — background learning with cost monitoring
|