| # Intent-Routing Eval (E3) |
|
|
| Scores the live 6-intent router (`OrchestratorAgent.classify`) against a golden |
| dataset of labelled messages. Run it before any deploy that touches the router |
| prompt (`src/config/prompts/intent_router.md`) or its few-shots. |
|
|
| ## Files |
|
|
| | File | What | |
| |---|---| |
| | `intent_dataset.json` | **Golden dataset** — `message` + known-correct `expected_intent` per case. The source of truth scoring compares against. | |
| | `run_eval.py` | Runner — calls the router per case, scores correctness, records latency + tokens. | |
| | `results/` | Timestamped run reports, one JSON per run (never overwritten). | |
|
|
| ## Run |
|
|
| Run as a module (`-m`), not the file path — module mode puts the repo root on |
| `sys.path` so `src` imports resolve; `python eval/intent/run_eval.py` fails. |
|
|
| ```bash |
| uv run python -m eval.intent.run_eval # full dataset |
| uv run python -m eval.intent.run_eval --limit 6 # quick smoke test |
| uv run python -m eval.intent.run_eval --langfuse # also stream traces to Langfuse |
| ``` |
|
|
| Needs a populated `.env` (Azure OpenAI) — it calls the live model and spends |
| tokens. Output: a per-case detail table + an aggregate summary in the terminal, |
| and `results/eval_result_<timestamp>.json`. |
|
|
| **Tracking is the committed result files, not Langfuse** — the JSON reports in |
| `results/` are the versionable audit trail (see below). `--langfuse` is an |
| *optional* extra: when set, each case is also sent as a Langfuse trace (grouped |
| under one `intent_eval_<ts>` session) with a `intent_correct` 1/0 score, so the |
| same run is browsable in the Langfuse dashboard. It is off by default and the |
| eval runs fully without Langfuse configured. |
|
|
| ## What's measured |
|
|
| - **correctness** — overall + per-intent + per-language accuracy (`got == expected`) |
| - **runtime** — average ms per case |
| - **tokens** — input / output / total (read from the model response, no Langfuse) |
|
|
| ### Content-filter blocks count as `out_of_scope` passes |
|
|
| Aggressive jailbreak / manipulation inputs are often rejected by Azure's own |
| content filter (HTTP 400, `code=content_filter`) *before* the router model runs. |
| The live app treats that as a refusal (`chat_handler._is_content_filter_error`), |
| so for an `out_of_scope` case the block **is** the correct end-to-end outcome. The |
| runner mirrors this: such a case is recorded as `got=blocked` and scored **correct** |
| (not `ERROR:BadRequestError`). This keeps `out_of_scope` accuracy honest — the |
| router isn't penalised for inputs the platform guardrail caught first. A |
| content-filter block on any *other* expected intent is still scored wrong (an |
| unexpected block). Non-filter exceptions remain `ERROR:<type>` and score wrong. |
|
|
| ## Commit convention for `results/` |
|
|
| The reports are **versionable**, not a scratch log: |
|
|
| - **Do commit** a result after a meaningful change — e.g. a new |
| `intent_router.md` version, or new dataset cases. The new timestamped file |
| *adds* to the history; old files are never replaced. This is how we answer |
| "did accuracy improve after prompt v2?" — diff two committed result files. |
| - **Don't commit** throwaway runs while iterating. Just leave them unstaged or |
| delete them. |
|
|
| So the audit trail = prompt versions (in `src/config/prompts/`) lined up against |
| the committed result files here. |
|
|
| ## Dataset notes |
|
|
| - 6 intents: `chat`, `help`, `check`, `unstructured_flow`, `structured_flow`, |
| `out_of_scope`. Each has 6+ **distinct** scenarios (not EN/ID translation |
| pairs), balanced across English + Indonesian. (`problem_statement` was dropped |
| from the router on 2026-06-24 — the goal is now user-entered `objective` + |
| `business_questions`, no agent validation — so its cases were removed here.) |
| - `carried_over: true` rows mirror the pre-rework `intent_router.md` examples |
| (regression). `lang` enables per-language scoring. `id` is a stable handle for |
| diffing the same case across runs. |
| - Routing labels are decided from the question **phrasing**, not from which file |
| holds the answer (the router has no catalog access). See the `_grounding` note |
| in `intent_dataset.json`. |
| - Owner: Rifqi (structured/DB-grounded rows) + Sofhia (unstructured/document + |
| tabular-file rows). Merge both into this one file. |
| ``` |
| |
