| # Help-skill eval |
|
|
| Scores the **live** Help skill (`src/agents/handlers/help.HelpAgent`) — the guide that |
| tells a user where they are and what to do next. Each golden case declares an analysis |
| state + report-readiness + chat history; the runner streams `HelpAgent.astream` for real |
| and asserts the **rules** the reply must obey. |
|
|
| Unlike `eval/readiness` (deterministic, no LLM), this calls the model, so it needs a |
| working `.env` (Azure OpenAI) and spends tokens. Run it before a deploy that touches |
| `config/prompts/help.md` — not on every commit. The fast, no-LLM guard is |
| `tests/unit/agents/handlers/test_help.py` (fake chain); this is the end-to-end |
| "does the model actually obey the prompt" layer on top. |
|
|
| ## Run |
|
|
| ```bash |
| uv run python -m eval.help.run_eval |
| uv run python -m eval.help.run_eval --limit 4 # smoke test |
| uv run python -m eval.help.run_eval --no-table # summary only |
| ``` |
|
|
| Each run writes a timestamped `results/help_result_<ts>.json` (never overwritten, |
| diffable across runs). |
|
|
| ## What it measures |
|
|
| Not accuracy — Help replies are free prose with no single correct wording. The metric is |
| **compliance**: the % of cases whose reply obeys every rule asserted for it. |
|
|
| - **`language`** — the reply must match the user's language. This is the regression guard |
| for the button-path bug (`/tools/help` passes `message=None`, and the reply used to |
| default to English even for an Indonesian conversation). |
| - **`report_guard`** — never suggest generating a report when `report_ready.ready=false`; |
| do suggest it when `true`. Since `generate_report` is the only gated action, this also |
| serves as the "no action leakage" check. |
| - **`orientation`** — quality of the suggested starter questions. **Manual review**: these |
| run but are excluded from the auto compliance rate. Read their `output_text` in the JSON. |
| |
| Assertion types: `language_match {expected}`, `must_not_contain_any {patterns}`, |
| `must_contain_any {patterns}`. |
| |
| ## Held-out vs carried-over (why the summary splits them) |
| |
| `carried_over: true` cases **mirror an example in `help.md`** — the case `id` *is* the |
| prompt's `<!-- id: ... -->`. They are a regression guard: if the prompt is refactored, the |
| demonstrated rule must still hold. What is mirrored is the **input spec + the assertion**, |
| never the example's reply text (temperature > 0 makes exact match invalid). |
| |
| Held-out cases (`carried_over: false`) are **absent from the prompt**; their compliance is |
| the real generalization signal. If held-out compliance drops while carried-over stays at |
| 100%, the prompt is overfitting to its own examples ("train on test set"). That's why the |
| two are reported separately. |
| |
| **Sync rule (manual, like `intent`):** if `help.md`'s Examples change, keep the mirrored |
| `id`s here in sync. Current mirrored ids: `help_ex_orient`, `help_ex_guard_delta`, |
| `help_ex_guard_ready`. |
|
|
| ## Dataset |
|
|
| `help_dataset.json` — see the `_about` / `_carried_over` doc keys in the file. Language |
| detection reuses `help._detect_reply_language`; `report_ready.missing` uses the codes |
| `analysis` / `delta` mapped to the real `is_report_ready` strings in the runner. |
|
|
