| # Planner eval (E-planner) |
|
|
| Scores the **live planner** (`PlannerService.plan`) on golden questions against a |
| fixture catalog grounded in `PA Data Dummy.xlsx`. Purpose: a **regression net** |
| for changes to `planner.md` / `examples.py` — before/after any prompt tweak, run |
| this and confirm the target cases improve while `carried_over` cases stay green. |
|
|
| ## Why rule-compliance scoring (not exact-match) |
|
|
| A question has **many valid IRs**, so we don't compare IRs verbatim. Each case |
| pins only the **properties that matter** (`expect` assertions): does a count |
| question use a `count` aggregate? does entity ranking `group_by` the entity? does |
| a fuzzy model filter use `like` instead of an enumerated `in`? See `_expect_keys` |
| in `planner_dataset.json` for the full assertion vocabulary. |
|
|
| ## Run |
|
|
| ```bash |
| uv run python -m eval.planner.run_eval # full run (needs Azure creds) |
| uv run python -m eval.planner.run_eval --limit 6 # smoke test |
| uv run python -m eval.planner.run_eval --selfcheck # test the scorer, no LLM |
| ``` |
|
|
| Each run writes `results/planner_result_<timestamp>.json` (never overwritten). |
| `id` is stable per case, so runs diff case-by-case over time. |
|
|
| ## What's covered |
|
|
| | category | targets | |
| |---|---| |
| | `count` | scalar count → `count` aggregate (shipped fix) | |
| | `ranking` | top/bottom-N entities → `group_by` + `avg` + `order_by` + `limit` (**Bug 1**) | |
| | `fuzzy_filter` | partial model ref → `like`, never enumerate from samples (**Bug 2**) | |
| | `column_disambiguation` | "trend PA" must select `PA_Percent`, NOT `Plan_PA_Percent` (a wrong-column pick hidden behind alias `pa_percent`) | |
| | `chart` | a trend chart must aggregate before `render_chart` (not feed it 9,729 raw rows) + pick the right column | |
| | `aggregate`, `descriptive`, `correlation`, `trend`, `merge` | believed-correct baselines | |
| | `counter_raw_rows` | "show N records" must stay raw rows (guards Bug 1 fix from over-aggregating) | |
| | `counter_exact_filter` | exact filters stay exact (guards Bug 2 fix from over-`like`ing) | |
| | `infeasible` | measures absent from the catalog → `infeasible_reason` | |
|
|
| `carried_over: true` = behavior believed correct today (regression guard); |
| `false` = the known bugs. **Expected baseline (before the planner fixes):** the |
| `ranking` and `fuzzy_filter` (777) cases FAIL, everything else green — that gap is |
| exactly what the planner fixes should close, without turning any `carried_over` |
| case red. |
|
|
| ## Files |
|
|
| - `planner_dataset.json` — cases (question + `expect` assertions) |
| - `catalog_fixture.py` — the `PA Data Dummy` catalog the planner plans against |
| - `run_eval.py` — runner + deterministic scorer (`--selfcheck`) |
|
|
| > Date columns are typed `date` in the fixture (the *post-fix* catalog). The live |
| > system currently mis-types Excel date serials as `int` — an **ingest** bug, not |
| > a planner one — so the fixture types them correctly to keep this eval about |
| > planner logic. |
|
|
