[NOTICKET] docs(plans): audit fix tracker (P0 list, decisions D1–D7, gates)

Tracks the P0 fixes from the 3 Nov prompt & tool-logic audit. P0-0 done: the
ranking regression (rank_units_worst_pa_*) passes 3/4 with the pre-71c6f2b2
planner.md + examples vs 3/8 with the current ones, so the 10-28 cut is the
likely cause; the fixture-only fix moves to P1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plans/AUDIT_FIXES_2026-11.md

@@ -0,0 +1,57 @@
+# Audit fixes — prompt & tool-logic audit, November 2026
+
+<!-- doc-status -->
+> 🔄 **LIVE PLAN — P0 in progress (started 2026-11-03).** Status legend: ⬜ not started · 🔄 in progress · ✅ done · ⛔ blocked · 🔎 verify · ⏸️ deferred.
+
+**Branch:** `pr/27`. **Why:** a stage-split audit (plan / execute & answer / report / small flows,
+4 auditors + an independent peer review, 3 Nov) of the FE test analysis 2d841e2f, the GPT-6 Luna
+trial run, the planner eval and Sofhia's period test. 40 findings collapse into 8 root causes;
+the P0 list below is what lands before the ~11 Nov final. The demo recording of 4 Nov stays on
+the version tested on 3 Nov (`main` + PR 27 as of `d0f8980e`) — nothing here is merged before
+its own 7-question check passes.
+
+## Decisions (Rifqi, 2026-11-03)
+| Id | Decision |
+|---|---|
+| D1 | **A** — code owns the objective's period (default filter on every `retrieve_data`, override path), like site/section |
+| D1a | The default period is released for forecast history, year-on-year / "dibanding tahun lalu", and when the question names its own period; the scope line always says which period was used |
+| D1b | For a month window, prefer the month-grain table; week numbers only when the question asks weekly |
+| D2 | Indonesian register for fixed texts and help: **Anda** |
+| D3 | Dictionary entries (derived-from for is_bad_actor / bad_actor_score / less_than_q1 / q1_bad_actor_score; "unit/equipment → eqm_no"; "MTTR = rata-rata per unit") — data, Rifqi uploads |
+| D4 | Sofhia owns the analysis-tool items (P0-8 and the tool P1 items) |
+| D5 | Run the ranking-regression check before any planner.md edit |
+| D6 | Measure the input guard on ordinary id questions — later |
+| D7 | Relative windows ("6 bulan terakhir") count back from **today**; the answer states where the data ends |
+
+## P0
+| Id | Change | Owner | Status | Evidence / commit |
+|---|---|---|---|---|
+| P0-0 | Confirm the ranking-regression cause (rank cases, pre-cut vs current planner.md + examples) | Rifqi (lead) | ✅ | Pre-cut 3/4 pass vs current 3/8 (3 Nov) → the 10-28 cut is the likely cause (meets the ≥3/4 bar). Only the eval fixture has a `Model_Unit` column; EMA demo data does not → fix moved to P1 (render distinct counts + "unit → equipment column" dictionary term). ~95k tokens |
+| P0-1 | Objective period owned by code (D1/D1a/D1b) | Rifqi (lead) | ✅ (uncommitted) | `src/agents/scope/period.py`: parse window (EN/ID ranges incl. s/d; none for Q/semester/lists/ids like "site 2009"), release when the user's own words name a period / the plan forecasts / the queries compare periods; applied to every retrieve_data (ISO weeks on week-only tables); coordinator hook; `TurnScope.periodic` + "semua periode" marker; period filters shown first in the scope line. Two independent review rounds (3 + 2 blockers found and fixed). D1b routing to the month table → P0-5 |
+| P0-2 | One validator rule for queries on repeated periodic rows + count→n_periods repair | Rifqi (lead) | ✅ (uncommitted) | validator Check 8m (ordered+limited raw rows of a year/month/week/quarter table) and 8n (segmentation fed raw period rows); repairer adds `COUNT(*) AS n_periods` to grouped rankings (not when grouped by a period). Replay of 135 stored eval plans: 0 new rejects. Reviewed |
+| P0-3 | Assembler sees executed table · rows · filters; planner notes labelled "written before execution" | Rifqi (lead) | 🔎 | `slow_path/prompt.py` renders `(ran on <table> · <n> rows · filters: …)` per retrieve; planner notes relabelled; 1 assembler.md line. Prompt change → needs the 7-question chat_sim gate |
+| P0-4 | Scope engine: only `${tN.col}` are lookups; strip the lookup table's own-key filters | Rifqi (lead) | ⬜ | |
+| P0-5 | Validator: relative-period question with a missing / whole-year period filter → retry (D7) | Rifqi (lead) | ⬜ | |
+| P0-6 | Report summary generated after the answers (+1 prompt line) | Rifqi (lead) | ⬜ | |
+| P0-7 | Report caps status on model-fit `quality.weak` only (importance, cluster, forecast, anomaly — not ab_test) | Rifqi (lead) | ⬜ | |
+| P0-8 | Trend/forecast keep empty periods empty + per-period n + stub flag; correlation n + p-value + quality | Sofhia | ⬜ | |
+| P0-9 | Exhausted plan gets its own message ("Anda", no data-gap claim) | Rifqi (lead) | ⬜ | |
+
+## Verification gates (paid runs on GPT-5.4-mini, serial, each with Rifqi's go)
+| Gate | Tokens (est.) | Status |
+|---|---|---|
+| Local tests + ruff + `import main` after every item | free | 🔄 (after P0-1..3: 1350 passed / 8 skipped, ruff clean, import OK) |
+| P0-0 rank cases | ~160k (used ~95k) | ✅ |
+| Planner rank subset after P0-2 | ~100k | ⬜ |
+| Report fixture live + 2d84 summary replay after P0-6 | ~10k | ⬜ |
+| One 7-question `chat_sim --suite ema_demo` after all P0 | ~350k | ⬜ |
+
+## P1 (after P0)
+Relative-period resolver · planner sees objective only (no BQs) · domain gap limits definitions,
+never the plan · report: not-applied scope and mixed periods lower the status · one count line,
+count words, primary chart first, evidence shows p/n/quality · limitations from backing records ·
+tool items (Sofhia: A/B effect size + per-unit option, segmentation boundaries, multiclass
+importance, single-feature leak flag, descriptions vs planner.md) · planner repairs (render_chart
+after a charting tool, value-type coercion, stringified tasks) · count_distinct for "how many units"
+· router keeps the user's language · help/refusal/latency items · eval runner list fix · PII mask
+on cluster members. Full detail: the lead's audit notes (session scratchpad `audit\final_plan.md`).