[NOTICKET] tools: quality blocks for forecast + ab_test, forecast backtest, CK7 fix

Extends the reserved `quality` convention to the tools where it means something, and
fixes a CK7 message that was wrong for half its inputs.

analyze_forecast -- the substantial one. It reported NO accuracy at all, so "how
accurate is it?" had no answer. Now backtests: hold out the last h periods (bounded by
a quarter of the series and by the fitting floor), refit, and score with MASE -- mean
absolute error over the naive "next = last" error on the training span. MASE is
scale-free (Hyndman & Koehler 2006) and, unlike MAPE, neither explodes near zero nor
punishes over-prediction asymmetrically. Both fits run in ONE run_fit call, so a
forecast still costs one thread hop and one timeout. A series too short to hold
anything out, or one that never moves, ships UNVALIDATED -- also weak, with the two
causes reported as distinct reasons rather than one wrong one.

analyze_ab_test -- free and exact: the statistical verdict IS the quality verdict. A
gap the test cannot separate from noise is weak evidence however large the headline
difference looks. `significant is None` (too few rows, zero variance) is weak too.

FOUR tools deliberately do NOT get a block, and this is a judgement not an oversight:
pareto, segmentation arithmetic. A flat distribution or an unbalanced bin is a TRUE
finding, not an untrustworthy one -- that is a caveat.
data_quality reports the quality of the DATA; a block about the quality of
its own analysis is circular, and "data_quality is weak" as a
CK7 note is nonsense.
anomaly tried, measured, reverted. Unsupervised, so no accuracy; the
flag rate is a PARAMETER, and sklearn's contamination="auto"
alone flags ~22% of clean random data, so any rate ceiling
marks almost every normal run weak. CK7's signal is only
useful while it stays rare -- a block that cries wolf is worse
than none.

CK7 wording fix. It read "scored X, at or below its Y floor", which is right for r2
and silhouette and BACKWARDS for MASE and p-value, where high is bad. It now states
the value/floor pair and defers to the tool's own `note`, so direction never has to be
inferred -- consistent with the convention that the tool owns both the verdict and its
explanation.

Verification (no LLM):
816 passed, 7 skipped -- unchanged; ruff clean on touched paths; import main OK
forecast MASE: 0.19 clean trend | 1.45 pure noise (flagged) | unvalidated on a
6-point series and on a flat series, each with its own reason
ab_test: p=2e-57 real gap (ok) | p=0.5307 noise gap (flagged)
CK7 end-to-end on all three metric directions -- messages read correctly for
lower-is-worse AND higher-is-worse

Note: statsmodels was declared in pyproject (4d76bd0) but had never been installed in
this venv, so analyze_forecast had never actually executed here. Synced with
`uv sync --inexact` per section 5.21; the OCR/report stack survived (verified).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

DEV_PLAN.md

@@ -539,7 +539,7 @@ ambiguous, "feature importance" as too technical); `analyze_comparison` → **`a
 | A17 | Group-first selection — router emits a group hint, planner registry filters deterministically, full-registry fallback | R | ⬜ | Asked for directly in the discussion: *"milih grup dulu... karena tools-nya bisa berkembang banyak banget"*. **No second LLM call** — the two-call invariant holds |
 | A18 | Recipes per group + few-shots for P1 tools | R | ⬜ | Extends the existing R1–R6 recipe table. **Carries a deferred eval debt (2026-09-16):** `planner.md` rule 5 was rewritten then — it claimed "no modeling tools… no predictions", which four shipped ML tools made false, and the CRISP-DM bullet said "Never `modeling`" though `modeling` is a legal `CrispStage`. That edit shipped **without** its §7B `eval.planner` run, deliberately: all 8 ML tools are DEFERRED so the new rule is dormant, and no eval case exercises a modeling tool, so a run then would have re-measured nothing. **A18 must run the full eval covering BOTH changes** and compare against `planner_result_2026-09-15_165216.json` (29/31, carried_over 22/22). If that run regresses, rule 5 is a candidate cause alongside the recipes |
 | A19 | Few-shot balancing + de-dupe | R | ✅ | done 2026-09-15. Exemplars were **13/13 English** while `eval/planner` is 17 ID / 14 EN — an all-EN set teaches the planner that Indonesian is out-of-distribution. Converted D/H/L/M to Indonesian (now 9 EN / 4 ID), which also breaks the 3× `retrieve_data → analyze_aggregate` duplicate stack (A/D/E) and the J/M chart near-duplicate. **H now demonstrates the reply-language rule** — ID question → ID `infeasible_reason`, every other field English — which NO example showed before despite `prompt.py:151` mandating it. J↔K left matched (the collision-fix pair). Module docstring gained a balance note so the mix isn't "helpfully" translated back. Full eval **29/31 (94%), carried_over 22/22 green**; both failures are `carried_over=false` known-bug targets (`rank_units_worst_pa_en`, `chart_trend_pa_id`), neither a regression. Result: `planner_result_2026-09-15_165216.json`. ruff at its 2 pre-existing errors |
-| A20 | Model-quality checks | R | 🔄 | **Rescoped 2026-09-16 after inspecting real tool output.** CK9 (leakage), CK10 (class imbalance) and CK11 (constant features) as specified need fields **no tool emits**, and keying rules to each tool's ad-hoc field names would be fragile. Replaced by ONE generic rule plus a convention: an ML tool publishes a reserved **`quality`** block `{metric, value, floor, weak}` and **owns the verdict** — only it knows what "good" means for its method — and **CK7** flags any output whose `weak` is true. A new ML tool is covered the moment it emits the block, with no checkpoint change. Verified end-to-end on real fits: noise target r2 0.006 → `degraded` with a named note; real signal r2 0.99 → `ok`; a tool with no block → untouched. **Still open:** `analyze_forecast`/`analyze_anomaly` don't emit `quality` yet (Sofhia), and the `summary` → assembler half has no mechanism since `ToolResult` was dropped — needs a reserved `summary` key decision |
+| A20 | Model-quality checks | R | ✅ | **Rescoped 2026-09-16, completed 2026-09-17.** CK9/CK10/CK11 as originally specified needed fields no tool emits; replaced by one generic rule plus a convention — an ML tool publishes a reserved **`quality`** block `{metric, value, floor, weak, note}` and **owns the verdict**, and **CK7** surfaces it. **Coverage now 4 tools:** `analyze_importance` (r2/accuracy), `analyze_cluster` (silhouette), `analyze_forecast` (**MASE from a real holdout backtest — new, this was the "how accurate is it?" demo gap**), `analyze_ab_test` (p-value; the statistical verdict IS the quality verdict). **Deliberately NOT given one:** `analyze_pareto` / `analyze_segmentation` (arithmetic — a flat distribution is a true finding, not an untrustworthy one), `analyze_data_quality` (reports quality of the *data*; a block about its own analysis is circular), `analyze_anomaly` (unsupervised, no accuracy; sklearn's `contamination="auto"` alone flags ~22% of clean random data, so a rate ceiling marks almost every normal run weak — tried, measured, reverted). **CK7 wording fixed**: it claimed "at or below its floor", wrong for MASE and p-value where high is bad; now states the value/floor pair and defers to the tool's `note`. Verified: forecast MASE 0.19 clean trend / 1.45 noise / unvalidated on short + flat series with distinct reasons; ab_test p=2e-57 vs p=0.53. **816 passed, 7 skipped**. **Still open:** the `summary` → assembler half has no mechanism since `ToolResult` was dropped — needs a reserved `summary` key decision |
 | A21 | Report embedding — auto-charts into the downloaded report | R | ✅ | done 2026-09-16. Closes the gap Sofhia flagged at handoff: an auto-chart appeared in chat and silently vanished from the downloaded report. Extraction is now **one shared definition** (`src/charts/extract.py::chart_envelopes`) used by both the chat persistence path and `generator._collect_charts`, so the two can never drift again. Envelopes must be **renderable** (carry a `plotly` object), not merely present — a fence the FE cannot draw is worse than none; that guard existed in the report path only and now applies to persistence too. `_CHARTS_MAX_PER_RECORD` trim moved after collection, since one output can now carry several charts. Readiness needs no change — auto-charts come from `analyze_*`, which `_is_analysis_tool` already counts. **816 passed, 7 skipped** (unchanged); ruff clean; `import main` OK |
 | A22 | Per-task progress → SSE `status` | R | ✅ | done 2026-09-16. Granularity, not a longer budget — past ~10s a waiting user needs evidence of progress, not a faster answer. `TaskRunner.run` takes an optional `progress` callback and announces each wave **before** running it; new `slow_path/progress.py` holds a bounded, **bilingual** activity vocabulary (the 3 existing stage pings were English-only on a majority-Indonesian product). A 4-task plan now emits 6 `status` events instead of 3. A task's `objective` is deliberately NOT used as status text — plan internals are English, so it would show English prose to an ID user. Unknown tools fall back to "analysing", so the 6 tools still to land need no change here. Progress is never-throw (`_report`): a raising callback logs and the run continues. Verified: 4/4 tasks over 3 waves EN + ID, parallel wave renders as "Step 2–3 of 4", raising callback survived, no-progress back-compat holds. **816 passed, 7 skipped**; ruff clean on all 3 touched files; `import main` OK. Wire contract unchanged — `status` is still optional text |
 | A23 | Demo dry-run + rehearsal | S+R | ⬜ | Audience: Mas Beta, Pak Ricky, Pak Alvin |

REPO_STATUS.md

@@ -5,7 +5,7 @@
 
 **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
 **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only — no roadmap or to-dos.
-**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§12/§13 — `ReportStore.save` never wrote `reports.user_id`, and on the new **Neon** dedorch instance that column is `text NOT NULL`, so **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates every deployment shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. **Root cause (verified against the Go source after pulling `dd37b38`):** the column is *not* new — `0001_create_core_schema.sql` has declared it `NOT NULL` all along, but `0002`/`0004` retrofit it onto pre-existing DBs as a **nullable** `ADD COLUMN IF NOT EXISTS`. The old DB therefore accepted the column-less INSERT and the bug stayed latent; a fresh DB enforces it. Generalized as a §13 gotcha — **Go's migration set is not convergent, so fresh and migrated dedorch DBs differ in NOT NULL constraints** and the migrations alone can't tell you which shape you have. A full ORM-vs-live drift scan the same day found all five Python-written tables clean; the remaining drift is on read-only or unwired paths. **Knowledge pipeline 2026-08-21:** new **§14** — both halves of the knowledge pipeline are now in-tree (`src/knowledge_parsing/` · `src/knowledge_extraction/` · `eval/knowledge/`), as **offline CLIs with no HTTP surface, no table and nothing mounted in `main.py`**. They meet at one versioned `ParsedDocument` artifact; extraction never takes a file path. Term-filter recall **0.8537**, matching the prototype baseline. The existing `src/knowledge/` OCR→pgvector RAG path is untouched. Deps: `rapidfuzz`/`gliner`/`torch` main, `mineru` an optional extra. **Superseded 2026-09-07:** `mineru` dropped (commented out) and `torch` pinned to the Linux CPU wheel — 36 `nvidia-*` packages and `triton` gone from `uv.lock`; §14. **Knowledge 8-Sep task list 2026-09-14 (pr/21):** §8/§14 — the discussion's structure-data, incremental-ingest and review items are built. `knowledge_entries.label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`) so a lookup is a dictionary hit rather than a scan, with `entity_id` untouched as the thing approvals hang on; a re-uploaded document becomes a **new version** rather than a new document, so its approvals survive; an already-approved term is no longer re-extracted (one LLM call per cluster is the whole cost of that branch); an approval goes stale only when the **content the expert ruled on** changed, never when a count moved; a withdrawn document stops feeding the baseline without anything being deleted; and ingest takes a `review_mode` — `llm` approves on arrival with no reviewer step, recorded against the person who chose it. Seven columns added to dedorch (run by hand 2026-09-14 **after** the handoff went to Harry, and verified read-only against live). **Analytical tools foundation 2026-09-17 (`pr/22`):** §3/§6/§9/§10/§13 — three new modules under `src/tools/` (`spec_builder` derives a `ToolSpec` from a pydantic `InputModel`; `ml/executor` offloads blocking sklearn/statsmodels fits to a dedicated bounded pool with a per-fit timeout; `analytics/charts` is the shared chart kit, with `visualization.py` refactored onto it). **Tools stay pure functions — the class-based contract was designed and dropped** (DEV_PLAN §0.9). Analysis tools now ship an **auto-chart** via a reserved `"charts"` key → `ToolOutput.meta["charts"]` → the existing ChartStore save loop; `render_chart` is unchanged. New deps `statsmodels` + an explicit `scikit-learn` pin, and two new env knobs (`ML_MAX_WORKERS`, `ML_FIT_TIMEOUT_SECONDS`). Report embedding of auto-charts is **not** covered yet (§0.9 A21). **Analytical tools — first ML tools 2026-09-17 (`pr/22`):** §6/§9 — the first two ML `analyze_*` tools land, both registered **DEFERRED** (withheld from the planner until A0 scoping locks P1): `analyze_forecast` (`dae102f6` — ETS Holt-Winters, additive trend, **no AutoML**, actual-vs-forecast line chart) and `analyze_anomaly` (`9f3267d5` — IsolationForest, `stats` output, scatter with anomaly points highlighted). Both fit via `run_fit` and ship their chart through the reserved `"charts"` key. `analyze_forecast` is the **first `run_fit` user**: the invoker now `await`s coroutine compute fns (`inspect.isawaitable`, opt-in — the many sync tools are untouched), so an ML tool is `async` solely to offload its fit; conventions §6 ("never async") to be softened to "sync unless it fits a model" — **done 2026-09-16**, the conventions doc now reads "sync unless the tool fits a model". **Per-step progress 2026-09-16 (`pr/22`, A22):** §4/§9 — the slow path reports each plan step as it starts instead of three stage pings for a ~12s turn (past ~10s a waiting user needs evidence of progress, not a faster answer). New `slow_path/progress.py` holds a bounded **bilingual** activity vocabulary; `TaskRunner.run` gains an optional never-throw `progress` callback; the three coordinator stage pings became bilingual (they were English-only on a majority-Indonesian product). A 4-step plan emits ~6 `status` events instead of 3. **Wire contract unchanged** — `status` is still optional text the FE may ignore; contract §chat notes the new volume. **Report auto-chart embedding 2026-09-16 (`pr/22`, A21):** §5/§9 — the downloaded report now embeds an `analyze_*` tool’s auto-chart, not just `render_chart` output. Chart extraction moved to one shared definition (`src/charts/extract.py::chart_envelopes`) used by BOTH the per-turn persistence path and the report generator, so a chart can no longer appear in chat and vanish from the report. Envelopes must carry a `plotly` object to be emitted (the report path already required it; persistence now does too). No schema change, no contract change. **Analytical tools — importance + cluster, unregistered 2026-09-16 (`pr/22`):** §6/§9 — `analyze_importance` (standardised Ridge/Logistic coefficients with a leakage-safe train-only scaler and a holdout score; NOT permutation importance, which is the latency tall pole and buys nothing for a linear model) and `analyze_cluster` (KMeans, k chosen by silhouette over 2..8, scatter on two real columns rather than PCA components). **Both files are inert** — deliberately not in `_DISPATCH` or either registry list, pending the `registry.py`/`invoker.py` ownership call, so they cost nothing until registered. Both emit the new reserved **`quality`** block `{metric, value, floor, weak}`; the checkpoint gains **CK7**, one generic rule that flags any output whose tool declared itself weak — the tool owns the verdict, so a new ML tool is covered with no checkpoint change. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
+**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§12/§13 — `ReportStore.save` never wrote `reports.user_id`, and on the new **Neon** dedorch instance that column is `text NOT NULL`, so **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates every deployment shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. **Root cause (verified against the Go source after pulling `dd37b38`):** the column is *not* new — `0001_create_core_schema.sql` has declared it `NOT NULL` all along, but `0002`/`0004` retrofit it onto pre-existing DBs as a **nullable** `ADD COLUMN IF NOT EXISTS`. The old DB therefore accepted the column-less INSERT and the bug stayed latent; a fresh DB enforces it. Generalized as a §13 gotcha — **Go's migration set is not convergent, so fresh and migrated dedorch DBs differ in NOT NULL constraints** and the migrations alone can't tell you which shape you have. A full ORM-vs-live drift scan the same day found all five Python-written tables clean; the remaining drift is on read-only or unwired paths. **Knowledge pipeline 2026-08-21:** new **§14** — both halves of the knowledge pipeline are now in-tree (`src/knowledge_parsing/` · `src/knowledge_extraction/` · `eval/knowledge/`), as **offline CLIs with no HTTP surface, no table and nothing mounted in `main.py`**. They meet at one versioned `ParsedDocument` artifact; extraction never takes a file path. Term-filter recall **0.8537**, matching the prototype baseline. The existing `src/knowledge/` OCR→pgvector RAG path is untouched. Deps: `rapidfuzz`/`gliner`/`torch` main, `mineru` an optional extra. **Superseded 2026-09-07:** `mineru` dropped (commented out) and `torch` pinned to the Linux CPU wheel — 36 `nvidia-*` packages and `triton` gone from `uv.lock`; §14. **Knowledge 8-Sep task list 2026-09-14 (pr/21):** §8/§14 — the discussion's structure-data, incremental-ingest and review items are built. `knowledge_entries.label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`) so a lookup is a dictionary hit rather than a scan, with `entity_id` untouched as the thing approvals hang on; a re-uploaded document becomes a **new version** rather than a new document, so its approvals survive; an already-approved term is no longer re-extracted (one LLM call per cluster is the whole cost of that branch); an approval goes stale only when the **content the expert ruled on** changed, never when a count moved; a withdrawn document stops feeding the baseline without anything being deleted; and ingest takes a `review_mode` — `llm` approves on arrival with no reviewer step, recorded against the person who chose it. Seven columns added to dedorch (run by hand 2026-09-14 **after** the handoff went to Harry, and verified read-only against live). **Analytical tools foundation 2026-09-17 (`pr/22`):** §3/§6/§9/§10/§13 — three new modules under `src/tools/` (`spec_builder` derives a `ToolSpec` from a pydantic `InputModel`; `ml/executor` offloads blocking sklearn/statsmodels fits to a dedicated bounded pool with a per-fit timeout; `analytics/charts` is the shared chart kit, with `visualization.py` refactored onto it). **Tools stay pure functions — the class-based contract was designed and dropped** (DEV_PLAN §0.9). Analysis tools now ship an **auto-chart** via a reserved `"charts"` key → `ToolOutput.meta["charts"]` → the existing ChartStore save loop; `render_chart` is unchanged. New deps `statsmodels` + an explicit `scikit-learn` pin, and two new env knobs (`ML_MAX_WORKERS`, `ML_FIT_TIMEOUT_SECONDS`). Report embedding of auto-charts is **not** covered yet (§0.9 A21). **Analytical tools — first ML tools 2026-09-17 (`pr/22`):** §6/§9 — the first two ML `analyze_*` tools land, both registered **DEFERRED** (withheld from the planner until A0 scoping locks P1): `analyze_forecast` (`dae102f6` — ETS Holt-Winters, additive trend, **no AutoML**, actual-vs-forecast line chart) and `analyze_anomaly` (`9f3267d5` — IsolationForest, `stats` output, scatter with anomaly points highlighted). Both fit via `run_fit` and ship their chart through the reserved `"charts"` key. `analyze_forecast` is the **first `run_fit` user**: the invoker now `await`s coroutine compute fns (`inspect.isawaitable`, opt-in — the many sync tools are untouched), so an ML tool is `async` solely to offload its fit; conventions §6 ("never async") to be softened to "sync unless it fits a model" — **done 2026-09-16**, the conventions doc now reads "sync unless the tool fits a model". **Per-step progress 2026-09-16 (`pr/22`, A22):** §4/§9 — the slow path reports each plan step as it starts instead of three stage pings for a ~12s turn (past ~10s a waiting user needs evidence of progress, not a faster answer). New `slow_path/progress.py` holds a bounded **bilingual** activity vocabulary; `TaskRunner.run` gains an optional never-throw `progress` callback; the three coordinator stage pings became bilingual (they were English-only on a majority-Indonesian product). A 4-step plan emits ~6 `status` events instead of 3. **Wire contract unchanged** — `status` is still optional text the FE may ignore; contract §chat notes the new volume. **Report auto-chart embedding 2026-09-16 (`pr/22`, A21):** §5/§9 — the downloaded report now embeds an `analyze_*` tool’s auto-chart, not just `render_chart` output. Chart extraction moved to one shared definition (`src/charts/extract.py::chart_envelopes`) used by BOTH the per-turn persistence path and the report generator, so a chart can no longer appear in chat and vanish from the report. Envelopes must carry a `plotly` object to be emitted (the report path already required it; persistence now does too). No schema change, no contract change. **Analytical tools — importance + cluster, unregistered 2026-09-16 (`pr/22`):** §6/§9 — `analyze_importance` (standardised Ridge/Logistic coefficients with a leakage-safe train-only scaler and a holdout score; NOT permutation importance, which is the latency tall pole and buys nothing for a linear model) and `analyze_cluster` (KMeans, k chosen by silhouette over 2..8, scatter on two real columns rather than PCA components). **Both files are inert** — deliberately not in `_DISPATCH` or either registry list, pending the `registry.py`/`invoker.py` ownership call, so they cost nothing until registered. Both emit the new reserved **`quality`** block `{metric, value, floor, weak}`; the checkpoint gains **CK7**, one generic rule that flags any output whose tool declared itself weak — the tool owns the verdict, so a new ML tool is covered with no checkpoint change. **Model-quality checks complete 2026-09-17 (`pr/22`, A20):** §9 — four tools now publish the reserved `quality` block and CK7 surfaces it: importance (r2/accuracy), cluster (silhouette), **forecast (MASE from a new holdout backtest — previously it reported no accuracy at all)** and ab_test (p-value). Pareto, segmentation, data_quality and anomaly deliberately do NOT emit one — arithmetic results, a circular self-report, and an unsupervised flag rate that would mark ~every default run weak. CK7's message was direction-wrong for MASE and p-value ("at or below its floor" when high is bad) and now states the value/floor pair and defers to the tool's note. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
 the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
 own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
 **`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet — those skills

docs/tools/ANALYTICAL_TOOLS_CONVENTIONS.md

@@ -195,8 +195,26 @@ clustering, and the two aren't comparable. The checkpoint just surfaces the verd
 which is why a new ML tool is covered the moment it emits the block, with no checkpoint
 change. A tool that omits `quality` is simply not checked.
 
-Any tool that fits a model should emit it. `analyze_importance` and `analyze_cluster`
-do; `analyze_forecast` and `analyze_anomaly` do not yet (DEV_PLAN §0.9).
+**The floor is not always a minimum.** r² and silhouette are bad when *low*; MASE and
+a p-value are bad when *high*. The tool decides `weak`, so direction never has to be
+inferred — and CK7's message states the value/floor pair and defers to your `note`
+rather than claiming "below the floor", which would be wrong half the time. Write a
+`note` that says what actually happened.
+
+**Emitting it is a judgement, not a formality.** A tool that fits a model or runs a
+test should: `analyze_importance` (r²/accuracy), `analyze_cluster` (silhouette),
+`analyze_forecast` (MASE from a holdout backtest), `analyze_ab_test` (p-value — the
+statistical verdict *is* the quality verdict).
+
+Arithmetic tools should not. `analyze_pareto` and `analyze_segmentation` compute
+exactly what was asked; a flat distribution or an unbalanced bin is a true finding, not
+an untrustworthy one — that belongs in a caveat. `analyze_data_quality` reports the
+quality of the *data*, so a block about the quality of its own analysis would be
+circular. And `analyze_anomaly` has no block on purpose: it is unsupervised, so there
+is no accuracy, and the flag rate is a *parameter* — sklearn's `contamination="auto"`
+alone flags ~20% of clean random data, so a rate ceiling would mark almost every normal
+run weak. A quality block that cries wolf is worse than none, because CK7's signal is
+only useful while it stays rare.
 
 ### The chart kit
 

src/agents/slow_path/checkpoint.py

@@ -118,10 +118,16 @@ def _assess(run_state: RunState, task_list: TaskList) -> RunAssessment:
             # emits the block, with no change here. Absent block -> nothing to check.
             quality = out.value.get("quality") if isinstance(out.value, dict) else None
             if isinstance(quality, dict) and quality.get("weak"):
+                # Direction-agnostic wording on purpose: "below the floor" is bad for
+                # r2 and silhouette but GOOD for MASE and p-value, so the message
+                # states the pair and lets the tool's own `note` explain it. The tool
+                # owns the verdict AND its explanation; the checkpoint only surfaces
+                # both. Without a note, fall back to the neutral phrasing.
+                detail = quality.get("note") or "the result is weak evidence"
                 flag(tid, "CK7",
-                     f"{out.tool} scored {quality.get('metric')}="
-                     f"{quality.get('value')}, at or below its "
-                     f"{quality.get('floor')} floor — the result is weak evidence",
+                     f"{out.tool} reported {quality.get('metric')}="
+                     f"{quality.get('value')} against a {quality.get('floor')} "
+                     f"floor — {detail}",
                      repairable=False)
 
     # CK5 — an analyze_* consumed a table whose column(s) are entirely null.

src/tools/analytics/ab_test.py

@@ -200,5 +200,21 @@ def analyze_ab_test(
         "diff_pct": diff_pct,
         "comparison": comparison,
         "test": test,
+        # Reserved `quality` block (conventions §3). For an A/B comparison the
+        # statistical verdict IS the quality verdict: a gap the test cannot separate
+        # from noise is weak evidence no matter how large the headline difference
+        # looks. `significant is None` means the test could not run (too few rows,
+        # zero variance) — also weak, for the same reason. CK7 surfaces it.
+        "quality": {
+            "metric": "p_value",
+            "value": None if test["p_value"] is None else round(test["p_value"], 4),
+            "floor": alpha,
+            "weak": not test["significant"],
+            "note": (
+                "the difference is not distinguishable from noise at this sample size"
+                if not test["significant"]
+                else f"significant at alpha={alpha}"
+            ),
+        },
         "charts": [_ab_chart(group_a, group_b, y_a, y_b, err_a, err_b, value_column)],
     }

src/tools/analytics/forecast.py

@@ -40,6 +40,12 @@ from src.tools.spec_builder import build_description
 # ETS with an additive trend and estimated initialisation needs a few points to
 # be meaningful; below this we raise rather than emit a garbage projection.
 _MIN_POINTS = 4
+# A holdout backtest needs points to train on AND points to score against; below this
+# the forecast ships UNVALIDATED, which is itself weak evidence (see `quality`).
+_MIN_BACKTEST_POINTS = 8
+# MASE >= 1 means the model did no better than "next = last" on the holdout — the
+# floor the reserved `quality` block reports and the checkpoint's CK7 reads.
+_MASE_FLOOR = 1.0
 
 
 class NotEnoughDataError(ValueError):
@@ -78,15 +84,54 @@ DESCRIPTION = build_description(
 )
 
 
-def _fit_ets(values: np.ndarray):
-    """Fit an additive-trend ETS model. Imported lazily so app startup doesn't pay
-    for statsmodels unless a forecast actually runs. Runs inside run_fit's thread."""
+def _ets(values: np.ndarray):
+    """An additive-trend ETS fit. Imported lazily so app startup doesn't pay for
+    statsmodels unless a forecast actually runs."""
     from statsmodels.tsa.holtwinters import ExponentialSmoothing
 
-    model = ExponentialSmoothing(
+    return ExponentialSmoothing(
         values, trend="add", seasonal=None, initialization_method="estimated"
-    )
-    return model.fit()
+    ).fit()
+
+
+def _fit_ets(values: np.ndarray, horizon: int):
+    """Fit the reported model AND backtest it. Runs inside run_fit's thread.
+
+    Both fits happen in one offloaded call so a forecast costs one thread hop and one
+    timeout, not two.
+
+    The backtest is a single holdout: train on everything but the last `h` periods,
+    forecast those, and score with **MASE** — mean absolute error divided by the mean
+    absolute error of the naive "next = last" forecast on the training span. MASE is
+    the scale-free choice (Hyndman & Koehler 2006) and, unlike MAPE, it neither
+    explodes near zero nor punishes over-prediction asymmetrically. It also reads
+    plainly: **below 1 beats doing nothing, at or above 1 does not.**
+
+    Returns (model, mase, reason). `mase` is None when no backtest was possible, and
+    `reason` then says WHY in the user's terms — the two causes are different and a
+    wrong explanation is worse than none.
+    """
+    model = _ets(values)
+
+    n = values.shape[0]
+    too_short = "too few periods to hold any out"
+    if n < _MIN_BACKTEST_POINTS:
+        return model, None, too_short
+    # Hold out the forecast horizon, but never more than a quarter of the series and
+    # never so much that training drops below the fitting floor.
+    holdout = max(1, min(horizon, n // 4, n - _MIN_POINTS))
+    train, test = values[:-holdout], values[-holdout:]
+    if train.shape[0] < _MIN_POINTS:
+        return model, None, too_short
+
+    predicted = np.asarray(_ets(train).forecast(holdout), dtype=float)
+    naive_error = float(np.mean(np.abs(np.diff(train))))
+    if naive_error == 0:
+        # The series never moves, so the naive baseline has zero error and MASE is
+        # undefined (not infinite). A forecast of a constant is trivially "right" and
+        # tells the user nothing — still weak, but for this reason, not shortness.
+        return model, None, "the series never changes, so there is nothing to validate against"
+    return model, float(np.mean(np.abs(test - predicted)) / naive_error), None
 
 
 def _forecast_chart(
@@ -182,7 +227,9 @@ async def analyze_forecast(
     # ETS needs a gap-free series; interpolate the empty interior periods.
     filled = resampled.interpolate(limit_direction="both")
 
-    model = await run_fit(_fit_ets, filled.to_numpy(dtype=float), label="analyze_forecast")
+    model, mase, mase_reason = await run_fit(
+        _fit_ets, filled.to_numpy(dtype=float), horizon, label="analyze_forecast"
+    )
     fc_values = [float(v) for v in np.asarray(model.forecast(horizon), dtype=float)]
 
     future_index = pd.date_range(
@@ -219,5 +266,22 @@ async def analyze_forecast(
         "change_abs": change_abs,
         "change_pct": change_pct,
         "direction": direction,
+        # Reserved `quality` block (conventions §3): the tool reports its own score and
+        # its own verdict; CK7 surfaces `weak`. An unvalidated forecast (series too
+        # short to hold anything out) counts as weak — shipping a projection nobody
+        # checked is exactly the case the plan's R-3 wanted flagged.
+        "quality": {
+            "metric": "mase",
+            "value": None if mase is None else round(mase, 4),
+            "floor": _MASE_FLOOR,
+            "weak": mase is None or mase >= _MASE_FLOOR,
+            "note": (
+                f"not validated — {mase_reason}"
+                if mase is None
+                else "did not beat a naive 'next = last' forecast on the holdout"
+                if mase >= _MASE_FLOOR
+                else "beats a naive 'next = last' forecast on the holdout"
+            ),
+        },
         "charts": [_forecast_chart(history, forecast_pts, value_column)],
     }