[KM-717] feat(planner): read the catalog data dictionary into the planner's catalog summary

Go 887d98a (KM-717) stores user-entered data-dictionary fields on each catalog
column. Python's Column model silently dropped them (pydantic extra="ignore"),
so none reached the planner.

- catalog/models.py: Column declares the eight optional fields (description,
business_meaning, unit_of_measure, classify, aggregation, is_key,
data_classification, validation_rules).
- planner/inputs.py: CatalogSummary renders one capped (300-char) "dictionary:"
line under a column that has an entry, inside the existing <data> block.
Governance fields are not rendered; PII columns show the entry but still no
values.
- A catalog without entries renders byte-identical (eval.planner fixtures, 5,109
bytes), so committed eval results stand; planner.md unchanged.

Docs: Go migration 0008 (66e70ca) adopts the five knowledge tables. REPO_STATUS
§8/§9/§12/§14, DEV_PLAN D2 and P2-I1..I3, knowledge_tables.sql,
KNOWLEDGE_PERSISTENCE_CONTRACT.md and docs/README.md updated.

Verified: 1016 passed, 7 skipped (1010 + 6 new local tests); ruff clean on
touched files; import main OK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

DEV_PLAN.md

@@ -438,7 +438,7 @@ scores **1.000** on a committed fixture; see the K9 row for how to read that num
 | S1 | Agree the parsed-artifact schema | Sofhia + Rifqi | ✅ | **Settled 2026-08-21.** `src/knowledge_parsing/contracts.py` (`schema_version 0.2.0`; **0.3.0** as of the 2026-08-27 fixture) is the shape of record and the extraction adapter consumes it directly. Envelope, verbatim heading, 0-based `page_idx`, prose `text` with `latex`/`table_html` beside it |
 | S5 | Keep the seam off the file path | Sofhia + Rifqi | ✅ | **Satisfied both sides.** `ParsedDocument` carries no handle to the parser, and `knowledge_extraction.adapter` accepts only the artifact. Locked by local seam tests importing the parsing contract directly |
 | X* | Build extraction v2 in this repo | Rifqi | ✅ | **Done 2026-08-21** — `src/knowledge_extraction/`: adapter · filters (span/cue/legend) · cluster · rank · extract (4 branches) · validate · diff · queue · service facade · CLI. Recall matches the frozen baseline exactly |
-| D2 | Artifact + candidate-entry persistence | Rifqi → Harry | ✅ (Python) / ⬜ (Go) | **Built and live 2026-09-07.** Five tables created in dedorch (`docs/knowledge/knowledge_tables.sql`), ORM models, `knowledge_store.py`, `PostgresJobStore`. Verified end to end: 88 entry rows persisted from the BUMA standard. **⚠️ Harry must still fold the SQL into Go's migration set** — the tables exist only because they were run by hand, the same gap that left `message_traceability`/`message_charts` out |
+| D2 | Artifact + candidate-entry persistence | Rifqi → Harry | ✅ (Python) / ✅ (Go) | **Go side done 2026-09-22 (recorded 2026-09-30):** Go migration `0008_create_knowledge_pipeline_schema.sql` (`66e70ca`) creates all five tables from the current `knowledge_tables.sql`, drift and §6/§7 columns included (read in the Go source at `887d98a`). Open: confirm the hand-created live Neon tables match `0008` via `information_schema` (as #32). **Built and live 2026-09-07.** Five tables created in dedorch (`docs/knowledge/knowledge_tables.sql`), ORM models, `knowledge_store.py`, `PostgresJobStore`. Verified end to end: 88 entry rows persisted from the BUMA standard. **⚠️ Harry must still fold the SQL into Go's migration set** — the tables exist only because they were run by hand, the same gap that left `message_traceability`/`message_charts` out |
 | D4 | Admin ingest surface | Rifqi | ✅ | **Built 2026-09-07.** 8 routes under `/api/v1/knowledge` (contract §Knowledge): ingest + poll, re-extract, entries, queue, reviews. Async batch (202 + poll, not SSE), company-scoped, rate-limited, spend-gated, single-flight. **Auth posture recorded, not solved:** unauthenticated like the rest of the surface, and `POST …/reviews` accepts content that reaches a later prompt — the strongest case yet for #43 |
 | R1 | Fix the literal-source-wording violation | Rifqi | ✅ | **Fixed and live-verified 2026-08-21.** `source_wording` is read from the document's own heading rather than asked of the model (asked directly, it quoted the normalised page-1 form); the span haystack includes headings; the queue flags the mismatch. PA now surfaces as *"source wording differs from the expanded name"* |
 | D3 | Provision a mini deployment | Rifqi | ⏸️ | **Deferred 2026-08-19: staying on nano.** The `__54n` quad is wired in `src/config/settings.py`. Accepted knowingly — nano measured 0.75 against a 0.80 line, and `rule`/`summary` run on the tier whose failure mode is least detectable |
@@ -679,6 +679,9 @@ E (~925k) → D (~865k) → C2 · F (optional). Ask before each full run.
 | P2-G3 | `chart_trend_pa_id` is fragile — retries in every run, failed the final-config run | Rifqi | 🔎 | if it recurs, look at D2 (the viz-tail row left the system prompt) first |
 | P2-H1 | PaddleOCR-VL parser for `POST /api/v2/document/process`, server toggle `DOCUMENT_PARSER_BACKEND` (default `paddleocr`); DOCX refused on it (option a) | Rifqi | ✅ | 2026-09-24 — built + 19 offline tests (64-page Komatsu result, fake job API). **Live:** 2-page job 15 s; full 64-page manual via v2 process → completed, 80 chunks, 27 figures, signed figure URL 200. Live-only finding fixed: `format_block_content: False` leaves image blocks empty → figures matched by label+bbox. Open: data residency (Baidu-hosted processing + result URLs) — Rifqi's call |
 | P2-H2 | DOCX without Mistral: local `python-docx` parser (no OCR, no vendor) | Rifqi | ⏸️ | option (b), deferred 2026-09-24 |
+| P2-I1 | **Data dictionary read side (KM-717)** — Go `887d98a` stores per-column dictionary fields; Python must surface them to the planner | Rifqi | ✅ | done 2026-09-30. `catalog/models.py::Column` declares the eight optional fields (they were silently dropped before — pydantic `extra="ignore"`); `planner/inputs.py::CatalogSummary` renders a capped `dictionary:` line under a column that has one, inside the existing `<data>` block. `planner.md` untouched. No-entry catalogs render **byte-identical** (eval.planner fixtures, 5,109 bytes), so the committed eval results stand. Suite **1016 passed / 7 skipped** (1010 + 6 new local tests); ruff clean; `import main` OK. Motivated by the BMDP demo (deck slide 10, fix list F4 option b) |
+| P2-I2 | `eval.planner` case with a dictionary entry (does the planner honour `aggregate with: avg` / the unit?) | Rifqi | ⬜ | Paid run — ask first. Until then the dictionary's *effect* on plans is unmeasured |
+| P2-I3 | FE handoff: tabular upload regression + analysis-catalog rebuild after a dictionary save | Rifqi → Harry | ⬜ | FE `f797178` sends CSV/XLSX to Python v2 (rejected) and offers only pdf/docx/txt; `rebuildAnalysisDataCatalog` is defined but never called, so a dictionary saved after binding never reaches that analysis. Handoff drafted 2026-09-30 |
 
 ## 5. Critical path & sequencing
 

REPO_STATUS.md

@@ -5,7 +5,7 @@
 
 **Audience:** teammates onboarding onto the Python repo (`Agentic-Service-Data-Eyond-Catalog`).
 **Scope:** what the code does **right now** (branch `pr/4`, ticket KM-652). Describes current state only — no roadmap or to-dos.
-**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§12/§13 — `ReportStore.save` never wrote `reports.user_id`, and on the new **Neon** dedorch instance that column is `text NOT NULL`, so **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates every deployment shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. **Root cause (verified against the Go source after pulling `dd37b38`):** the column is *not* new — `0001_create_core_schema.sql` has declared it `NOT NULL` all along, but `0002`/`0004` retrofit it onto pre-existing DBs as a **nullable** `ADD COLUMN IF NOT EXISTS`. The old DB therefore accepted the column-less INSERT and the bug stayed latent; a fresh DB enforces it. Generalized as a §13 gotcha — **Go's migration set is not convergent, so fresh and migrated dedorch DBs differ in NOT NULL constraints** and the migrations alone can't tell you which shape you have. A full ORM-vs-live drift scan the same day found all five Python-written tables clean; the remaining drift is on read-only or unwired paths. **Knowledge pipeline 2026-08-21:** new **§14** — both halves of the knowledge pipeline are now in-tree (`src/knowledge_parsing/` · `src/knowledge_extraction/` · `eval/knowledge/`), as **offline CLIs with no HTTP surface, no table and nothing mounted in `main.py`**. They meet at one versioned `ParsedDocument` artifact; extraction never takes a file path. Term-filter recall **0.8537**, matching the prototype baseline. The existing `src/knowledge/` OCR→pgvector RAG path is untouched. Deps: `rapidfuzz`/`gliner`/`torch` main, `mineru` an optional extra. **Superseded 2026-09-07:** `mineru` dropped (commented out) and `torch` pinned to the Linux CPU wheel — 36 `nvidia-*` packages and `triton` gone from `uv.lock`; §14. **Knowledge 8-Sep task list 2026-09-14 (pr/21):** §8/§14 — the discussion's structure-data, incremental-ingest and review items are built. `knowledge_entries.label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`) so a lookup is a dictionary hit rather than a scan, with `entity_id` untouched as the thing approvals hang on; a re-uploaded document becomes a **new version** rather than a new document, so its approvals survive; an already-approved term is no longer re-extracted (one LLM call per cluster is the whole cost of that branch); an approval goes stale only when the **content the expert ruled on** changed, never when a count moved; a withdrawn document stops feeding the baseline without anything being deleted; and ingest takes a `review_mode` — `llm` approves on arrival with no reviewer step, recorded against the person who chose it. Seven columns added to dedorch (run by hand 2026-09-14 **after** the handoff went to Harry, and verified read-only against live). **Analytical tools foundation 2026-09-17 (`pr/22`):** §3/§6/§9/§10/§13 — three new modules under `src/tools/` (`spec_builder` derives a `ToolSpec` from a pydantic `InputModel`; `ml/executor` offloads blocking sklearn/statsmodels fits to a dedicated bounded pool with a per-fit timeout; `analytics/charts` is the shared chart kit, with `visualization.py` refactored onto it). **Tools stay pure functions — the class-based contract was designed and dropped** (DEV_PLAN §0.9). Analysis tools now ship an **auto-chart** via a reserved `"charts"` key → `ToolOutput.meta["charts"]` → the existing ChartStore save loop; `render_chart` is unchanged. New deps `statsmodels` + an explicit `scikit-learn` pin, and two new env knobs (`ML_MAX_WORKERS`, `ML_FIT_TIMEOUT_SECONDS`). Report embedding of auto-charts is **not** covered yet (§0.9 A21). **Analytical tools — first ML tools 2026-09-17 (`pr/22`):** §6/§9 — the first two ML `analyze_*` tools land, both registered **DEFERRED** (withheld from the planner until A0 scoping locks P1): `analyze_forecast` (`dae102f6` — ETS Holt-Winters, additive trend, **no AutoML**, actual-vs-forecast line chart) and `analyze_anomaly` (`9f3267d5` — IsolationForest, `stats` output, scatter with anomaly points highlighted). Both fit via `run_fit` and ship their chart through the reserved `"charts"` key. `analyze_forecast` is the **first `run_fit` user**: the invoker now `await`s coroutine compute fns (`inspect.isawaitable`, opt-in — the many sync tools are untouched), so an ML tool is `async` solely to offload its fit; conventions §6 ("never async") to be softened to "sync unless it fits a model" — **done 2026-09-16**, the conventions doc now reads "sync unless the tool fits a model". **Per-step progress 2026-09-16 (`pr/22`, A22):** §4/§9 — the slow path reports each plan step as it starts instead of three stage pings for a ~12s turn (past ~10s a waiting user needs evidence of progress, not a faster answer). New `slow_path/progress.py` holds a bounded **bilingual** activity vocabulary; `TaskRunner.run` gains an optional never-throw `progress` callback; the three coordinator stage pings became bilingual (they were English-only on a majority-Indonesian product). A 4-step plan emits ~6 `status` events instead of 3. **Wire contract unchanged** — `status` is still optional text the FE may ignore; contract §chat notes the new volume. **Report auto-chart embedding 2026-09-16 (`pr/22`, A21):** §5/§9 — the downloaded report now embeds an `analyze_*` tool’s auto-chart, not just `render_chart` output. Chart extraction moved to one shared definition (`src/charts/extract.py::chart_envelopes`) used by BOTH the per-turn persistence path and the report generator, so a chart can no longer appear in chat and vanish from the report. Envelopes must carry a `plotly` object to be emitted (the report path already required it; persistence now does too). No schema change, no contract change. **Analytical tools — importance + cluster, unregistered 2026-09-16 (`pr/22`):** §6/§9 — `analyze_importance` (standardised Ridge/Logistic coefficients with a leakage-safe train-only scaler and a holdout score; NOT permutation importance, which is the latency tall pole and buys nothing for a linear model) and `analyze_cluster` (KMeans, k chosen by silhouette over 2..8, scatter on two real columns rather than PCA components). **Both files are inert** — deliberately not in `_DISPATCH` or either registry list, pending the `registry.py`/`invoker.py` ownership call, so they cost nothing until registered. Both emit the new reserved **`quality`** block `{metric, value, floor, weak}`; the checkpoint gains **CK7**, one generic rule that flags any output whose tool declared itself weak — the tool owns the verdict, so a new ML tool is covered with no checkpoint change. **Model-quality checks complete 2026-09-17 (`pr/22`, A20):** §9 — four tools now publish the reserved `quality` block and CK7 surfaces it: importance (r2/accuracy), cluster (silhouette), **forecast (MASE from a new holdout backtest — previously it reported no accuracy at all)** and ab_test (p-value). Pareto, segmentation, data_quality and anomaly deliberately do NOT emit one — arithmetic results, a circular self-report, and an unsupervised flag rate that would mark ~every default run weak. CK7's message was direction-wrong for MASE and p-value ("at or below its floor" when high is bad) and now states the value/floor pair and defers to the tool's note. **Group-first tool selection 2026-09-17 (`pr/22`, A17):** §4/§9 — A0 activated every analytics tool, putting the planner-visible set at **18**, inside the 15–20 band where tool-selection accuracy degrades. Rather than withhold tools, the router now emits a **`group_hint`** on its existing structured-output call and a deterministic filter (`src/agents/planner/groups.py`, 0 LLM) shows the planner only that group's tools plus the always-on core — **8–9 instead of 18**, 11 for a two-group question. Groups are derived from `ToolSpec.category` rather than stored on it, so the tool layer is untouched. **Fail-open by design:** the router is catalog-blind (it runs before the catalog read), so an absent, empty or unknown hint yields the FULL registry — a filtered-out tool is one the planner cannot ask for. The two-LLM-call slow-path invariant is unchanged. **All tools active + two suppression fixes 2026-09-17 (`pr/22`, A15/A24/A25):** §9 — A15 moved all 8 remaining specs to ACTIVE (`DEFERRED` is now empty, 18 planner-visible unscoped). Two things would have kept the new ML tools unusable and are fixed: `prompt.py` injected **`modeling_allowed: False (no modeling tools exist in v1…)`** on every planner call — contradicting the rewritten `planner.md` rule 5 — so the default is now True and the text follows the flag; and `render_examples()` ignored group scoping, showing a scoped turn examples for tools it could not call, so it now takes the visible tool set. Both infeasible examples survive every scope (an infeasible plan calls no tools), so the refusal discipline is never scoped away. **Tool honesty + repair telemetry 2026-09-21 (`pr/22`, A32–A34):** §9 — three tools stopped claiming more than they computed, and the checkpoint started measuring. `analyze_data_quality` advertised duplicate detection in `use_when` and computed none; it now returns `duplicate_rows {count, rate}` + `issues.duplicate_row_count`, always present (0 when none). `analyze_anomaly` could not **name** what it flagged — a record carried only the scored numeric columns plus a score, so "which units are unusual?" was structurally unanswerable; it now takes optional **`label_columns`** carried into each record (identifiers first, never scored) and publishes a `quality` block (`anomaly_rate`, weak only **above 0.5**). That ceiling was **re-measured, not re-assumed** — the A20 revert stands for a naive low ceiling, but clean gaussian noise flags 0.14–0.22 (p95 0.22) and a genuine 30%-outlier frame flags 0.30, so 0.5 clears both while still catching the demo's degenerate 0.86. Planner side: recipe **R14** and exemplar **U** now retrieve an identifier and pass it as `label_columns` (the tool-side capability is useless until a plan supplies one), and exemplar **P** was widened from two columns to four — it had reused the correlation exemplar's retrieve verbatim, leaving `analyze_importance` exactly ONE candidate feature, a "ranking" of one. `eval.planner` **31/31**, carried_over 22/22. **Unstructured v2 update 2026-09-23 (`pr/23`, UNSTRUCTURED_V2_PLAN U0–U8):** §4/§6/§7/§8/§9/§13 — pdf/docx/txt ingestion moves from Go to Python (`/api/v2/document*`: doctypes · upload · process · list; **delete stays on Go**): Parsing v2 → figures to the private Supabase bucket → embeddings in the existing `documents` collection tagged `parser_version: v2`. Answers carry `![](asset://<id>)` figure markers, resolved to 1-hour URLs by `POST /api/v2/document/image-token`. Retrieval is now **v2-only and analysis-scoped** (no scope row → no documents), and `structured_flow` shows the planner the bound documents so one plan can combine `retrieve_knowledge` with data. Router untouched. `eval.planner` 32/33, carried_over 22/22. **PR23 part 2 update 2026-09-23 (`pr/23`):** §4/§6/§9 — planner, router and evaluation. **Retrieve-then-plan:** when the analysis has bound documents, `structured_flow` retrieves the top-3 excerpts under cosine distance 0.5 before planning and gives them to the planner and the assembler (planner span Langfuse-masked when present; traceability gains a `retrieve_knowledge` call with `stage: planning` — additive contract note). **Planner prompt:** recipe table + close pairs and the joins rule render per turn only when they apply, compact exemplars, question last — `planner.md` 6,299 → 5,182 tokens, per-call prompt −17%; a section reorder was tried and reverted. **Router:** intents defined once, 21 few-shots (was 32), sharper tool-group definitions. **Correctness:** stable SQL rows past the 10k cap, plan-time check of handoff columns (Check 11), `stats` trimmed in saved records. **Eval:** planner eval gains scoped, database and document cases; router `group_hint` is scored; new `eval/faithfulness`; prompt-cache counters; paid runs now keep full plans, every attempt and a console log. Results: `eval.planner` 39/40 (carried_over 22/22), `eval.intent` 57/57 with `group_hint` 12 exact / 3 superset / 0 wrong, `eval/tools` 35/35. Plan: `docs/plans/PR23_PART2_PLANNER_EVAL.md`. **v2 parser toggle 2026-09-24 (`pr/23`):** §9/§10/§13 — `POST /api/v2/document/process` now parses with **PaddleOCR-VL-1.6** (Baidu AI Studio hosted job API) by default, selected by the server setting `DOCUMENT_PARSER_BACKEND` (`paddleocr` | `mistral`). New `knowledge_parsing/paddle_ocr.py` (submit → poll → JSONL, figures downloaded at parse time) + `normalize_paddle.py` (same `ParsedDocument`). PaddleOCR takes PDF/images only, so a **DOCX fails with a clear message** on that backend. The offline knowledge pipeline is unchanged (default still Mistral). Needs `PADDLEOCR_ACCESS_TOKEN`. Prompted by the Mistral key's account-side zero limit (2026-09-24). **Drift reconcile 2026-09-29 (docs only, verified against the code):** §4/§6/§8/§9/§11/§12/§13/§14 — sections that still described superseded behavior are corrected in place: 6 router intents (not 5); 18 planner-visible tools, none deferred; auto-charts already reach the report; the report floor's 2026-07-24 narrowing; a missing analysis-scope row yields an **empty** catalog on every chat path (~~only the report's data-source appendix still falls back to user-scope~~ — the report appendix too, fixed 2026-09-30); Python reads but never writes `analyses_messages`; the old `src/knowledge/` path serves no RAG; Go migration `0007` does create `message_traceability` + `message_charts`. **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
+**Snapshot date:** 2026-06-25. **Data-layer reconcile 2026-07-01:** §8/§12 updated — dedorch cutover done, `data_catalog` model reconciled. **Query-path fix 2026-07-02:** §8/§13 — dedorch catalogs ship no FKs → Python infers them (`fk_inference.py`); shared-Fernet-key gotcha documented. **Agent-quality fixes 2026-07-08 (pr/13):** from the scoped live-test review — the planner gains an explicit **infeasible** outcome (`TaskList.infeasible_reason` → deterministic EN/ID data-gap reply via `refusals.data_gap_message`; no more force-mapping absent measures like `pa` AS "revenue"), the IR validator rejects bare selects under `group_by` (self-corrects via the planner retry), `analyze_trend` handles integer year/month columns (was collapsing every row into one 1970-01 bucket), planner few-shots add top-N (Example G) + infeasible (Example H), numeric catalog `sample_values` are base64-decoded at read (`catalog/sample_decode.py` — stopgap for Go's byte-marshaling; primary fix is Go-side), traceability no longer emits null source rows for failed retrievals, and `check_data` hides `-1` row counts. **Report v2 + analyze_merge planner support 2026-07-09 (pr/13):** Sofia's `analyze_merge` tool (8abf635, KM-703) is now planner-supported (`_validate_data_source` guards `data_right`, two-retrieve→merge few-shot Example I, planner.md "Two measures per entity" rule); the report gains per-business-question answers (`bq_answers` — drafted by the SAME single LLM call, index-based record refs, deterministic fallback unchanged), "Attempted, Unresolved" + "Excluded Analyses" sections (failed runs are no longer silently dropped), evidence tables copied from `results_snapshot` (table-kind outputs, ≤3/record ≤10 rows ≤8 cols, `check_*` skipped), normalized caveat dedupe with caps (12/10), and single-language output via `detect_reply_language`; the report surface adds `GET /tools/report/{analysis_id}/records` (curation list), `GET …/readiness` (FE delta guard), and `exclude_record_ids` on POST — see API_CONTRACT_BE_PYTHON.md. **Report compaction 2026-07-09 (pr/13):** the rendered markdown drops the "Notes & Limitations", "Attempted, Unresolved", and "How This Was Analyzed" sections (team decision — compact report; render blocks commented out in `report/generator.py`, not deleted). The JSON body keeps `caveats`/`open_questions`/`unresolved`/`method_steps` and the curation/records endpoints are unchanged. **Traceability `data_used` layer 2026-07-13 (pr/15):** `GET /api/v1/traceability` gains a resolved, user-facing `data_used[]` block (one per `retrieve_data` call) — real source/table/column names, joins, plain-language filters, and result columns split into read-from-data vs `computed` (with `formula`, e.g. `total_revenue` = `SUM(line_total)`, so an alias is never shown as a real column). Ids are kept but **machine-only (FE must not render)**. Also adds `tool_calls[].summary`, and `sources[]` now carry `source_name` + every table touched. Deterministic catalog resolution (new `src/traceability/resolve.py`), no LLM, never-throw; catalog threaded to the scratchpad at the slow-path composition root. Additive/non-breaking; contract + `TRACEABILITY_FE_HANDOFF.md` updated. **Spine v2 W2+W1 2026-07-13 (pr/16):** §6/§7/§8/§9/§12 — `render_chart` tool lands (first of the `render_*` family: deterministic Plotly-JSON `dataeyond.chart.v1` envelope, hand-built, **no plotly dependency**; planner-selected only on an explicit chart ask, EN/ID) + Python-owned `message_charts` store + `GET /api/v1/charts` (FE fetches on `done`, same pattern as traceability; empty list = valid 200); planner gains a named **recipe table** + viz few-shots (Example J tail, Example K viz-infeasible) + validator Check 10 (`render_chart.data` must reference a table-producing task); and the slow path gains the **S1a quality checkpoint** (`slow_path/checkpoint.py`, 0 LLM, never-throw, between runner and assembler: CK1 all-failed → deterministic honest-failure answer with **no** assembler call, CK2 empty retrieve + downstream, CK3 10k-cap truncation, CK4 single trend bucket, CK5 all-null column, CK6 chart-spec sanity; flags render as an "Execution assessment" block in the assembler input and every flag logs `repair_candidate` — the S1b evidence base). Design + handoff doc: `SPINE_V2_PLAN.md`. **LLM model + env rename 2026-07-14 (pr/17):** §2/§3/§9/§13 — the generation LLM is now **Azure GPT-5.4-mini** (deployment `gpt-5.4-mini`), not GPT-4o, and the settings quad is renamed `azureai__*__4o` → **`azureai__*__54m`** across all 9 LLM call sites. **Hard rename, no `__4o` fallback** — an environment that still sets only `__4o` resolves to empty strings and fails on the first LLM call, by design: the silent-wrong-model drift is exactly how HF stayed on GPT-4o while local ran 5.4-mini (identical question, identical catalog, divergent planner output — HF hallucinated catalog ids, local planned correctly). Deploying requires all four `__54m` vars set in the HF Space secrets. **`reports.user_id` fix 2026-07-22 (pr/18):** §8/§9/§12/§13 — `ReportStore.save` never wrote `reports.user_id`, and on the new **Neon** dedorch instance that column is `text NOT NULL`, so **every `POST /api/v1/tools/report` 500'd** (`NotNullViolationError`; report persistence deliberately re-raises rather than degrading). The ORM gains `user_id` (declared **nullable** so Python tolerates every deployment shape) and the store writes `report.user_id`, which the endpoint already required as a query param and threaded through the generator. Live-verified against `information_schema` + a green 201. **Root cause (verified against the Go source after pulling `dd37b38`):** the column is *not* new — `0001_create_core_schema.sql` has declared it `NOT NULL` all along, but `0002`/`0004` retrofit it onto pre-existing DBs as a **nullable** `ADD COLUMN IF NOT EXISTS`. The old DB therefore accepted the column-less INSERT and the bug stayed latent; a fresh DB enforces it. Generalized as a §13 gotcha — **Go's migration set is not convergent, so fresh and migrated dedorch DBs differ in NOT NULL constraints** and the migrations alone can't tell you which shape you have. A full ORM-vs-live drift scan the same day found all five Python-written tables clean; the remaining drift is on read-only or unwired paths. **Knowledge pipeline 2026-08-21:** new **§14** — both halves of the knowledge pipeline are now in-tree (`src/knowledge_parsing/` · `src/knowledge_extraction/` · `eval/knowledge/`), as **offline CLIs with no HTTP surface, no table and nothing mounted in `main.py`**. They meet at one versioned `ParsedDocument` artifact; extraction never takes a file path. Term-filter recall **0.8537**, matching the prototype baseline. The existing `src/knowledge/` OCR→pgvector RAG path is untouched. Deps: `rapidfuzz`/`gliner`/`torch` main, `mineru` an optional extra. **Superseded 2026-09-07:** `mineru` dropped (commented out) and `torch` pinned to the Linux CPU wheel — 36 `nvidia-*` packages and `triton` gone from `uv.lock`; §14. **Knowledge 8-Sep task list 2026-09-14 (pr/21):** §8/§14 — the discussion's structure-data, incremental-ingest and review items are built. `knowledge_entries.label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`) so a lookup is a dictionary hit rather than a scan, with `entity_id` untouched as the thing approvals hang on; a re-uploaded document becomes a **new version** rather than a new document, so its approvals survive; an already-approved term is no longer re-extracted (one LLM call per cluster is the whole cost of that branch); an approval goes stale only when the **content the expert ruled on** changed, never when a count moved; a withdrawn document stops feeding the baseline without anything being deleted; and ingest takes a `review_mode` — `llm` approves on arrival with no reviewer step, recorded against the person who chose it. Seven columns added to dedorch (run by hand 2026-09-14 **after** the handoff went to Harry, and verified read-only against live). **Analytical tools foundation 2026-09-17 (`pr/22`):** §3/§6/§9/§10/§13 — three new modules under `src/tools/` (`spec_builder` derives a `ToolSpec` from a pydantic `InputModel`; `ml/executor` offloads blocking sklearn/statsmodels fits to a dedicated bounded pool with a per-fit timeout; `analytics/charts` is the shared chart kit, with `visualization.py` refactored onto it). **Tools stay pure functions — the class-based contract was designed and dropped** (DEV_PLAN §0.9). Analysis tools now ship an **auto-chart** via a reserved `"charts"` key → `ToolOutput.meta["charts"]` → the existing ChartStore save loop; `render_chart` is unchanged. New deps `statsmodels` + an explicit `scikit-learn` pin, and two new env knobs (`ML_MAX_WORKERS`, `ML_FIT_TIMEOUT_SECONDS`). Report embedding of auto-charts is **not** covered yet (§0.9 A21). **Analytical tools — first ML tools 2026-09-17 (`pr/22`):** §6/§9 — the first two ML `analyze_*` tools land, both registered **DEFERRED** (withheld from the planner until A0 scoping locks P1): `analyze_forecast` (`dae102f6` — ETS Holt-Winters, additive trend, **no AutoML**, actual-vs-forecast line chart) and `analyze_anomaly` (`9f3267d5` — IsolationForest, `stats` output, scatter with anomaly points highlighted). Both fit via `run_fit` and ship their chart through the reserved `"charts"` key. `analyze_forecast` is the **first `run_fit` user**: the invoker now `await`s coroutine compute fns (`inspect.isawaitable`, opt-in — the many sync tools are untouched), so an ML tool is `async` solely to offload its fit; conventions §6 ("never async") to be softened to "sync unless it fits a model" — **done 2026-09-16**, the conventions doc now reads "sync unless the tool fits a model". **Per-step progress 2026-09-16 (`pr/22`, A22):** §4/§9 — the slow path reports each plan step as it starts instead of three stage pings for a ~12s turn (past ~10s a waiting user needs evidence of progress, not a faster answer). New `slow_path/progress.py` holds a bounded **bilingual** activity vocabulary; `TaskRunner.run` gains an optional never-throw `progress` callback; the three coordinator stage pings became bilingual (they were English-only on a majority-Indonesian product). A 4-step plan emits ~6 `status` events instead of 3. **Wire contract unchanged** — `status` is still optional text the FE may ignore; contract §chat notes the new volume. **Report auto-chart embedding 2026-09-16 (`pr/22`, A21):** §5/§9 — the downloaded report now embeds an `analyze_*` tool’s auto-chart, not just `render_chart` output. Chart extraction moved to one shared definition (`src/charts/extract.py::chart_envelopes`) used by BOTH the per-turn persistence path and the report generator, so a chart can no longer appear in chat and vanish from the report. Envelopes must carry a `plotly` object to be emitted (the report path already required it; persistence now does too). No schema change, no contract change. **Analytical tools — importance + cluster, unregistered 2026-09-16 (`pr/22`):** §6/§9 — `analyze_importance` (standardised Ridge/Logistic coefficients with a leakage-safe train-only scaler and a holdout score; NOT permutation importance, which is the latency tall pole and buys nothing for a linear model) and `analyze_cluster` (KMeans, k chosen by silhouette over 2..8, scatter on two real columns rather than PCA components). **Both files are inert** — deliberately not in `_DISPATCH` or either registry list, pending the `registry.py`/`invoker.py` ownership call, so they cost nothing until registered. Both emit the new reserved **`quality`** block `{metric, value, floor, weak}`; the checkpoint gains **CK7**, one generic rule that flags any output whose tool declared itself weak — the tool owns the verdict, so a new ML tool is covered with no checkpoint change. **Model-quality checks complete 2026-09-17 (`pr/22`, A20):** §9 — four tools now publish the reserved `quality` block and CK7 surfaces it: importance (r2/accuracy), cluster (silhouette), **forecast (MASE from a new holdout backtest — previously it reported no accuracy at all)** and ab_test (p-value). Pareto, segmentation, data_quality and anomaly deliberately do NOT emit one — arithmetic results, a circular self-report, and an unsupervised flag rate that would mark ~every default run weak. CK7's message was direction-wrong for MASE and p-value ("at or below its floor" when high is bad) and now states the value/floor pair and defers to the tool's note. **Group-first tool selection 2026-09-17 (`pr/22`, A17):** §4/§9 — A0 activated every analytics tool, putting the planner-visible set at **18**, inside the 15–20 band where tool-selection accuracy degrades. Rather than withhold tools, the router now emits a **`group_hint`** on its existing structured-output call and a deterministic filter (`src/agents/planner/groups.py`, 0 LLM) shows the planner only that group's tools plus the always-on core — **8–9 instead of 18**, 11 for a two-group question. Groups are derived from `ToolSpec.category` rather than stored on it, so the tool layer is untouched. **Fail-open by design:** the router is catalog-blind (it runs before the catalog read), so an absent, empty or unknown hint yields the FULL registry — a filtered-out tool is one the planner cannot ask for. The two-LLM-call slow-path invariant is unchanged. **All tools active + two suppression fixes 2026-09-17 (`pr/22`, A15/A24/A25):** §9 — A15 moved all 8 remaining specs to ACTIVE (`DEFERRED` is now empty, 18 planner-visible unscoped). Two things would have kept the new ML tools unusable and are fixed: `prompt.py` injected **`modeling_allowed: False (no modeling tools exist in v1…)`** on every planner call — contradicting the rewritten `planner.md` rule 5 — so the default is now True and the text follows the flag; and `render_examples()` ignored group scoping, showing a scoped turn examples for tools it could not call, so it now takes the visible tool set. Both infeasible examples survive every scope (an infeasible plan calls no tools), so the refusal discipline is never scoped away. **Tool honesty + repair telemetry 2026-09-21 (`pr/22`, A32–A34):** §9 — three tools stopped claiming more than they computed, and the checkpoint started measuring. `analyze_data_quality` advertised duplicate detection in `use_when` and computed none; it now returns `duplicate_rows {count, rate}` + `issues.duplicate_row_count`, always present (0 when none). `analyze_anomaly` could not **name** what it flagged — a record carried only the scored numeric columns plus a score, so "which units are unusual?" was structurally unanswerable; it now takes optional **`label_columns`** carried into each record (identifiers first, never scored) and publishes a `quality` block (`anomaly_rate`, weak only **above 0.5**). That ceiling was **re-measured, not re-assumed** — the A20 revert stands for a naive low ceiling, but clean gaussian noise flags 0.14–0.22 (p95 0.22) and a genuine 30%-outlier frame flags 0.30, so 0.5 clears both while still catching the demo's degenerate 0.86. Planner side: recipe **R14** and exemplar **U** now retrieve an identifier and pass it as `label_columns` (the tool-side capability is useless until a plan supplies one), and exemplar **P** was widened from two columns to four — it had reused the correlation exemplar's retrieve verbatim, leaving `analyze_importance` exactly ONE candidate feature, a "ranking" of one. `eval.planner` **31/31**, carried_over 22/22. **Unstructured v2 update 2026-09-23 (`pr/23`, UNSTRUCTURED_V2_PLAN U0–U8):** §4/§6/§7/§8/§9/§13 — pdf/docx/txt ingestion moves from Go to Python (`/api/v2/document*`: doctypes · upload · process · list; **delete stays on Go**): Parsing v2 → figures to the private Supabase bucket → embeddings in the existing `documents` collection tagged `parser_version: v2`. Answers carry `![](asset://<id>)` figure markers, resolved to 1-hour URLs by `POST /api/v2/document/image-token`. Retrieval is now **v2-only and analysis-scoped** (no scope row → no documents), and `structured_flow` shows the planner the bound documents so one plan can combine `retrieve_knowledge` with data. Router untouched. `eval.planner` 32/33, carried_over 22/22. **PR23 part 2 update 2026-09-23 (`pr/23`):** §4/§6/§9 — planner, router and evaluation. **Retrieve-then-plan:** when the analysis has bound documents, `structured_flow` retrieves the top-3 excerpts under cosine distance 0.5 before planning and gives them to the planner and the assembler (planner span Langfuse-masked when present; traceability gains a `retrieve_knowledge` call with `stage: planning` — additive contract note). **Planner prompt:** recipe table + close pairs and the joins rule render per turn only when they apply, compact exemplars, question last — `planner.md` 6,299 → 5,182 tokens, per-call prompt −17%; a section reorder was tried and reverted. **Router:** intents defined once, 21 few-shots (was 32), sharper tool-group definitions. **Correctness:** stable SQL rows past the 10k cap, plan-time check of handoff columns (Check 11), `stats` trimmed in saved records. **Eval:** planner eval gains scoped, database and document cases; router `group_hint` is scored; new `eval/faithfulness`; prompt-cache counters; paid runs now keep full plans, every attempt and a console log. Results: `eval.planner` 39/40 (carried_over 22/22), `eval.intent` 57/57 with `group_hint` 12 exact / 3 superset / 0 wrong, `eval/tools` 35/35. Plan: `docs/plans/PR23_PART2_PLANNER_EVAL.md`. **v2 parser toggle 2026-09-24 (`pr/23`):** §9/§10/§13 — `POST /api/v2/document/process` now parses with **PaddleOCR-VL-1.6** (Baidu AI Studio hosted job API) by default, selected by the server setting `DOCUMENT_PARSER_BACKEND` (`paddleocr` | `mistral`). New `knowledge_parsing/paddle_ocr.py` (submit → poll → JSONL, figures downloaded at parse time) + `normalize_paddle.py` (same `ParsedDocument`). PaddleOCR takes PDF/images only, so a **DOCX fails with a clear message** on that backend. The offline knowledge pipeline is unchanged (default still Mistral). Needs `PADDLEOCR_ACCESS_TOKEN`. Prompted by the Mistral key's account-side zero limit (2026-09-24). **Drift reconcile 2026-09-29 (docs only, verified against the code):** §4/§6/§8/§9/§11/§12/§13/§14 — sections that still described superseded behavior are corrected in place: 6 router intents (not 5); 18 planner-visible tools, none deferred; auto-charts already reach the report; the report floor's 2026-07-24 narrowing; a missing analysis-scope row yields an **empty** catalog on every chat path (~~only the report's data-source appendix still falls back to user-scope~~ — the report appendix too, fixed 2026-09-30); Python reads but never writes `analyses_messages`; the old `src/knowledge/` path serves no RAG; Go migration `0007` does create `message_traceability` + `message_charts`. **Go/FE sync + data dictionary 2026-09-30:** §8/§9/§12/§14 — re-read Go at `887d98a` and FE at `f797178`: Go `0008` adopts the five knowledge tables; Go KM-717 adds a per-column **data dictionary** and per-binding **table selection** to the catalog; Python now reads the dictionary into the planner's catalog summary (no change when absent); the FE sends pdf/docx/txt upload·process·list·doctypes·image-token to Python's `/api/v2/document*` (delete stays on Go). **Cross-repo update 2026-06-29:** §2/§8/§11/§12 re-verified against
 the **Go source** (`Orchestrator-Agent-Service`), not its docs. The Go service has moved well past its
 own (uncommitted, stale) design docs: it now hosts the **dedorch SQL migrations** in-repo and a full
 **`/api/v1/analyses` + `/api/v1/skills`** REST surface. Go does **not** call Python yet — those skills
@@ -201,7 +201,7 @@ unless `SKIP_INIT_DB=true`.
 |---|---|---|---|
 | `users`, `rooms`, `chat_messages`, `message_sources` | base app | chat endpoint, Go | chat history |
 | `documents`, `databases` | uploads + DB creds (Fernet-encrypted) | Go ingestion; **since 2026-09-23 Python writes `documents` rows for pdf/docx/txt** (v2 upload/process, U-D1 — Go's shape and status vocabulary `uploading·uploaded·processing·completed·failed`; ORM gained nullable `chunks_count`, column already live) | executor cred resolution; v2 document routes |
-| `data_catalog` *(dedorch, Go-owned)* | `id` uuid, `scope_type` ('user'\|'analysis'), `user_id`, `analysis_id`, **`catalog_payload`** jsonb (the `Catalog`: Source → Table → Column), schema_version, generated_at, updated_at; partial-unique on `user_id WHERE scope_type='user'` | **Go `catalog.Service`** (all writes: DB/file ingestion) | CatalogReader → CatalogStore (**read-only**), planner, tools |
+| `data_catalog` *(dedorch, Go-owned)* | `id` uuid, `scope_type` ('user'\|'analysis'), `user_id`, `analysis_id`, **`catalog_payload`** jsonb (the `Catalog`: Source → Table → Column), schema_version, generated_at, updated_at; partial-unique on `user_id WHERE scope_type='user'`. **2026-09-30:** columns may carry a user-entered **data dictionary** (KM-717) and analysis bindings may select specific tables — see the catalog-shape note below | **Go `catalog.Service`** (all writes: DB/file ingestion) | CatalogReader → CatalogStore (**read-only**), planner, tools |
 | `langchain_pg_embedding` | PGVector document chunks (collection `documents`) | Go ingestion (v1, Tesseract); **since 2026-09-23 Python v2 ingestion** — Go's `cmetadata` shape plus `parser_version: "v2"`, `chunk_id`, `assets[{asset_id, storage_key}]` | DocumentRetriever — **v2 rows only, analysis-scoped** (2026-09-23) |
 | `report_inputs` *(was `analysis_records`)* | jsonb `AnalysisRecord`, one per slow-path run; **Python-owned** | slow path | ReportGenerator, report readiness |
 | `analyses` *(dedorch, plural)* | uuid `id`, `user_id`, `analysis_title`, `objective`, `business_questions` jsonb, `status` (active\|inactive), `data_bind`(+`data_bind_version`), `report_id`, `report_collection` — **defined by Go migrations**; `problem_statement`/`problem_validated`/`owner_id` already **dropped** there (`0003`/`0004`) | Go `/api/v1/analyses`; Python state store | gate (no-op), Help, report |
@@ -210,7 +210,7 @@ unless `SKIP_INIT_DB=true`.
 | `analyses_messages` *(dedorch)* | the analysis chat room (`role ∈ user\|ai`); replaces deprecated `rooms`/`chat_messages` | Go `/analyses/{id}/messages` — **Go only**; Python never writes it (DEV_PLAN #25) | ~~Python chat path **not yet migrated here** (§12)~~ **Corrected 2026-09-29:** Python chat + `/tools/help` read turn history here (`load_history`, latest 10, oldest-first; v2 chat scopes it by `user_id`) |
 | `message_traceability` *(Python-owned)* | one jsonb `TraceabilityPayload` per assistant turn (PK `message_id`); flushed before `done`. **2026-09-29:** created by Go migration `0007` (with `message_charts`) — see §12 | chat pipeline (KM-691) | `GET /api/v1/traceability` |
 | `message_charts` *(Python-owned, added 2026-07-13)* | one row per `render_chart` chart — `spec` jsonb holds the full `dataeyond.chart.v1` envelope; keyed (`analysis_id`, `message_id`), multiple rows per turn allowed; written before `done`, never-throw | slow-path chart persist (`chat_handler._run_slow_path`) | `GET /api/v1/charts` |
-| `knowledge_documents` · `knowledge_extraction_runs` · `knowledge_entries` · `knowledge_reviews` · `knowledge_jobs` *(Python-written, Go-owned schema)* | The knowledge pipeline's five tables (§14). **2026-09-14: 7 columns added** — `knowledge_entries.doc_ids`/`.content`/`.provenance` (promoted out of `payload`), `knowledge_documents.withdrawn_at`, `knowledge_jobs.review_mode`, `knowledge_reviews.decided_by`/`.authorised_by`. `label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`), not the term. DDL + adoption status: `docs/knowledge/knowledge_tables.sql` | knowledge ingest pipeline | `/api/v1/knowledge/*`, `KnowledgeService` |
+| `knowledge_documents` · `knowledge_extraction_runs` · `knowledge_entries` · `knowledge_reviews` · `knowledge_jobs` *(Python-written, Go-owned schema)* | The knowledge pipeline's five tables (§14). **2026-09-14: 7 columns added** — `knowledge_entries.doc_ids`/`.content`/`.provenance` (promoted out of `payload`), `knowledge_documents.withdrawn_at`, `knowledge_jobs.review_mode`, `knowledge_reviews.decided_by`/`.authorised_by`. `label` now carries a readable `TYPE_SUBJECT` key (`GLOSSARY_PA`), not the term. DDL + adoption status: `docs/knowledge/knowledge_tables.sql`. **2026-09-30:** created by Go migration `0008` (`66e70ca`) — see §14 | knowledge ingest pipeline | `/api/v1/knowledge/*`, `KnowledgeService` |
 
 > ✅ **Python ORM ↔ dedorch drift — reconciled 2026-07-01.** `AnalysisStateRow` (`analyses`) dropped
 > `problem_statement`/`problem_validated` and added `objective`/`business_questions` (Harry's #3);
@@ -225,6 +225,19 @@ unless `SKIP_INIT_DB=true`.
 **Catalog shape** (the jsonb in `data_catalog`):
 `Catalog → Source[ {source_id, source_type ∈ schema|tabular|unstructured, name, location_ref} → Table[ {table_id, name, row_count, foreign_keys[]} → Column[ {column_id, name, data_type, nullable, pii_flag, sample_values|null, stats} ] ] ]`. PII columns have `sample_values: null` so real values never enter prompts.
 
+> **Data dictionary + table selection (Go KM-717 `887d98a`, 2026-09-29; Python read side 2026-09-30).**
+> A `Column` may now carry user-entered, omit-when-empty fields: `description`, `business_meaning`,
+> `unit_of_measure`, `classify` (`dimension`|`measure`), `aggregation` (`sum|avg|min|max|count`),
+> `is_key`, `data_classification`, `validation_rules`. Go saves them on the **user-scope** row via
+> `POST /api/v1/data-catalog/{user_id}/sources/{source_id}/dictionary` and preserves them across
+> re-introspection; analysis-scope rows are copied from user-scope **at build time**, so an analysis
+> bound before a dictionary edit sees it only after `POST /api/v1/analyses/{id}/data-catalog/rebuild`
+> (or a re-bind). Python's `catalog/models.py::Column` declares all eight (optional), and the planner
+> sees a `dictionary:` line under each column that has one (§9 Planner). Separately, a binding item may
+> list `tables` — the analysis-scope row then holds **only those tables**, so a join target left out
+> of the selection cannot be inferred (§8 FK gotcha); a database binding whose source is absent from
+> the user-scope catalog is dropped from the analysis catalog without an error.
+
 > ⚠️ **dedorch catalogs ship empty `foreign_keys`** (Go's introspection drops FK constraints), yet the IR validator only allows FK-backed joins — so every cross-table question failed validation until 2026-07-02. `src/catalog/fk_inference.py` (wired into `CatalogStore.get`) now infers the obvious `<base>_id → <table>.id` edges at read time: conservative (single unambiguous target, matching `data_type`, schema sources only) and **self-disabling** once any real FK is present. It's a **stopgap** — the durable fix is Go emitting real FKs during introspection.
 
 **QueryIR shape** (`src/query/ir/models.py`):
@@ -291,6 +304,16 @@ rest are labelled 2/3/4/6/7 — plus sub-checks 8a–8h and Checks 9, 10, 11 in
   **Check 11 (2026-09-23, D-5):** a value-handoff `${t<id>.<column>}` must name a column `t<id>`
   outputs (exact case) — an unknown one used to resolve to an empty list, so `not_in` matched
   everything. Now rejected at plan time and retried.
+  **Data dictionary in the catalog summary (2026-09-30, KM-717 read side):** `CatalogSummary`
+  (`planner/inputs.py`) renders a column's dictionary entry as one extra line under it —
+  `dictionary: meaning: … | unit: … | role: measure | aggregate with: avg | key column` — capped at
+  300 chars, inside the catalog's existing `<data>` block (hard rule 8: data, never instructions).
+  Shown for PII columns too (it is text about the column, not a value from it);
+  `data_classification`/`validation_rules` are not rendered. **Emitted only when an entry exists**,
+  so a catalog without one renders byte-for-byte as before — verified on the `eval.planner` fixture
+  catalogs (5,109 bytes, identical), which is why no paid planner eval was re-run. `planner.md` is
+  unchanged. Whether the planner *uses* an entry (e.g. `aggregate with: avg` on PA) is unmeasured
+  until a dictionary case is added to `eval.planner`.
 - **TaskRunner** (`slow_path/task_runner.py`) — deterministic, 0 LLM. Wave-based execution,
   `${t<id>}` placeholder resolution (Pattern A), never-throw invocation, **degrade-and-continue**
   (failed task → dependents skipped, independent branches run). No replanning.
@@ -637,8 +660,10 @@ Python is consumer-only). State **re-verified against the Go source 2026-06-29**
   **RESOLVED 2026-07-22:** `report_inputs` ships in Go's `0001` (above), so a fresh dedorch DB has it.
   ~~The same is **not** true of `message_traceability`/`message_charts` — those still need the manual DDL
   on any newly provisioned instance (DEV_PLAN #32).~~ **Corrected 2026-09-29:** they ship in Go's `0007`
-  (above), so a newly provisioned instance gets them too. The five knowledge tables are still in no
-  Go migration (§14).
+  (above), so a newly provisioned instance gets them too. ~~The five knowledge tables are still in no
+  Go migration (§14).~~ **Corrected 2026-09-30:** they are — Go `0008_create_knowledge_pipeline_schema.sql`
+  (`66e70ca`, 2026-09-22) creates all five (§14). That line was written 2026-09-29 against a stale
+  local Go checkout (`ffc2aa9`, 2026-07-27).
 
 **⚠️ Integration gap (verified — the big one).** Go's `/api/v1/analyses` and `/api/v1/skills`
 (`help` / `report`) are **placeholders that return dummy data** — the `SendMessage` / `GenerateReport`
@@ -761,6 +786,18 @@ tables. This is the same hand-over the 2026-07-13 `message_charts` run skipped,
 `message_traceability` and `message_charts` are still missing from Go's migrations today. Python
 executes no DDL: `SKIP_INIT_DB` stays true, no `create_all`, no Alembic.
 
+> **✅ Adopted 2026-09-22 (recorded 2026-09-30, read in the Go source at `887d98a`).** Go migration
+> `0008_create_knowledge_pipeline_schema.sql` (`66e70ca`) creates all five tables and their indexes
+> from the **current** `knowledge_tables.sql` — including the post-2026-09-07 drift (the `'document'`
+> kind, `knowledge_entries_parent_check`, nullable `run_id`/`knowledge_document_id`) and the §6/§7
+> columns (`doc_ids`, `content`, `provenance`, `withdrawn_at`, `review_mode`, `decided_by`,
+> `authorised_by`). It also adds `knowledge_documents.document_id` (FK → `documents(id)`) and parser
+> columns, which the ORM already declares. A fresh environment now gets the tables. (`message_traceability`
+> / `message_charts` were likewise adopted by `0007` — §12 — so the sentence above is historical.)
+> ⚠️ **Not verified:** the live Neon tables predate `0008`, so its `CREATE TABLE IF NOT EXISTS` no-ops
+> there (only its `ADD COLUMN IF NOT EXISTS` steps apply) — confirm the live shape with
+> `information_schema`, as was done for #32.
+
 **Ingestion is asynchronous and admin-frequency.** A batch is one click over all the caller's
 parseable documents; parse (~80 s) plus extract (~170 s) for nine pages is far past any HTTP
 timeout, so the route returns `202 {job_id}` and the caller polls. One job runs per company at a

docs/README.md

@@ -42,7 +42,7 @@ Client documents → candidate knowledge entries an expert reviews rather than a
 | [knowledge/TABLE_BRANCH_PROPOSAL.md](knowledge/TABLE_BRANCH_PROPOSAL.md) | 💡 | Step 8 — the symptom x cause matrix a shop manual actually holds. Deterministic triples, LLM only to phrase. **Proposal, nothing built** |
 | [knowledge/TERM_HIERARCHY_PROPOSAL.md](knowledge/TERM_HIERARCHY_PROPOSAL.md) | 💡 | Step 7 + R7 — parent/child terms and cross-document identity, designed together. **Touches `entity_id`; read before building** |
 | [knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V4.md](knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V4.md) | 💡 | What a *real* domain context is. v3's object is a document brief wearing the name. **Proposal, nothing built** |
-| [knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md](knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md) | 💡 | The four-table DDL sent to Harry 2026-09-01. **Pending — no ORM written against it yet** |
+| [knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md](knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md) | ✅ | The DDL handed to Harry (sent 2026-09-01, five tables since). ~~**Pending — no ORM written against it yet**~~ **Adopted by Go migration `0008` (2026-09-22; recorded 2026-09-30)**, ORM live since 2026-09-07 |
 | [knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V3.md](knowledge/KNOWLEDGE_DOMAIN_CONTEXT_V3.md) | ✅ | `DomainContext` design reasoning. **Shipped in full 2026-09-02** — the live shape is the output contract §8 |
 
 Roll-up status: DEV_PLAN §0.8 · built state: REPO_STATUS §14 · eval harness: [../eval/knowledge/README.md](../eval/knowledge/README.md)

docs/knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md

@@ -2,10 +2,13 @@
 
 <!-- doc-status -->
 > ✅ **LIVE — the five tables exist in dedorch as of 2026-09-07.** Agreed by Rifqi and created by a
-> one-time manual DDL run, exactly as [knowledge_tables.sql](knowledge_tables.sql). **Still owed:
+> one-time manual DDL run, exactly as [knowledge_tables.sql](knowledge_tables.sql). ~~**Still owed:
 > Harry folds that SQL into Go's migration set** — until he does, the tables exist only because they
 > were run by hand, and a fresh environment comes up without them. That is the same gap that still
-> leaves `message_traceability` and `message_charts` out of the migrations (§7).
+> leaves `message_traceability` and `message_charts` out of the migrations (§7).~~ **✅ Adopted
+> 2026-09-22 (recorded 2026-09-30):** Go migration `0008_create_knowledge_pipeline_schema.sql`
+> (`66e70ca`) creates all five tables from the current file, and `0007` had already adopted
+> `message_traceability`/`message_charts`. Remaining: confirm the hand-created live tables match `0008`.
 >
 > **Updated 2026-09-16 — §2 now matches the live schema.** Seven columns were added on
 > 2026-09-14 and this document did not know about them; it described the 2026-09-07 shape and

docs/knowledge/knowledge_tables.sql

@@ -1,5 +1,12 @@
 -- Knowledge pipeline tables — dedorch
 --
+-- ✅ ADOPTED 2026-09-22 (recorded 2026-09-30). Go migration
+-- `0008_create_knowledge_pipeline_schema.sql` (Orchestrator-Agent-Service
+-- `66e70ca`) reproduces this file — §1-§5 including the drift below, and the
+-- §6/§7 columns — so a fresh environment now gets these tables from Go. This
+-- file stays as the record of what was run by hand; Go's migration is now the
+-- schema of record. Still open: confirm the hand-created live tables match 0008.
+--
 -- HANDOFF TO HARRY. Updated 2026-09-14. Source of record:
 -- docs/knowledge/KNOWLEDGE_PERSISTENCE_CONTRACT.md §2.
 --

src/agents/planner/inputs.py

@@ -80,6 +80,59 @@ class ColumnSummary(BaseModel):
     #: which is the other half of that fix.
     value_min: Any | None = None
     value_max: Any | None = None
+    #: The column's data-dictionary entry (KM-717), pre-rendered to one line, or
+    #: None when the user wrote none. This is what lets the planner use the
+    #: company's own definition of a column ("PA = Physical Availability, %,
+    #: average it") instead of guessing from its name.
+    #:
+    #: Emitted for PII columns too: it is text the user wrote ABOUT the column, not a
+    #: value out of it. It renders inside the catalog's `<data>` block, which planner
+    #: hard rule 8 already treats as data, never as instructions.
+    dictionary: str | None = None
+
+
+#: Cap on one column's rendered dictionary line. Descriptions are free text; this
+#: keeps one long paragraph from eating the catalog budget (`_MAX_CATALOG_CHARS`).
+_MAX_DICTIONARY_CHARS = 300
+
+
+def _dictionary_line(col: Any) -> str | None:
+    """One compact line from a column's data-dictionary fields, or None if empty.
+
+    Only fields that change a planning decision are shown: what the column means,
+    its unit, whether it is a measure or a dimension, how to aggregate it, and
+    whether it is a key. `data_classification` and `validation_rules` stay out —
+    governance labels and rule objects, not planning inputs.
+    """
+    def _clean(value: Any) -> str:
+        return " ".join(str(value).split()) if value is not None else ""
+
+    parts: list[str] = []
+    meaning = "; ".join(
+        dict.fromkeys(  # dedupe when description == business_meaning
+            t for t in (_clean(getattr(col, "description", None)),
+                        _clean(getattr(col, "business_meaning", None))) if t
+        )
+    )
+    if meaning:
+        parts.append(f"meaning: {meaning}")
+    unit = _clean(getattr(col, "unit_of_measure", None))
+    if unit:
+        parts.append(f"unit: {unit}")
+    role = _clean(getattr(col, "classify", None))
+    if role:
+        parts.append(f"role: {role}")
+    agg = _clean(getattr(col, "aggregation", None))
+    if agg:
+        parts.append(f"aggregate with: {agg}")
+    if getattr(col, "is_key", False):
+        parts.append("key column")
+    if not parts:
+        return None
+    line = " | ".join(parts)
+    if len(line) > _MAX_DICTIONARY_CHARS:
+        line = line[: _MAX_DICTIONARY_CHARS - 1].rstrip() + "…"
+    return line
 
 
 class ForeignKeySummary(BaseModel):
@@ -163,6 +216,7 @@ class CatalogSummary(BaseModel):
                                 if col.pii_flag or col.stats is None
                                 else col.stats.max
                             ),
+                            dictionary=_dictionary_line(col),
                         )
                         for col in table.columns
                     ],
@@ -261,6 +315,11 @@ class CatalogSummary(BaseModel):
                         f"    - {col.name} [{col.data_type}]: "
                         f"samples={samples}{top}{rng}{const} — id={col.column_id}"
                     )
+                    # Data dictionary (KM-717): a second line, emitted only when the
+                    # user wrote an entry, so a catalog without one renders exactly as
+                    # before (no planner-prompt change for existing analyses).
+                    if col.dictionary:
+                        lines.append(f"      dictionary: {col.dictionary}")
                 for fk in table.foreign_keys:
                     tgt_table = table_name_by_id.get(fk.target_table_id, fk.target_table_id)
                     tgt_col = col_name_by_id.get(fk.target_column_id, fk.target_column_id)

src/catalog/models.py

@@ -48,6 +48,21 @@ class Column(BaseModel):
     pii_flag: bool = False
     sample_values: list[Any] | None = None
     stats: ColumnStats | None = None
+    # Data dictionary (KM-717, Go `887d98a`, 2026-09-29): user-entered column
+    # metadata, written by Go's `POST /api/v1/data-catalog/{user_id}/sources/
+    # {source_id}/dictionary` and carried into analysis-scope catalogs. All optional
+    # and omitted by Go when empty, so pre-dictionary catalogs parse unchanged.
+    # `classify` is "dimension" | "measure"; `aggregation` is sum|avg|min|max|count
+    # (FE `DataDictionaryForm.tsx`). Free text — rendered only inside the planner's
+    # `<data>` block.
+    description: str | None = None
+    business_meaning: str | None = None
+    unit_of_measure: str | None = None
+    classify: str | None = None
+    aggregation: str | None = None
+    is_key: bool = False
+    data_classification: str | None = None
+    validation_rules: Any | None = None
 
 
 class ForeignKey(BaseModel):