# Backend Agentic Service API Contract

<!-- doc-status -->
> ✅ **LIVE — CANONICAL: the contract of record** for the FE-facing surface. This is the one that governs endpoint behaviour. The frozen 2026-06-30 snapshot sent to Harry is [docs/archive/API_ENDPOINTS_RESTRUCTURE.md](docs/archive/API_ENDPOINTS_RESTRUCTURE.md) — do not implement from that one.

This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and traceability data shown alongside chat answers.

Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.

## Overview

The Python backend owns the generative AI interaction surface:

1. Stream chat answers from the AI agent.
2. Execute tool-style actions for help and report generation.
3. Return report versions and report details.
4. Return traceability/provenance for a completed assistant answer.

The frontend uses this service during the analysis conversation flow:

1. User sends a chat message.
2. Frontend calls `POST /api/v2/chat/stream` and renders the streamed answer.
3. When the stream emits `done`, frontend stores or reads the returned `message_id`.
4. Frontend calls `GET /api/v1/traceability` for planning, tool calls, and source provenance.
5. Frontend calls `/api/v1/tools/help` for guided help and `/api/v1/tools/report` for report generation.

## Endpoint Summary

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/v2/chat/stream` | Stream an AI chat answer for one analysis conversation. |
| `GET` | `/api/v1/tools/list` | List available frontend tools. |
| `POST` | `/api/v1/tools/help` | Stream contextual help for the current analysis conversation. |
| `POST` | `/api/v1/tools/report` | Generate and persist a new report version. |
| `GET` | `/api/v1/tools/report/{analysis_id}` | List report versions for an analysis. |
| `GET` | `/api/v1/tools/report/{analysis_id}/records` | List analysis records for report curation (added 2026-07-09). |
| `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). |
| `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
| `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
| `GET` | `/api/v1/charts` | Retrieve chart(s) produced by `render_chart` for one assistant answer (added 2026-07-13). |
| `POST` | `/api/v1/knowledge/ingest` | Extract knowledge from a user's documents — async batch (added 2026-09-07). |
| `GET` | `/api/v1/knowledge/jobs/{job_id}` | Poll one ingestion job (added 2026-09-07). |
| `GET` | `/api/v1/knowledge/jobs` | Recent ingestion jobs for a user (added 2026-09-07). |
| `GET` | `/api/v1/knowledge/documents` | Parsed artifacts stored for the caller's company (added 2026-09-07). |
| `POST` | `/api/v1/knowledge/documents/{id}/extract` | Re-extract a stored artifact, no re-parsing (added 2026-09-07). |
| `GET` | `/api/v1/knowledge/entries` | Candidate knowledge entries (added 2026-09-07). |
| `GET` | `/api/v1/knowledge/queue` | The expert review queue (added 2026-09-07). |
| `POST` | `/api/v1/knowledge/entries/{entity_id}/reviews` | Record an expert decision (added 2026-09-07). |

## Authentication

**None. The live surface is unauthenticated — do not expose it beyond the demo.**

The service accepts `user_id` and `analysis_id` as ordinary request fields with no verification, and there is no caller authentication in front of any endpoint.

**History (2026-07-23 → 2026-07-27).** A shared service-secret gate (`X-Dataeyond-Service-Secret`, router-level, constant-time compare) shipped 2026-07-23, inert until `dataeyond__service__secret` was set. It was **unwired on 2026-07-27** (DEV_PLAN #37): the only caller of this service is the browser SPA, which cannot be changed to send the header, so the gate could never be armed without a 401 outage. The gate code is parked in `src/middlewares/service_auth.py` (not deleted) and restores in one edit if the caller situation changes.

**Updated 2026-09-07 — the `/knowledge` group raises the stakes.** Those endpoints are *writes* that spend LLM budget, and one of them (`POST …/reviews`) accepts content that reaches a later prompt. They are bounded by a rate limit, one job per company, a pre-flight spend ceiling and content-addressed idempotency — none of which identifies the caller. The posture is recorded rather than inherited: see the Knowledge section.

**The real fix (DEV_PLAN #43).** Per-user authorization requires a verified identity forwarded by the Go service; once Go forwards a token, `user_id` comes from the verified claims rather than the request body. Until then the `user_id` predicates in the stores (F-1) are defensive-in-depth only, not an access control.

## Common Concepts

### Identifiers

- `user_id`: user identifier passed by the frontend.
- `analysis_id`: analysis conversation identifier.
- `message_id`: assistant answer identifier used to correlate chat streaming with traceability. **Server-minted, never accepted from the client.** **Updated 2026-07-09:** it is a UUID string (e.g. `77f06761-0fdf-4cc5-84f8-5f81bcbb6f84`), matching the shape of Go's `analyses_messages.id`. The `msg_…` values in the examples below are illustrative placeholders only.

### Server-Sent Events

Chat and help endpoints return `text/event-stream`.

Frontend should parse events by `event` name and `data` payload. Blank lines separate SSE events.

Common event types:

| Event | Data | Meaning |
| --- | --- | --- |
| `sources` | JSON array | Always `[]` — sources moved to `GET /api/v1/traceability` (KM-691). Event kept for backward-compat; read `sources[]` from the traceability call. |
| `status` | text | Optional progress update for slower paths. |
| `chunk` | text | Answer text fragment. Concatenate chunks in order. |
| `done` | JSON object | Terminal success event. Includes `message_id`. |
| `error` | text | Terminal error event. Stream stops after this. |

The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.

**Charts (added 2026-07-13, SPINE_V2_PLAN §4.5):** the `done` event is unchanged by charts — no additive `chart_count`/`chart_ids` field yet (open question owned by Harry). Until that's resolved, the frontend fetches `GET /api/v1/charts` unconditionally on every `done`, the same fetch-on-`done` pattern already used for traceability. SSE order and every existing field stay exactly as documented above; `sources` stays `[]`.

## Chat

### `POST /api/v2/chat/stream`

Streams an AI answer for one user message in an analysis conversation.

Request body:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}
```

Fields:

| Field | Required | Description |
| --- | --- | --- |
| `user_id` | Yes | User identifier. |
| `analysis_id` | Yes | Analysis conversation identifier. |
| ~~`message_id`~~ | — | **Updated 2026-07-09:** not a request field. Python always mints the id server-side and returns it on `done`; any caller-sent value is ignored (server-authoritative — open-Q #1). |
| `message` | Yes | User message text. |

Response: `text/event-stream`.

Example structured answer:

```text
event: sources
data: []

event: status
data: Planning analysis...

event: status
data: Running 3 steps...

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}
```

Example simple chat answer:

```text
event: sources
data: []

event: chunk
data: I'm your AI data analyst. Connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}
```

Behavior notes:

- Greeting and farewell messages may use a fast canned path.
- Stateless `chat` intent may use a 1-hour Redis response cache.
- The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`.
- `sources` in the stream is **always `[]`** (KM-691) — read the real `sources[]` from `GET /api/v1/traceability` after `done`.
- `status` events are optional and should be safe for the frontend to ignore.
- **Fixed 2026-07-23 — `sources` is now genuinely always first, on every path.** It previously did not match this document: the `check` and router-`help` turns emitted **no `sources` event at all**, and the `structured_flow` (slow) path emitted `status` *before* `sources`, inverting the documented order on exactly the turns that take longest. A frontend that initializes per-turn state on `sources` therefore never initialized on check/help turns and initialized late on slow turns. The change is **purely additive** — no event was removed or renamed, and `sources` is still `[]` — so no frontend change is required; behaviour simply now matches what this contract always said.
- **`analysis_id` must be a UUID (2026-07-23).** `POST /api/v2/chat/stream` and `POST /api/v1/tools/help` now return **`422`** for a non-UUID `analysis_id` instead of accepting it. It was never actually usable: `analyses.id` is `uuid NOT NULL`, so a non-UUID id silently matched no analysis and the turn ran with no state — help, readiness, the report write-back, and the traceability/chart rows all quietly no-op'd while the user still saw a normal answer. The frontend passes Go-issued analysis ids, which are UUIDs, so this should never fire in practice.

## Tools

### `GET /api/v1/tools/list`

Returns the deterministic list of tools available to the frontend.

Request: none.

Response `200`:

```json
{
  "count": 1,
  "tools": [
    {
      "command": "/help",
      "name": "help",
      "type": "skill",
      "description": "Show what the assistant can do and guide your next step."
    }
  ]
}
```

> The catalog is `/help` only (KM-711). `/report` was removed as a slash command —
> report generation is a right-side **Generate** button, not a `/` command. The report
> HTTP endpoint (`POST /api/v1/tools/report`) still exists; the button calls it.

Tool item shape:

```json
{
  "command": "/help",
  "name": "help",
  "type": "skill",
  "description": "Show what the assistant can do and guide your next step."
}
```

Frontend behavior:

- Surface `/help` in the slash menu.
- Surface report generation as a button or explicit UI action.

### `POST /api/v1/tools/help`

Streams contextual guidance for the current analysis conversation.

Request body:

```json
{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}
```

Response: `text/event-stream` using the same event shape as chat.

Help responses usually emit `sources: []` and no `status` pings.

Example:

```text
event: sources
data: []

event: chunk
data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}
```

## Reports

### `POST /api/v1/tools/report`

Generates, persists, and returns a new report version for an analysis.

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `analysis_id` | Yes | Analysis identifier. |
| `user_id` | Yes | User identifier. |
| `exclude_record_ids` | No | Record ids to leave out of this version (repeat the param per id). Added 2026-07-09; get ids from `GET /tools/report/{analysis_id}/records`. Excluded runs are listed in the report's "Excluded Analyses" section. Excluding every substantive record returns `409`. |

Example:

```text
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c&exclude_record_ids=rec_a1&exclude_record_ids=rec_c3
```

Status codes:

| Status | Meaning |
| --- | --- |
| `201` | New report version generated. |
| `409` | Report floor/precondition not met. |
| `500` | Generation or persistence failed. |

Response `201`:

```json
{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
  "bq_answers": [
    {
      "question": "Which regions contribute most to total revenue?",
      "answer": "The Central region leads with 38% of total revenue.",
      "status": "answered",
      "record_ids": ["rec_a1"]
    },
    {
      "question": "Did any region decline quarter-over-quarter?",
      "answer": "Yes — the West region fell 12% QoQ.",
      "status": "answered",
      "record_ids": ["rec_b2"]
    }
  ],
  "findings": [
    {
      "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"],
      "supporting_data": null
    },
    {
      "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"],
      "supporting_data": null
    }
  ],
  "caveats": [
    {
      "text": "March data for the East region was partially missing, around 6% of rows.",
      "record_ids": ["rec_b2"]
    }
  ],
  "open_questions": [
    {
      "text": "What drove the West region's QoQ decline?",
      "record_ids": ["rec_b2"]
    }
  ],
  "unresolved": [
    {
      "text": "Correlate churn with tenure — churn column not found in the source.",
      "record_ids": ["rec_d4"]
    }
  ],
  "excluded": [],
  "evidence_tables": {
    "rec_a1": [
      {
        "title": "Aggregate revenue by region",
        "columns": ["region", "total_revenue"],
        "rows": [["Central", "18321"], ["West", "9954"]],
        "truncated": false
      }
    ]
  },
  "data_sources": [
    {
      "source_id": "src_sales_db",
      "name": "orders",
      "source_type": "postgres",
      "detail": {
        "tables": ["orders"],
        "row_count": 48213,
        "columns": ["region", "amount", "ordered_at"]
      }
    }
  ],
  "method_steps": [
    {
      "task_id": "t1",
      "stage": "data_understanding",
      "objective": "Inventory the sales source",
      "status": "success",
      "tools_used": ["check_data"]
    },
    {
      "task_id": "t2",
      "stage": "modeling",
      "objective": "Aggregate revenue by region",
      "status": "success",
      "tools_used": ["analyze_aggregate"]
    }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
}
```

Response `409`:

```json
{
  "detail": "Not ready to generate a report - still needs at least one completed analysis."
}
```

Report v2 fields (added 2026-07-09; all default-empty, so older stored reports read back unchanged):

- `bq_answers` — one entry per business question. `status` is `answered` | `partial` | `unanswered`; `record_ids` cite the backing analyses. Written in the analysis's language (Indonesian objective → Indonesian answers).
- `unresolved` — runs that were attempted but produced no usable evidence (every `analyze_*` step failed). Not part of the findings body.
- `excluded` — runs the caller excluded via `exclude_record_ids`.
- `evidence_tables` — `record_id` → small result tables copied from the run's stored outputs (max 3 tables per record, max 10 rows each; `truncated: true` when rows were capped). Rendered as markdown tables under the matching Key Findings group in `rendered_markdown`. **PII redaction (added 2026-07-24):** cells in a column the catalog flags as PII are replaced with `"[redacted]"`; column headers are kept. A report is permanent and versioned, so this is where an unmasked customer name would have lasted longest. The narrative (`executive_summary`, findings, `bq_answers`) is generated from the real values and is unaffected — only the raw evidence dump is redacted. Reports generated before this date are unchanged.
- `charts` *(added 2026-07-14)* — `record_id` → `dataeyond.chart.v1` envelopes (see §Charts) copied verbatim from the run's stored outputs (max 3 per record). `rendered_markdown` gains an `## EDA` section where each chart appears as a fenced block the frontend renders with plotly.js:

  ````text
  ```plotly
  {
    "schema": "dataeyond.chart.v1",
    "chart_type": "bar",
    "title": "…",
    "plotly": { "data": [ … ], "layout": { … } }
  }
  ```
  ````

  The fence content is the **full envelope** (same shape as `charts[].spec` on `GET /api/v1/charts`) — the FE hook parses it and renders `Plotly.newPlot(el, parsed.plotly.data, parsed.plotly.layout)`. A bold caption line (chart title) precedes each fence.

Precondition:

- Reports require at least one completed analysis record for the session (*updated 2026-07-14:* a run whose `render_chart` succeeded counts — a chart-only session can generate a report).
- If slow-path analysis recording is disabled, report generation can return `409` by design.

### `GET /api/v1/tools/report/{analysis_id}`

Lists report versions for one analysis, oldest first.

Response `200`:

```json
[
  {
    "report_id": "1b2c3d4e",
    "version": 1,
    "generated_at": "2026-06-24T15:02:11Z",
    "record_count": 1
  },
  {
    "report_id": "8f3a2b1c",
    "version": 2,
    "generated_at": "2026-06-25T09:14:33Z",
    "record_count": 2
  }
]
```

If no reports exist, returns `[]`.

### `GET /api/v1/tools/report/{analysis_id}/records` (added 2026-07-09)

Lists the persisted analysis runs a report would be built from, oldest first. The frontend shows this before generating so the user can deselect runs; the chosen ids go to `POST /tools/report` as `exclude_record_ids`.

Response `200`:

```json
[
  {
    "record_id": "rec_a1",
    "goal_restated": "Rank regions by total revenue",
    "created_at": "2026-06-30T08:55:02Z",
    "substantive": true,
    "findings_count": 2
  },
  {
    "record_id": "rec_d4",
    "goal_restated": "Correlate churn with tenure",
    "created_at": "2026-06-30T09:01:47Z",
    "substantive": false,
    "findings_count": 1
  }
]
```

`substantive: true` means **the run will appear in the report's findings body**; `substantive: false` means it will be listed in the report's `unresolved` JSON field instead. (Since 2026-07-09 the rendered markdown is compact and no longer includes "Attempted, Unresolved" / "Notes & Limitations" / "How This Was Analyzed" sections; the JSON fields `unresolved` / `caveats` / `open_questions` / `method_steps` are unchanged.) If no runs exist, returns `[]`.

> **Changed 2026-07-23 — behavioral, non-breaking (field name, type and position unchanged).** `substantive` previously meant the narrower "a successful `analyze_*` step exists". Planner recipes R2/R2b legitimately answer a question with a single aggregate `retrieve_data` and **no** `analyze_*` step, so under the old rule such a run was reported `substantive: false`, was dropped from the report body, and its business question rendered **"Unanswered"** in `bq_answers` — while the chat had answered it correctly. `substantive` now tracks the report body exactly. **FE impact:** none required. A run that previously showed `false` and was silently excluded may now show `true` and be included; no field was added, removed or retyped.

### `GET /api/v1/tools/report/{analysis_id}/readiness` (added 2026-07-09)

Deterministic report-readiness signal for the Generate-Report button — the same producer as Help's readiness signal, including the advisory delta-since-report check, so the button, Help, and this endpoint never disagree.

Response `200`:

```json
{
  "ready": false,
  "missing": ["a new analysis since the last report"]
}
```

Note: `POST /tools/report` itself only enforces the floor (`at least one completed analysis`) — a new version is always allowed. The delta gap in `missing` is a soft warning the frontend can surface ("nothing new since the last report") without blocking the button.

### `GET /api/v1/tools/report/{analysis_id}/{version}`

Returns one report version. Shape is the same as the `201` response from `POST /api/v1/tools/report`.

Response `404`:

```json
{
  "detail": "No report v3 for analysis 'an_42'."
}
```

## Traceability

> Renamed from `observability` (team decision 2026-07-06) so it is never confused with the internal Langfuse *observability* stack (engineering telemetry, PII-masked). Traceability is **user-facing** provenance — real tool args, output previews, and the executed query — shown alongside the answer.

### `GET /api/v1/traceability`

Returns user-facing provenance for one assistant answer.

The frontend should call this after the chat/help stream emits `done`, using the `message_id` from the `done` event. The row is written **before** `done`, so an immediate GET returns `200` (no polling race). A `404` means the id is unknown or the turn errored before completing (error turns never produce a row).

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `analysis_id` | Yes | Analysis identifier. |
| `message_id` | Yes | Assistant answer identifier returned by the stream. |

Example:

```text
GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1
```

`intent` values the frontend may see: `chat` · `help` · `check` · `unstructured_flow` · `structured_flow` · `out_of_scope` · `blocked` (`blocked` = input-guard or Azure content-filter refusal; `chat` also covers the greeting fast-path and cache replays).

Field rules:

- `planning`: present only when the planner ran (`structured_flow`); otherwise `null`.
- `thinking`: **always `null` in v1** — our agents are plain chat completions with no native reasoning output, and synthesizing it post-hoc would be unfaithful. The field stays in the payload so it can be populated later without a contract change.
- `tool_calls`: every invoked tool with `summary` (plain-English one-liner), `input`, `output`, `status`, `task_id` (nullable), and `error` (nullable); empty for chat / help / greeting / refusal paths. `input`/`output` are the raw tool I/O (opaque ids) — render them in a collapsible "technical details" section, not the headline; use `summary` for the headline.
- `data_used`: one entry per `retrieve_data` call, resolved to **real names** for display (present only when a structured pull ran; empty otherwise). Split into `columns_read` (columns read straight from the user's data, each tagged with its `roles`) and `output_columns` (`kind: "column"` = read from data, `kind: "computed"` = calculated, carrying a `formula` and **no id**). Also carries `tables` (all touched, incl. join targets), `joins`, `filters` (with a plain-language `description`), `group_by`, `order_by`, `limit`, `rows_returned`, and the executed `query`. Built by deterministic catalog lookup — no LLM.
- **`id` fields are machine-only.** Every `id` in `data_used` (`source.id`, `tables[].id`, `columns_read[].id`) is for linking/reconciliation/audit — **the frontend must never render it.** Show `name` (qualified as `table.name`). A `computed` output column has no id by design.
- `sources`: required for retrieval flows; empty for chat / help / refusal paths and for `check`. Database sources also carry `source_name` (the DB's real name) and `tables` (every table touched).
- `thinking`, `filters[].description`, `tool_calls[].summary` are built from fixed templates, never an LLM — traceability adds no latency or token cost and cannot hallucinate.
- The payload also carries an internal `user_id` (ownership); the frontend may ignore it.
- Truncation: `preview` ≤ 5 rows; any string inside `input`/`output`/`preview`/`snippet` ≤ 300 chars (executed `query` ≤ 2000); rows beyond the preview are dropped (`row_count` is preserved).
- **PII redaction in `preview` (added 2026-07-24).** Cells belonging to a column the catalog flags as PII are replaced with the literal string `"[redacted]"`. When any cell was redacted the output also carries `pii_masked`: the list of column names affected — render that so the user understands the blanks are deliberate, not missing data. The **column headers are never redacted**, only the values. This affects the stored provenance record only: the chat answer itself is generated from the real values, so a question like "list our top customers" still answers normally. Records written before this date have no `pii_masked` key and are returned unchanged.

Response `200` for `structured_flow`:

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "user_id": "user_7",
  "intent": "structured_flow",
  "generated_at": "2026-07-06T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": [],
    "steps": [
      {
        "step": 1,
        "stage": "data_understanding",
        "objective": "Inventory the sales source",
        "status": "success",
        "tools_used": ["check_data"]
      },
      {
        "step": 2,
        "stage": "modeling",
        "objective": "Aggregate revenue by region",
        "status": "success",
        "tools_used": ["retrieve_data", "analyze_aggregate"]
      }
    ]
  },
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "check_data",
      "summary": "Inspected your data source structure",
      "input": { "source_hint": "structured" },
      "output": {
        "kind": "table",
        "columns": ["source_id", "name", "source_type", "table_count"],
        "row_count": 1,
        "preview": [["src_sales_db", "orders", "schema", 1]]
      },
      "status": "success",
      "error": null
    },
    {
      "order": 2,
      "task_id": null,
      "name": "retrieve_data",
      "summary": "Retrieved 4 rows across 2 columns from orders",
      "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } },
      "output": {
        "kind": "table",
        "columns": ["region", "total"],
        "row_count": 4,
        "preview": [["Central", 1210000], ["East", 740000]]
      },
      "status": "success",
      "error": null
    }
  ],
  "data_used": [
    {
      "source": { "id": "src_sales_db", "name": "sales db", "type": "schema" },
      "tables": [ { "id": "orders", "name": "orders", "role": "base" } ],
      "joins": [],
      "columns_read": [
        { "id": "c_region", "name": "region", "table": "orders", "data_type": "string",  "pii": false, "roles": ["selected", "grouped"] },
        { "id": "c_amount", "name": "amount", "table": "orders", "data_type": "decimal", "pii": false, "roles": ["aggregated"] }
      ],
      "output_columns": [
        { "name": "region", "kind": "column",   "from": "orders.region" },
        { "name": "total",  "kind": "computed", "from": "orders.amount", "formula": "SUM(orders.amount)" }
      ],
      "filters": [],
      "group_by": ["orders.region"],
      "order_by": [],
      "limit": null,
      "rows_returned": 4,
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region"
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "source_name": "sales db",
      "name": "orders",
      "tables": ["orders"],
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": {
        "table": "orders",
        "row_count": 4
      }
    }
  ]
}
```

> Note: `tool_calls[].input` is the raw compiled query IR (opaque `column_id`/`table_id`) — the technical layer. **`data_used` is the user-facing layer**: the same pull resolved to real names, with computed columns (e.g. `total`) flagged `kind: "computed"` so they are never shown as if they were columns in the user's database. Every `id` there is machine-only.

Response `200` for `unstructured_flow`:

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "user_id": "user_7",
  "intent": "unstructured_flow",
  "generated_at": "2026-07-06T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project" },
      "output": { "kind": "documents", "row_count": 4 },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "document",
      "document_id": "doc_7",
      "filename": "tech_handbook.pdf",
      "page_label": "12",
      "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
      "score": 0.83
    }
  ]
}
```

Response `200` for chat / greeting / help / refusals (`out_of_scope`, `blocked`):

```json
{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "user_id": "user_7",
  "intent": "chat",
  "generated_at": "2026-07-06T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}
```

Response `404`:

```json
{
  "detail": "No traceability for message 'msg_88f1' yet."
}
```

Frontend rendering guidance:

- Render traceability separately from the streamed answer.
- Default state can be collapsed.
- Show planning, tool calls, and sources as separate sections.
- Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.

## Charts

> Added 2026-07-13 (SPINE_V2_PLAN §4.4/§4.5, S2 visualization). A chart is a `render_chart` tool output, planner-selected only when the user explicitly asks to plot/visualize. Delivery mirrors traceability: a Python-owned store plus a dedicated `GET` endpoint; the streamed answer (SSE) stays text-only — charts are never embedded in `chunk`.

### `GET /api/v1/charts`

Returns every chart produced by `render_chart` during one assistant answer.

The frontend should call this after the chat stream emits `done`, using the `message_id` from the `done` event — the same fetch-on-`done` pattern as traceability; the row(s) are written **before** `done`, so there is no polling race.

> **Updated 2026-07-13 (lead review):** lookup is now by **`message_id` alone** (it is a server-minted UUID — globally unique; `analysis_id` was removed from the query), and every response is **HTTP 200 with an explicit `status` marker** instead of a bare list: `success` (≥1 chart), `empty` (the turn completed but produced no charts — the common case), `not_found` (no completed turn is known for this `message_id`: mistyped/stale id, or an error turn). *(Note the asymmetry: `GET /api/v1/traceability` still takes `analysis_id` + `message_id` — aligning it is a separate change if wanted.)*

Query params:

| Query | Required | Description |
| --- | --- | --- |
| `message_id` | Yes | Assistant answer identifier returned by the stream's `done` event. |

Example:

```text
GET /api/v1/charts?message_id=88f10c3a-6f03-4204-bf98-41ffc20388b2
```

The `dataeyond.chart.v1` envelope (the shape of `charts[].spec`, verbatim from `render_chart`):

```json
{
  "schema": "dataeyond.chart.v1",
  "chart_type": "bar",
  "title": "Revenue by region",
  "plotly": {
    "data":   [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
    "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
                "yaxis": {"title": {"text": "revenue"}} }
  }
}
```

Frontend renders it with `Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. v1 chart types: `bar`, `line`, `pie`, `scatter`.

Response `200` (`status: success` — ≥1 chart):

```json
{
  "status": "success",
  "message": "1 chart(s) for this message.",
  "count": 1,
  "charts": [
    {
      "chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e",
      "chart_type": "bar",
      "title": "Revenue by region",
      "spec": {
        "schema": "dataeyond.chart.v1",
        "chart_type": "bar",
        "title": "Revenue by region",
        "plotly": {
          "data":   [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
          "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
                      "yaxis": {"title": {"text": "revenue"}} }
        }
      },
      "created_at": "2026-07-13T03:21:09.114Z"
    }
  ]
}
```

Response `200` (`status: empty` — chartless turn, the common case; not an error):

```json
{
  "status": "empty",
  "message": "This message completed without producing charts.",
  "count": 0,
  "charts": []
}
```

Response `200` (`status: not_found` — no completed turn known for this id):

```json
{
  "status": "not_found",
  "message": "No completed turn is known for this message_id.",
  "count": 0,
  "charts": []
}
```

Field rules:

- `status` is the outcome marker: `success` | `empty` | `not_found` (always HTTP 200 — the FE fetches unconditionally, so a missing turn is a data state, not a transport failure). `empty` vs `not_found` is decided against the turn's traceability row (written before `done`), so `not_found` reliably means "this id never completed a turn".
- `message` is a human-readable line for logs/debugging — do not parse it; branch on `status`.
- `spec` is the full `dataeyond.chart.v1` envelope, unmodified — it is the source of truth, not a projection; render straight from it.
- `chart_type` / `title` are copied out of `spec` for convenience (list rendering without parsing `spec`); `title` may be `null`.
- A turn can produce more than one chart (multiple `render_chart` calls in the same plan); `charts` is ordered by creation time.
- The payload carries no `user_id` / `analysis_id` — charts are keyed by `message_id` alone.

> **DDL note (Harry / dedorch migration):** the original manual index is `(analysis_id, message_id)`, which does not serve a `message_id`-only lookup. Additive index for the migration (also safe to run manually now):
> ```sql
> CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id);
> ```

Frontend rendering guidance:

- Fetch unconditionally on every `done` — see the SSE note above (no `chart_count` hint yet); an empty `charts[]` means render nothing extra.
- Render each chart under the assistant message it belongs to, via `Plotly.newPlot`.
- "Chart iteration" v1 = a follow-up chat turn (e.g. "make it a line chart") — the planner re-emits `render_chart` with patched args. There is no separate edit endpoint.

## Knowledge

> Added 2026-09-07 (KNOWLEDGE_SURFACE_PLAN.md). Turns a user's uploaded documents into reviewable knowledge artifacts: glossary terms, rules, formulas and one domain context per document. **Admin-shaped, not part of the chat flow** — no chat endpoint calls any of this.
>
> **Corrected 2026-09-14:** the line that stood here ("the analytics path does not yet read what it produces") is out of date — the consumption path was armed 2026-09-08 (DEV_PLAN §0.8 K5/K9/K11). The planner reads the company's approved terminology on every slow-path turn. **Approved entries only**, so it stays inert until an expert works the queue.

**Asynchronous by necessity.** Parsing a nine-page document takes ~80 s and extracting from it ~170 s, far past any HTTP timeout, so ingestion admits a batch and returns immediately. **Polling, not SSE** (decided 2026-09-07): a job row survives a page refresh and a dropped connection, a stream does not, and a four-minute stream would hold a worker for its duration.

**Scope.** Knowledge belongs to the **company**, not the uploader: two employees' documents are one body of knowledge and must share a glossary. Every row carries `scope_id` (the tenant) beside `user_id` (who uploaded). **`scope_id` is never accepted from the caller** — it is derived server-side from the user's company. Accepting a tenant key on an unauthenticated route would be a cross-tenant read with extra steps. One click currently covers **the caller's own documents**; company-wide is a one-line change behind a single resolver when wanted.

**Cost controls.** This is the only endpoint group that spends LLM budget, so it carries four: a rate limit (5/minute), one running job per company, a pre-flight spend ceiling (the free stages build the exact prompts and quote a real number before anything is sent), and content-addressed idempotency — re-ingesting an unchanged document re-spends nothing. **None of them identifies the caller**; see Authentication.

### `POST /api/v1/knowledge/ingest`

Extract knowledge from a user's documents. Returns `202` immediately; poll the job.

Body:

| Field | Required | Description |
| --- | --- | --- |
| `user_id` | Yes | The caller. Must be a UUID (422 otherwise). Scope is derived from it. |
| `document_ids` | No | Omit for the one-click case: every parseable document this user owns. Named ids are still filtered by owner and type. |
| `reparse` | No | Re-parse even when the content hash matches a stored artifact. Default `false` — on by default it silently re-spends. |
| `review_mode` | No | **Added 2026-09-14.** `expert` (default) or `llm`. Who rules on what this batch produces. |

Only `pdf` and `docx` are parsed; spreadsheets and text files belong to the structured-data path and come back in `skipped` **with a reason**, never silently dropped.

**`review_mode` (2026-09-14).** `expert` leaves every entry waiting for a human. `llm` approves
them on arrival with **no reviewer step and no second opinion** — the entries were written by a
model and are accepted as-is; there is no second model call. What the FE must surface when
offering it: choosing `llm` records `authorised_by` against the caller, so an entry approved this
way can always answer *who allowed this*. Default is `expert`, so nobody gets LLM approval by
accident. Entries are written through the ordinary append-only review path, so an expert can
still disagree with any of them afterwards.

`reused: true` means a job for this company was already running and is being returned instead of a second one starting — a double click is not an error.

```json
{
  "job_id": "8f3c…", "status": "queued", "stage": null,
  "n_requested": 1, "reused": false,
  "skipped": [{"document_id": "120a…", "filename": "pa.csv", "reason": "unsupported_type:csv"}],
  "message": "Ingesting 1 document(s)."
}
```

### `GET /api/v1/knowledge/jobs/{job_id}`

Job state and per-document outcomes. **404** when the id is unknown — unlike `GET /charts`, whose absence is normal; a `job_id` was handed to the caller, so its absence is exceptional.

`status` is `queued` · `running` · `succeeded` · `partial` · `failed` · `blocked`. Two are worth reading carefully:

- **`partial`** — some documents landed and some did not. One unreadable PDF must not fail a batch of twenty, so `results[]` carries each document's outcome.
- **`blocked`** — **not a failure.** The pre-flight estimate exceeded the spend ceiling and the job declined to continue. It is the one outcome an operator can act on.

`stage` (`parsing` · `filtering` · `estimating` · `extracting` · `persisting`) is ordered by cost: everything up to and including `estimating` is free, and the ceiling is enforced between `estimating` and `extracting`.

> **Updated 2026-09-11 — the estimate is now returned, and `blocked` is now reachable.**
>
> `estimated_cost` was on the job model from the start and was **never populated**, so the field a
> caller would poll to answer "what will this cost, and why was that document refused" always came
> back `null`. It now carries the pre-flight shape of the batch, per document and in total. A
> wholly-refused batch also settles **`blocked`** rather than `failed` — previously every spend
> refusal was reported as a generic failure, which hid the one outcome with an action attached.
>
> **Spend is driven by CLUSTER COUNT** — one LLM call per term cluster — not by page count, so the
> per-document breakdown is what tells an operator which document to split. There are now **two
> ceilings**, and `blocked_by` says which one bound:
>
> | `blocked_by` | Meaning | The operator's move |
> |---|---|---|
> | `clusters` | Too many term clusters (exact, known free before spending) | Split the document |
> | `prompt_tokens` | Too many estimated prompt tokens | The document is simply long |
>
> ```json
> {
>   "status": "blocked",
>   "estimated_cost": {
>     "clusters": 2083, "total_calls": 2150, "estimated_prompt_tokens": 5100000,
>     "per_document": {
>       "d1": {"clusters": 83,   "total_calls": 106, "estimated_prompt_tokens": 201751},
>       "d2": {"clusters": 2000, "total_calls": 2044, "estimated_prompt_tokens": 4898249}
>     },
>     "blocked": {"d2": "clusters"}
>   },
>   "results": [
>     {"document_id": "d2", "status": "failed", "blocked_by": "clusters",
>      "error": "estimated 2,000 term clusters exceeds the ceiling of 750 (267% of it) — nothing was spent; 2,044 LLM calls were declined"}
>   ]
> }
> ```
>
> Each `results[]` entry also carries `estimate` — **on success as well as refusal**, because a
> number a caller only ever sees when it is already too late cannot be planned against.
>
> `estimate.genre` is advisory document classification (`standard` · `diagnostic` · `conceptual` ·
> `unknown`) with a `well_served` flag. **Nothing is routed on it yet**; it exists so a caller can be
> told *before* spending that a troubleshooting manual will yield almost no definitions.

### `GET /api/v1/knowledge/jobs`

Recent jobs for one `user_id`, newest first — lets a caller recover after a refresh without having stashed a `job_id`.

### `GET /api/v1/knowledge/documents`

Parsed artifacts stored for the caller's company: page and chunk counts, parser, `schema_version`, `content_hash`. This is what a re-extraction is chosen from.

### `POST /api/v1/knowledge/documents/{knowledge_document_id}/extract`

Re-run extraction over a **stored** artifact. Same job machinery and same polling route, but no parsing — prompt retuning is the common case and must not re-pay the parser for a byte-identical artifact. Takes `user_id` as a query parameter.

### `GET /api/v1/knowledge/entries`

Candidate entries, filterable by `doc_id` and `kind` (`glossary` · `rule` · `formula` · `domain`), each annotated with the current expert decision if one exists. `payload` is the full typed entry and is the source of truth; the promoted fields beside it (`label`, `mention_count`, `diff_status`, …) exist so a list can be ordered without opening it.

`diff_status` (`new` · `duplicate` · `conflicting`) is the **cross-document** relationship. Entries append and never merge: the same term in two documents keeps two ids, because deciding they are one concept is a judgement about the world rather than a string comparison.

### `GET /api/v1/knowledge/queue`

The review queue — conflicts first, then descending mention count, each row carrying a plain-language `review_reason`.

**Derived at read time, never stored:** persisting an ordering creates a second truth that drifts from the entries it came from.

⚠️ Every row carries **`row_kind`** and a consumer **must switch on it**. A `key_parameter` row has a `surface` and no `term`, no definition and no mention count; a `term` row has all of them. Provenance carries both `page` (0-based, as the parser reports) and **`page_no` (1-based)** — **a UI must render `page_no`**.

### `POST /api/v1/knowledge/entries/{entity_id}/reviews`

Record one expert decision: `approved` · `edited` · `rejected`. **Append-only** — a changed mind is a new row, never an update.

Keyed on **`entity_id`**, not the entry's row id: `entity_id` is deterministic and content-derived, so it survives re-extraction while row ids do not. That is what lets approvals outlive a prompt retune. `content_hash` is **derived server-side** from the artifact the entry came from, never taken from the request, so a stale client cannot attribute a ruling to the wrong version. `edited` requires `edited_payload` (422 otherwise); an unknown `entity_id` is 404.

**This closes the loop.** Approved and edited entries become the baseline the *next* extraction diffs against, so working the queue improves the following run rather than only annotating this one.

> ⚠️ **Injection path, recorded not fixed.** Because an `edited_payload` reaches a future prompt through that baseline, and this surface is unauthenticated, anyone who can reach it can influence later extractions. This is the strongest argument for the forwarded identity in DEV_PLAN #43, and it should be closed before an expert relies on the queue.