Rifqi Hafizuddin
Claude Opus 5
[NOTICKET] docs: sync REPO_STATUS / DEV_PLAN / contract for the 8 Sep list (K24)
be227a8 | ο»Ώ# Backend Agentic Service API Contract | |
| <!-- doc-status --> | |
| > β **LIVE β CANONICAL: the contract of record** for the FE-facing surface. This is the one that governs endpoint behaviour. The frozen 2026-06-30 snapshot sent to Harry is [docs/archive/API_ENDPOINTS_RESTRUCTURE.md](docs/archive/API_ENDPOINTS_RESTRUCTURE.md) β do not implement from that one. | |
| This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and traceability data shown alongside chat answers. | |
| Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL. | |
| ## Overview | |
| The Python backend owns the generative AI interaction surface: | |
| 1. Stream chat answers from the AI agent. | |
| 2. Execute tool-style actions for help and report generation. | |
| 3. Return report versions and report details. | |
| 4. Return traceability/provenance for a completed assistant answer. | |
| The frontend uses this service during the analysis conversation flow: | |
| 1. User sends a chat message. | |
| 2. Frontend calls `POST /api/v2/chat/stream` and renders the streamed answer. | |
| 3. When the stream emits `done`, frontend stores or reads the returned `message_id`. | |
| 4. Frontend calls `GET /api/v1/traceability` for planning, tool calls, and source provenance. | |
| 5. Frontend calls `/api/v1/tools/help` for guided help and `/api/v1/tools/report` for report generation. | |
| ## Endpoint Summary | |
| | Method | Path | Purpose | | |
| | --- | --- | --- | | |
| | `POST` | `/api/v2/chat/stream` | Stream an AI chat answer for one analysis conversation. | | |
| | `GET` | `/api/v1/tools/list` | List available frontend tools. | | |
| | `POST` | `/api/v1/tools/help` | Stream contextual help for the current analysis conversation. | | |
| | `POST` | `/api/v1/tools/report` | Generate and persist a new report version. | | |
| | `GET` | `/api/v1/tools/report/{analysis_id}` | List report versions for an analysis. | | |
| | `GET` | `/api/v1/tools/report/{analysis_id}/records` | List analysis records for report curation (added 2026-07-09). | | |
| | `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). | | |
| | `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. | | |
| | `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. | | |
| | `GET` | `/api/v1/charts` | Retrieve chart(s) produced by `render_chart` for one assistant answer (added 2026-07-13). | | |
| | `POST` | `/api/v1/knowledge/ingest` | Extract knowledge from a user's documents β async batch (added 2026-09-07). | | |
| | `GET` | `/api/v1/knowledge/jobs/{job_id}` | Poll one ingestion job (added 2026-09-07). | | |
| | `GET` | `/api/v1/knowledge/jobs` | Recent ingestion jobs for a user (added 2026-09-07). | | |
| | `GET` | `/api/v1/knowledge/documents` | Parsed artifacts stored for the caller's company (added 2026-09-07). | | |
| | `POST` | `/api/v1/knowledge/documents/{id}/extract` | Re-extract a stored artifact, no re-parsing (added 2026-09-07). | | |
| | `GET` | `/api/v1/knowledge/entries` | Candidate knowledge entries (added 2026-09-07). | | |
| | `GET` | `/api/v1/knowledge/queue` | The expert review queue (added 2026-09-07). | | |
| | `POST` | `/api/v1/knowledge/entries/{entity_id}/reviews` | Record an expert decision (added 2026-09-07). | | |
| ## Authentication | |
| **None. The live surface is unauthenticated β do not expose it beyond the demo.** | |
| The service accepts `user_id` and `analysis_id` as ordinary request fields with no verification, and there is no caller authentication in front of any endpoint. | |
| **History (2026-07-23 β 2026-07-27).** A shared service-secret gate (`X-Dataeyond-Service-Secret`, router-level, constant-time compare) shipped 2026-07-23, inert until `dataeyond__service__secret` was set. It was **unwired on 2026-07-27** (DEV_PLAN #37): the only caller of this service is the browser SPA, which cannot be changed to send the header, so the gate could never be armed without a 401 outage. The gate code is parked in `src/middlewares/service_auth.py` (not deleted) and restores in one edit if the caller situation changes. | |
| **Updated 2026-09-07 β the `/knowledge` group raises the stakes.** Those endpoints are *writes* that spend LLM budget, and one of them (`POST β¦/reviews`) accepts content that reaches a later prompt. They are bounded by a rate limit, one job per company, a pre-flight spend ceiling and content-addressed idempotency β none of which identifies the caller. The posture is recorded rather than inherited: see the Knowledge section. | |
| **The real fix (DEV_PLAN #43).** Per-user authorization requires a verified identity forwarded by the Go service; once Go forwards a token, `user_id` comes from the verified claims rather than the request body. Until then the `user_id` predicates in the stores (F-1) are defensive-in-depth only, not an access control. | |
| ## Common Concepts | |
| ### Identifiers | |
| - `user_id`: user identifier passed by the frontend. | |
| - `analysis_id`: analysis conversation identifier. | |
| - `message_id`: assistant answer identifier used to correlate chat streaming with traceability. **Server-minted, never accepted from the client.** **Updated 2026-07-09:** it is a UUID string (e.g. `77f06761-0fdf-4cc5-84f8-5f81bcbb6f84`), matching the shape of Go's `analyses_messages.id`. The `msg_β¦` values in the examples below are illustrative placeholders only. | |
| ### Server-Sent Events | |
| Chat and help endpoints return `text/event-stream`. | |
| Frontend should parse events by `event` name and `data` payload. Blank lines separate SSE events. | |
| Common event types: | |
| | Event | Data | Meaning | | |
| | --- | --- | --- | | |
| | `sources` | JSON array | Always `[]` β sources moved to `GET /api/v1/traceability` (KM-691). Event kept for backward-compat; read `sources[]` from the traceability call. | | |
| | `status` | text | Optional progress update for slower paths. | | |
| | `chunk` | text | Answer text fragment. Concatenate chunks in order. | | |
| | `done` | JSON object | Terminal success event. Includes `message_id`. | | |
| | `error` | text | Terminal error event. Stream stops after this. | | |
| The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done. | |
| **Charts (added 2026-07-13, SPINE_V2_PLAN Β§4.5):** the `done` event is unchanged by charts β no additive `chart_count`/`chart_ids` field yet (open question owned by Harry). Until that's resolved, the frontend fetches `GET /api/v1/charts` unconditionally on every `done`, the same fetch-on-`done` pattern already used for traceability. SSE order and every existing field stay exactly as documented above; `sources` stays `[]`. | |
| ## Chat | |
| ### `POST /api/v2/chat/stream` | |
| Streams an AI answer for one user message in an analysis conversation. | |
| Request body: | |
| ```json | |
| { | |
| "user_id": "u_1a2b3c", | |
| "analysis_id": "an_42", | |
| "message": "What were total sales by region last quarter?" | |
| } | |
| ``` | |
| Fields: | |
| | Field | Required | Description | | |
| | --- | --- | --- | | |
| | `user_id` | Yes | User identifier. | | |
| | `analysis_id` | Yes | Analysis conversation identifier. | | |
| | ~~`message_id`~~ | β | **Updated 2026-07-09:** not a request field. Python always mints the id server-side and returns it on `done`; any caller-sent value is ignored (server-authoritative β open-Q #1). | | |
| | `message` | Yes | User message text. | | |
| Response: `text/event-stream`. | |
| Example structured answer: | |
| ```text | |
| event: sources | |
| data: [] | |
| event: status | |
| data: Planning analysis... | |
| event: status | |
| data: Running 3 steps... | |
| event: chunk | |
| data: Total sales by region last quarter: | |
| event: chunk | |
| data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ). | |
| event: done | |
| data: {"message_id":"msg_88f1"} | |
| ``` | |
| Example simple chat answer: | |
| ```text | |
| event: sources | |
| data: [] | |
| event: chunk | |
| data: I'm your AI data analyst. Connect a source or ask a question to get started. | |
| event: done | |
| data: {"message_id":"msg_12"} | |
| ``` | |
| Behavior notes: | |
| - Greeting and farewell messages may use a fast canned path. | |
| - Stateless `chat` intent may use a 1-hour Redis response cache. | |
| - The router may classify messages into intents such as `chat`, `help`, `check`, `unstructured_flow`, or `structured_flow`. | |
| - `sources` in the stream is **always `[]`** (KM-691) β read the real `sources[]` from `GET /api/v1/traceability` after `done`. | |
| - `status` events are optional and should be safe for the frontend to ignore. | |
| - **Fixed 2026-07-23 β `sources` is now genuinely always first, on every path.** It previously did not match this document: the `check` and router-`help` turns emitted **no `sources` event at all**, and the `structured_flow` (slow) path emitted `status` *before* `sources`, inverting the documented order on exactly the turns that take longest. A frontend that initializes per-turn state on `sources` therefore never initialized on check/help turns and initialized late on slow turns. The change is **purely additive** β no event was removed or renamed, and `sources` is still `[]` β so no frontend change is required; behaviour simply now matches what this contract always said. | |
| - **`analysis_id` must be a UUID (2026-07-23).** `POST /api/v2/chat/stream` and `POST /api/v1/tools/help` now return **`422`** for a non-UUID `analysis_id` instead of accepting it. It was never actually usable: `analyses.id` is `uuid NOT NULL`, so a non-UUID id silently matched no analysis and the turn ran with no state β help, readiness, the report write-back, and the traceability/chart rows all quietly no-op'd while the user still saw a normal answer. The frontend passes Go-issued analysis ids, which are UUIDs, so this should never fire in practice. | |
| ## Tools | |
| ### `GET /api/v1/tools/list` | |
| Returns the deterministic list of tools available to the frontend. | |
| Request: none. | |
| Response `200`: | |
| ```json | |
| { | |
| "count": 1, | |
| "tools": [ | |
| { | |
| "command": "/help", | |
| "name": "help", | |
| "type": "skill", | |
| "description": "Show what the assistant can do and guide your next step." | |
| } | |
| ] | |
| } | |
| ``` | |
| > The catalog is `/help` only (KM-711). `/report` was removed as a slash command β | |
| > report generation is a right-side **Generate** button, not a `/` command. The report | |
| > HTTP endpoint (`POST /api/v1/tools/report`) still exists; the button calls it. | |
| Tool item shape: | |
| ```json | |
| { | |
| "command": "/help", | |
| "name": "help", | |
| "type": "skill", | |
| "description": "Show what the assistant can do and guide your next step." | |
| } | |
| ``` | |
| Frontend behavior: | |
| - Surface `/help` in the slash menu. | |
| - Surface report generation as a button or explicit UI action. | |
| ### `POST /api/v1/tools/help` | |
| Streams contextual guidance for the current analysis conversation. | |
| Request body: | |
| ```json | |
| { | |
| "user_id": "u_1a2b3c", | |
| "analysis_id": "an_42" | |
| } | |
| ``` | |
| Response: `text/event-stream` using the same event shape as chat. | |
| Help responses usually emit `sources: []` and no `status` pings. | |
| Example: | |
| ```text | |
| event: sources | |
| data: [] | |
| event: chunk | |
| data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report. | |
| event: done | |
| data: {"message_id":"msg_h7"} | |
| ``` | |
| ## Reports | |
| ### `POST /api/v1/tools/report` | |
| Generates, persists, and returns a new report version for an analysis. | |
| Query params: | |
| | Query | Required | Description | | |
| | --- | --- | --- | | |
| | `analysis_id` | Yes | Analysis identifier. | | |
| | `user_id` | Yes | User identifier. | | |
| | `exclude_record_ids` | No | Record ids to leave out of this version (repeat the param per id). Added 2026-07-09; get ids from `GET /tools/report/{analysis_id}/records`. Excluded runs are listed in the report's "Excluded Analyses" section. Excluding every substantive record returns `409`. | | |
| Example: | |
| ```text | |
| POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c | |
| POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c&exclude_record_ids=rec_a1&exclude_record_ids=rec_c3 | |
| ``` | |
| Status codes: | |
| | Status | Meaning | | |
| | --- | --- | | |
| | `201` | New report version generated. | | |
| | `409` | Report floor/precondition not met. | | |
| | `500` | Generation or persistence failed. | | |
| Response `201`: | |
| ```json | |
| { | |
| "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c", | |
| "analysis_id": "an_42", | |
| "user_id": "u_1a2b3c", | |
| "version": 2, | |
| "generated_at": "2026-06-30T09:14:33.512Z", | |
| "problem_statement": { | |
| "objective": "Understand which regions drive revenue and why Q1 dipped.", | |
| "business_questions": [ | |
| "Which regions contribute most to total revenue?", | |
| "Did any region decline quarter-over-quarter?" | |
| ] | |
| }, | |
| "record_ids": ["rec_a1", "rec_b2"], | |
| "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.", | |
| "bq_answers": [ | |
| { | |
| "question": "Which regions contribute most to total revenue?", | |
| "answer": "The Central region leads with 38% of total revenue.", | |
| "status": "answered", | |
| "record_ids": ["rec_a1"] | |
| }, | |
| { | |
| "question": "Did any region decline quarter-over-quarter?", | |
| "answer": "Yes β the West region fell 12% QoQ.", | |
| "status": "answered", | |
| "record_ids": ["rec_b2"] | |
| } | |
| ], | |
| "findings": [ | |
| { | |
| "text": "Central region contributed 38% of total revenue, the largest share.", | |
| "record_ids": ["rec_a1"], | |
| "supporting_data": null | |
| }, | |
| { | |
| "text": "West region revenue fell 12% quarter-over-quarter.", | |
| "record_ids": ["rec_b2"], | |
| "supporting_data": null | |
| } | |
| ], | |
| "caveats": [ | |
| { | |
| "text": "March data for the East region was partially missing, around 6% of rows.", | |
| "record_ids": ["rec_b2"] | |
| } | |
| ], | |
| "open_questions": [ | |
| { | |
| "text": "What drove the West region's QoQ decline?", | |
| "record_ids": ["rec_b2"] | |
| } | |
| ], | |
| "unresolved": [ | |
| { | |
| "text": "Correlate churn with tenure β churn column not found in the source.", | |
| "record_ids": ["rec_d4"] | |
| } | |
| ], | |
| "excluded": [], | |
| "evidence_tables": { | |
| "rec_a1": [ | |
| { | |
| "title": "Aggregate revenue by region", | |
| "columns": ["region", "total_revenue"], | |
| "rows": [["Central", "18321"], ["West", "9954"]], | |
| "truncated": false | |
| } | |
| ] | |
| }, | |
| "data_sources": [ | |
| { | |
| "source_id": "src_sales_db", | |
| "name": "orders", | |
| "source_type": "postgres", | |
| "detail": { | |
| "tables": ["orders"], | |
| "row_count": 48213, | |
| "columns": ["region", "amount", "ordered_at"] | |
| } | |
| } | |
| ], | |
| "method_steps": [ | |
| { | |
| "task_id": "t1", | |
| "stage": "data_understanding", | |
| "objective": "Inventory the sales source", | |
| "status": "success", | |
| "tools_used": ["check_data"] | |
| }, | |
| { | |
| "task_id": "t2", | |
| "stage": "modeling", | |
| "objective": "Aggregate revenue by region", | |
| "status": "success", | |
| "tools_used": ["analyze_aggregate"] | |
| } | |
| ], | |
| "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..." | |
| } | |
| ``` | |
| Response `409`: | |
| ```json | |
| { | |
| "detail": "Not ready to generate a report - still needs at least one completed analysis." | |
| } | |
| ``` | |
| Report v2 fields (added 2026-07-09; all default-empty, so older stored reports read back unchanged): | |
| - `bq_answers` β one entry per business question. `status` is `answered` | `partial` | `unanswered`; `record_ids` cite the backing analyses. Written in the analysis's language (Indonesian objective β Indonesian answers). | |
| - `unresolved` β runs that were attempted but produced no usable evidence (every `analyze_*` step failed). Not part of the findings body. | |
| - `excluded` β runs the caller excluded via `exclude_record_ids`. | |
| - `evidence_tables` β `record_id` β small result tables copied from the run's stored outputs (max 3 tables per record, max 10 rows each; `truncated: true` when rows were capped). Rendered as markdown tables under the matching Key Findings group in `rendered_markdown`. **PII redaction (added 2026-07-24):** cells in a column the catalog flags as PII are replaced with `"[redacted]"`; column headers are kept. A report is permanent and versioned, so this is where an unmasked customer name would have lasted longest. The narrative (`executive_summary`, findings, `bq_answers`) is generated from the real values and is unaffected β only the raw evidence dump is redacted. Reports generated before this date are unchanged. | |
| - `charts` *(added 2026-07-14)* β `record_id` β `dataeyond.chart.v1` envelopes (see Β§Charts) copied verbatim from the run's stored outputs (max 3 per record). `rendered_markdown` gains an `## EDA` section where each chart appears as a fenced block the frontend renders with plotly.js: | |
| ````text | |
| ```plotly | |
| { | |
| "schema": "dataeyond.chart.v1", | |
| "chart_type": "bar", | |
| "title": "β¦", | |
| "plotly": { "data": [ β¦ ], "layout": { β¦ } } | |
| } | |
| ``` | |
| ```` | |
| The fence content is the **full envelope** (same shape as `charts[].spec` on `GET /api/v1/charts`) β the FE hook parses it and renders `Plotly.newPlot(el, parsed.plotly.data, parsed.plotly.layout)`. A bold caption line (chart title) precedes each fence. | |
| Precondition: | |
| - Reports require at least one completed analysis record for the session (*updated 2026-07-14:* a run whose `render_chart` succeeded counts β a chart-only session can generate a report). | |
| - If slow-path analysis recording is disabled, report generation can return `409` by design. | |
| ### `GET /api/v1/tools/report/{analysis_id}` | |
| Lists report versions for one analysis, oldest first. | |
| Response `200`: | |
| ```json | |
| [ | |
| { | |
| "report_id": "1b2c3d4e", | |
| "version": 1, | |
| "generated_at": "2026-06-24T15:02:11Z", | |
| "record_count": 1 | |
| }, | |
| { | |
| "report_id": "8f3a2b1c", | |
| "version": 2, | |
| "generated_at": "2026-06-25T09:14:33Z", | |
| "record_count": 2 | |
| } | |
| ] | |
| ``` | |
| If no reports exist, returns `[]`. | |
| ### `GET /api/v1/tools/report/{analysis_id}/records` (added 2026-07-09) | |
| Lists the persisted analysis runs a report would be built from, oldest first. The frontend shows this before generating so the user can deselect runs; the chosen ids go to `POST /tools/report` as `exclude_record_ids`. | |
| Response `200`: | |
| ```json | |
| [ | |
| { | |
| "record_id": "rec_a1", | |
| "goal_restated": "Rank regions by total revenue", | |
| "created_at": "2026-06-30T08:55:02Z", | |
| "substantive": true, | |
| "findings_count": 2 | |
| }, | |
| { | |
| "record_id": "rec_d4", | |
| "goal_restated": "Correlate churn with tenure", | |
| "created_at": "2026-06-30T09:01:47Z", | |
| "substantive": false, | |
| "findings_count": 1 | |
| } | |
| ] | |
| ``` | |
| `substantive: true` means **the run will appear in the report's findings body**; `substantive: false` means it will be listed in the report's `unresolved` JSON field instead. (Since 2026-07-09 the rendered markdown is compact and no longer includes "Attempted, Unresolved" / "Notes & Limitations" / "How This Was Analyzed" sections; the JSON fields `unresolved` / `caveats` / `open_questions` / `method_steps` are unchanged.) If no runs exist, returns `[]`. | |
| > **Changed 2026-07-23 β behavioral, non-breaking (field name, type and position unchanged).** `substantive` previously meant the narrower "a successful `analyze_*` step exists". Planner recipes R2/R2b legitimately answer a question with a single aggregate `retrieve_data` and **no** `analyze_*` step, so under the old rule such a run was reported `substantive: false`, was dropped from the report body, and its business question rendered **"Unanswered"** in `bq_answers` β while the chat had answered it correctly. `substantive` now tracks the report body exactly. **FE impact:** none required. A run that previously showed `false` and was silently excluded may now show `true` and be included; no field was added, removed or retyped. | |
| ### `GET /api/v1/tools/report/{analysis_id}/readiness` (added 2026-07-09) | |
| Deterministic report-readiness signal for the Generate-Report button β the same producer as Help's readiness signal, including the advisory delta-since-report check, so the button, Help, and this endpoint never disagree. | |
| Response `200`: | |
| ```json | |
| { | |
| "ready": false, | |
| "missing": ["a new analysis since the last report"] | |
| } | |
| ``` | |
| Note: `POST /tools/report` itself only enforces the floor (`at least one completed analysis`) β a new version is always allowed. The delta gap in `missing` is a soft warning the frontend can surface ("nothing new since the last report") without blocking the button. | |
| ### `GET /api/v1/tools/report/{analysis_id}/{version}` | |
| Returns one report version. Shape is the same as the `201` response from `POST /api/v1/tools/report`. | |
| Response `404`: | |
| ```json | |
| { | |
| "detail": "No report v3 for analysis 'an_42'." | |
| } | |
| ``` | |
| ## Traceability | |
| > Renamed from `observability` (team decision 2026-07-06) so it is never confused with the internal Langfuse *observability* stack (engineering telemetry, PII-masked). Traceability is **user-facing** provenance β real tool args, output previews, and the executed query β shown alongside the answer. | |
| ### `GET /api/v1/traceability` | |
| Returns user-facing provenance for one assistant answer. | |
| The frontend should call this after the chat/help stream emits `done`, using the `message_id` from the `done` event. The row is written **before** `done`, so an immediate GET returns `200` (no polling race). A `404` means the id is unknown or the turn errored before completing (error turns never produce a row). | |
| Query params: | |
| | Query | Required | Description | | |
| | --- | --- | --- | | |
| | `analysis_id` | Yes | Analysis identifier. | | |
| | `message_id` | Yes | Assistant answer identifier returned by the stream. | | |
| Example: | |
| ```text | |
| GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1 | |
| ``` | |
| `intent` values the frontend may see: `chat` Β· `help` Β· `check` Β· `unstructured_flow` Β· `structured_flow` Β· `out_of_scope` Β· `blocked` (`blocked` = input-guard or Azure content-filter refusal; `chat` also covers the greeting fast-path and cache replays). | |
| Field rules: | |
| - `planning`: present only when the planner ran (`structured_flow`); otherwise `null`. | |
| - `thinking`: **always `null` in v1** β our agents are plain chat completions with no native reasoning output, and synthesizing it post-hoc would be unfaithful. The field stays in the payload so it can be populated later without a contract change. | |
| - `tool_calls`: every invoked tool with `summary` (plain-English one-liner), `input`, `output`, `status`, `task_id` (nullable), and `error` (nullable); empty for chat / help / greeting / refusal paths. `input`/`output` are the raw tool I/O (opaque ids) β render them in a collapsible "technical details" section, not the headline; use `summary` for the headline. | |
| - `data_used`: one entry per `retrieve_data` call, resolved to **real names** for display (present only when a structured pull ran; empty otherwise). Split into `columns_read` (columns read straight from the user's data, each tagged with its `roles`) and `output_columns` (`kind: "column"` = read from data, `kind: "computed"` = calculated, carrying a `formula` and **no id**). Also carries `tables` (all touched, incl. join targets), `joins`, `filters` (with a plain-language `description`), `group_by`, `order_by`, `limit`, `rows_returned`, and the executed `query`. Built by deterministic catalog lookup β no LLM. | |
| - **`id` fields are machine-only.** Every `id` in `data_used` (`source.id`, `tables[].id`, `columns_read[].id`) is for linking/reconciliation/audit β **the frontend must never render it.** Show `name` (qualified as `table.name`). A `computed` output column has no id by design. | |
| - `sources`: required for retrieval flows; empty for chat / help / refusal paths and for `check`. Database sources also carry `source_name` (the DB's real name) and `tables` (every table touched). | |
| - `thinking`, `filters[].description`, `tool_calls[].summary` are built from fixed templates, never an LLM β traceability adds no latency or token cost and cannot hallucinate. | |
| - The payload also carries an internal `user_id` (ownership); the frontend may ignore it. | |
| - Truncation: `preview` β€ 5 rows; any string inside `input`/`output`/`preview`/`snippet` β€ 300 chars (executed `query` β€ 2000); rows beyond the preview are dropped (`row_count` is preserved). | |
| - **PII redaction in `preview` (added 2026-07-24).** Cells belonging to a column the catalog flags as PII are replaced with the literal string `"[redacted]"`. When any cell was redacted the output also carries `pii_masked`: the list of column names affected β render that so the user understands the blanks are deliberate, not missing data. The **column headers are never redacted**, only the values. This affects the stored provenance record only: the chat answer itself is generated from the real values, so a question like "list our top customers" still answers normally. Records written before this date have no `pii_masked` key and are returned unchanged. | |
| Response `200` for `structured_flow`: | |
| ```json | |
| { | |
| "analysis_id": "an_42", | |
| "message_id": "msg_88f1", | |
| "user_id": "user_7", | |
| "intent": "structured_flow", | |
| "generated_at": "2026-07-06T03:21:09.114Z", | |
| "planning": { | |
| "goal_restated": "Find which regions drive revenue and why Q1 dipped.", | |
| "assumptions": [], | |
| "steps": [ | |
| { | |
| "step": 1, | |
| "stage": "data_understanding", | |
| "objective": "Inventory the sales source", | |
| "status": "success", | |
| "tools_used": ["check_data"] | |
| }, | |
| { | |
| "step": 2, | |
| "stage": "modeling", | |
| "objective": "Aggregate revenue by region", | |
| "status": "success", | |
| "tools_used": ["retrieve_data", "analyze_aggregate"] | |
| } | |
| ] | |
| }, | |
| "thinking": null, | |
| "tool_calls": [ | |
| { | |
| "order": 1, | |
| "task_id": null, | |
| "name": "check_data", | |
| "summary": "Inspected your data source structure", | |
| "input": { "source_hint": "structured" }, | |
| "output": { | |
| "kind": "table", | |
| "columns": ["source_id", "name", "source_type", "table_count"], | |
| "row_count": 1, | |
| "preview": [["src_sales_db", "orders", "schema", 1]] | |
| }, | |
| "status": "success", | |
| "error": null | |
| }, | |
| { | |
| "order": 2, | |
| "task_id": null, | |
| "name": "retrieve_data", | |
| "summary": "Retrieved 4 rows across 2 columns from orders", | |
| "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } }, | |
| "output": { | |
| "kind": "table", | |
| "columns": ["region", "total"], | |
| "row_count": 4, | |
| "preview": [["Central", 1210000], ["East", 740000]] | |
| }, | |
| "status": "success", | |
| "error": null | |
| } | |
| ], | |
| "data_used": [ | |
| { | |
| "source": { "id": "src_sales_db", "name": "sales db", "type": "schema" }, | |
| "tables": [ { "id": "orders", "name": "orders", "role": "base" } ], | |
| "joins": [], | |
| "columns_read": [ | |
| { "id": "c_region", "name": "region", "table": "orders", "data_type": "string", "pii": false, "roles": ["selected", "grouped"] }, | |
| { "id": "c_amount", "name": "amount", "table": "orders", "data_type": "decimal", "pii": false, "roles": ["aggregated"] } | |
| ], | |
| "output_columns": [ | |
| { "name": "region", "kind": "column", "from": "orders.region" }, | |
| { "name": "total", "kind": "computed", "from": "orders.amount", "formula": "SUM(orders.amount)" } | |
| ], | |
| "filters": [], | |
| "group_by": ["orders.region"], | |
| "order_by": [], | |
| "limit": null, | |
| "rows_returned": 4, | |
| "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region" | |
| } | |
| ], | |
| "sources": [ | |
| { | |
| "type": "database", | |
| "source_id": "src_sales_db", | |
| "source_name": "sales db", | |
| "name": "orders", | |
| "tables": ["orders"], | |
| "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region", | |
| "detail": { | |
| "table": "orders", | |
| "row_count": 4 | |
| } | |
| } | |
| ] | |
| } | |
| ``` | |
| > Note: `tool_calls[].input` is the raw compiled query IR (opaque `column_id`/`table_id`) β the technical layer. **`data_used` is the user-facing layer**: the same pull resolved to real names, with computed columns (e.g. `total`) flagged `kind: "computed"` so they are never shown as if they were columns in the user's database. Every `id` there is machine-only. | |
| Response `200` for `unstructured_flow`: | |
| ```json | |
| { | |
| "analysis_id": "an_42", | |
| "message_id": "msg_55", | |
| "user_id": "user_7", | |
| "intent": "unstructured_flow", | |
| "generated_at": "2026-07-06T03:40:02.001Z", | |
| "planning": null, | |
| "thinking": null, | |
| "tool_calls": [ | |
| { | |
| "order": 1, | |
| "task_id": null, | |
| "name": "retrieve_knowledge", | |
| "input": { "query": "technology stack used in this project" }, | |
| "output": { "kind": "documents", "row_count": 4 }, | |
| "status": "success", | |
| "error": null | |
| } | |
| ], | |
| "sources": [ | |
| { | |
| "type": "document", | |
| "document_id": "doc_7", | |
| "filename": "tech_handbook.pdf", | |
| "page_label": "12", | |
| "query": "technology stack used in this project", | |
| "snippet": "The backend is built on FastAPI with async SQLAlchemy...", | |
| "score": 0.83 | |
| } | |
| ] | |
| } | |
| ``` | |
| Response `200` for chat / greeting / help / refusals (`out_of_scope`, `blocked`): | |
| ```json | |
| { | |
| "analysis_id": "an_42", | |
| "message_id": "msg_12", | |
| "user_id": "user_7", | |
| "intent": "chat", | |
| "generated_at": "2026-07-06T03:05:00.000Z", | |
| "planning": null, | |
| "thinking": null, | |
| "tool_calls": [], | |
| "sources": [] | |
| } | |
| ``` | |
| Response `404`: | |
| ```json | |
| { | |
| "detail": "No traceability for message 'msg_88f1' yet." | |
| } | |
| ``` | |
| Frontend rendering guidance: | |
| - Render traceability separately from the streamed answer. | |
| - Default state can be collapsed. | |
| - Show planning, tool calls, and sources as separate sections. | |
| - Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states. | |
| ## Charts | |
| > Added 2026-07-13 (SPINE_V2_PLAN Β§4.4/Β§4.5, S2 visualization). A chart is a `render_chart` tool output, planner-selected only when the user explicitly asks to plot/visualize. Delivery mirrors traceability: a Python-owned store plus a dedicated `GET` endpoint; the streamed answer (SSE) stays text-only β charts are never embedded in `chunk`. | |
| ### `GET /api/v1/charts` | |
| Returns every chart produced by `render_chart` during one assistant answer. | |
| The frontend should call this after the chat stream emits `done`, using the `message_id` from the `done` event β the same fetch-on-`done` pattern as traceability; the row(s) are written **before** `done`, so there is no polling race. | |
| > **Updated 2026-07-13 (lead review):** lookup is now by **`message_id` alone** (it is a server-minted UUID β globally unique; `analysis_id` was removed from the query), and every response is **HTTP 200 with an explicit `status` marker** instead of a bare list: `success` (β₯1 chart), `empty` (the turn completed but produced no charts β the common case), `not_found` (no completed turn is known for this `message_id`: mistyped/stale id, or an error turn). *(Note the asymmetry: `GET /api/v1/traceability` still takes `analysis_id` + `message_id` β aligning it is a separate change if wanted.)* | |
| Query params: | |
| | Query | Required | Description | | |
| | --- | --- | --- | | |
| | `message_id` | Yes | Assistant answer identifier returned by the stream's `done` event. | | |
| Example: | |
| ```text | |
| GET /api/v1/charts?message_id=88f10c3a-6f03-4204-bf98-41ffc20388b2 | |
| ``` | |
| The `dataeyond.chart.v1` envelope (the shape of `charts[].spec`, verbatim from `render_chart`): | |
| ```json | |
| { | |
| "schema": "dataeyond.chart.v1", | |
| "chart_type": "bar", | |
| "title": "Revenue by region", | |
| "plotly": { | |
| "data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }], | |
| "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}}, | |
| "yaxis": {"title": {"text": "revenue"}} } | |
| } | |
| } | |
| ``` | |
| Frontend renders it with `Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. v1 chart types: `bar`, `line`, `pie`, `scatter`. | |
| Response `200` (`status: success` β β₯1 chart): | |
| ```json | |
| { | |
| "status": "success", | |
| "message": "1 chart(s) for this message.", | |
| "count": 1, | |
| "charts": [ | |
| { | |
| "chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e", | |
| "chart_type": "bar", | |
| "title": "Revenue by region", | |
| "spec": { | |
| "schema": "dataeyond.chart.v1", | |
| "chart_type": "bar", | |
| "title": "Revenue by region", | |
| "plotly": { | |
| "data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }], | |
| "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}}, | |
| "yaxis": {"title": {"text": "revenue"}} } | |
| } | |
| }, | |
| "created_at": "2026-07-13T03:21:09.114Z" | |
| } | |
| ] | |
| } | |
| ``` | |
| Response `200` (`status: empty` β chartless turn, the common case; not an error): | |
| ```json | |
| { | |
| "status": "empty", | |
| "message": "This message completed without producing charts.", | |
| "count": 0, | |
| "charts": [] | |
| } | |
| ``` | |
| Response `200` (`status: not_found` β no completed turn known for this id): | |
| ```json | |
| { | |
| "status": "not_found", | |
| "message": "No completed turn is known for this message_id.", | |
| "count": 0, | |
| "charts": [] | |
| } | |
| ``` | |
| Field rules: | |
| - `status` is the outcome marker: `success` | `empty` | `not_found` (always HTTP 200 β the FE fetches unconditionally, so a missing turn is a data state, not a transport failure). `empty` vs `not_found` is decided against the turn's traceability row (written before `done`), so `not_found` reliably means "this id never completed a turn". | |
| - `message` is a human-readable line for logs/debugging β do not parse it; branch on `status`. | |
| - `spec` is the full `dataeyond.chart.v1` envelope, unmodified β it is the source of truth, not a projection; render straight from it. | |
| - `chart_type` / `title` are copied out of `spec` for convenience (list rendering without parsing `spec`); `title` may be `null`. | |
| - A turn can produce more than one chart (multiple `render_chart` calls in the same plan); `charts` is ordered by creation time. | |
| - The payload carries no `user_id` / `analysis_id` β charts are keyed by `message_id` alone. | |
| > **DDL note (Harry / dedorch migration):** the original manual index is `(analysis_id, message_id)`, which does not serve a `message_id`-only lookup. Additive index for the migration (also safe to run manually now): | |
| > ```sql | |
| > CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id); | |
| > ``` | |
| Frontend rendering guidance: | |
| - Fetch unconditionally on every `done` β see the SSE note above (no `chart_count` hint yet); an empty `charts[]` means render nothing extra. | |
| - Render each chart under the assistant message it belongs to, via `Plotly.newPlot`. | |
| - "Chart iteration" v1 = a follow-up chat turn (e.g. "make it a line chart") β the planner re-emits `render_chart` with patched args. There is no separate edit endpoint. | |
| ## Knowledge | |
| > Added 2026-09-07 (KNOWLEDGE_SURFACE_PLAN.md). Turns a user's uploaded documents into reviewable knowledge artifacts: glossary terms, rules, formulas and one domain context per document. **Admin-shaped, not part of the chat flow** β no chat endpoint calls any of this. | |
| > | |
| > **Corrected 2026-09-14:** the line that stood here ("the analytics path does not yet read what it produces") is out of date β the consumption path was armed 2026-09-08 (DEV_PLAN Β§0.8 K5/K9/K11). The planner reads the company's approved terminology on every slow-path turn. **Approved entries only**, so it stays inert until an expert works the queue. | |
| **Asynchronous by necessity.** Parsing a nine-page document takes ~80 s and extracting from it ~170 s, far past any HTTP timeout, so ingestion admits a batch and returns immediately. **Polling, not SSE** (decided 2026-09-07): a job row survives a page refresh and a dropped connection, a stream does not, and a four-minute stream would hold a worker for its duration. | |
| **Scope.** Knowledge belongs to the **company**, not the uploader: two employees' documents are one body of knowledge and must share a glossary. Every row carries `scope_id` (the tenant) beside `user_id` (who uploaded). **`scope_id` is never accepted from the caller** β it is derived server-side from the user's company. Accepting a tenant key on an unauthenticated route would be a cross-tenant read with extra steps. One click currently covers **the caller's own documents**; company-wide is a one-line change behind a single resolver when wanted. | |
| **Cost controls.** This is the only endpoint group that spends LLM budget, so it carries four: a rate limit (5/minute), one running job per company, a pre-flight spend ceiling (the free stages build the exact prompts and quote a real number before anything is sent), and content-addressed idempotency β re-ingesting an unchanged document re-spends nothing. **None of them identifies the caller**; see Authentication. | |
| ### `POST /api/v1/knowledge/ingest` | |
| Extract knowledge from a user's documents. Returns `202` immediately; poll the job. | |
| Body: | |
| | Field | Required | Description | | |
| | --- | --- | --- | | |
| | `user_id` | Yes | The caller. Must be a UUID (422 otherwise). Scope is derived from it. | | |
| | `document_ids` | No | Omit for the one-click case: every parseable document this user owns. Named ids are still filtered by owner and type. | | |
| | `reparse` | No | Re-parse even when the content hash matches a stored artifact. Default `false` β on by default it silently re-spends. | | |
| | `review_mode` | No | **Added 2026-09-14.** `expert` (default) or `llm`. Who rules on what this batch produces. | | |
| Only `pdf` and `docx` are parsed; spreadsheets and text files belong to the structured-data path and come back in `skipped` **with a reason**, never silently dropped. | |
| **`review_mode` (2026-09-14).** `expert` leaves every entry waiting for a human. `llm` approves | |
| them on arrival with **no reviewer step and no second opinion** β the entries were written by a | |
| model and are accepted as-is; there is no second model call. What the FE must surface when | |
| offering it: choosing `llm` records `authorised_by` against the caller, so an entry approved this | |
| way can always answer *who allowed this*. Default is `expert`, so nobody gets LLM approval by | |
| accident. Entries are written through the ordinary append-only review path, so an expert can | |
| still disagree with any of them afterwards. | |
| `reused: true` means a job for this company was already running and is being returned instead of a second one starting β a double click is not an error. | |
| ```json | |
| { | |
| "job_id": "8f3cβ¦", "status": "queued", "stage": null, | |
| "n_requested": 1, "reused": false, | |
| "skipped": [{"document_id": "120aβ¦", "filename": "pa.csv", "reason": "unsupported_type:csv"}], | |
| "message": "Ingesting 1 document(s)." | |
| } | |
| ``` | |
| ### `GET /api/v1/knowledge/jobs/{job_id}` | |
| Job state and per-document outcomes. **404** when the id is unknown β unlike `GET /charts`, whose absence is normal; a `job_id` was handed to the caller, so its absence is exceptional. | |
| `status` is `queued` Β· `running` Β· `succeeded` Β· `partial` Β· `failed` Β· `blocked`. Two are worth reading carefully: | |
| - **`partial`** β some documents landed and some did not. One unreadable PDF must not fail a batch of twenty, so `results[]` carries each document's outcome. | |
| - **`blocked`** β **not a failure.** The pre-flight estimate exceeded the spend ceiling and the job declined to continue. It is the one outcome an operator can act on. | |
| `stage` (`parsing` Β· `filtering` Β· `estimating` Β· `extracting` Β· `persisting`) is ordered by cost: everything up to and including `estimating` is free, and the ceiling is enforced between `estimating` and `extracting`. | |
| > **Updated 2026-09-11 β the estimate is now returned, and `blocked` is now reachable.** | |
| > | |
| > `estimated_cost` was on the job model from the start and was **never populated**, so the field a | |
| > caller would poll to answer "what will this cost, and why was that document refused" always came | |
| > back `null`. It now carries the pre-flight shape of the batch, per document and in total. A | |
| > wholly-refused batch also settles **`blocked`** rather than `failed` β previously every spend | |
| > refusal was reported as a generic failure, which hid the one outcome with an action attached. | |
| > | |
| > **Spend is driven by CLUSTER COUNT** β one LLM call per term cluster β not by page count, so the | |
| > per-document breakdown is what tells an operator which document to split. There are now **two | |
| > ceilings**, and `blocked_by` says which one bound: | |
| > | |
| > | `blocked_by` | Meaning | The operator's move | | |
| > |---|---|---| | |
| > | `clusters` | Too many term clusters (exact, known free before spending) | Split the document | | |
| > | `prompt_tokens` | Too many estimated prompt tokens | The document is simply long | | |
| > | |
| > ```json | |
| > { | |
| > "status": "blocked", | |
| > "estimated_cost": { | |
| > "clusters": 2083, "total_calls": 2150, "estimated_prompt_tokens": 5100000, | |
| > "per_document": { | |
| > "d1": {"clusters": 83, "total_calls": 106, "estimated_prompt_tokens": 201751}, | |
| > "d2": {"clusters": 2000, "total_calls": 2044, "estimated_prompt_tokens": 4898249} | |
| > }, | |
| > "blocked": {"d2": "clusters"} | |
| > }, | |
| > "results": [ | |
| > {"document_id": "d2", "status": "failed", "blocked_by": "clusters", | |
| > "error": "estimated 2,000 term clusters exceeds the ceiling of 750 (267% of it) β nothing was spent; 2,044 LLM calls were declined"} | |
| > ] | |
| > } | |
| > ``` | |
| > | |
| > Each `results[]` entry also carries `estimate` β **on success as well as refusal**, because a | |
| > number a caller only ever sees when it is already too late cannot be planned against. | |
| > | |
| > `estimate.genre` is advisory document classification (`standard` Β· `diagnostic` Β· `conceptual` Β· | |
| > `unknown`) with a `well_served` flag. **Nothing is routed on it yet**; it exists so a caller can be | |
| > told *before* spending that a troubleshooting manual will yield almost no definitions. | |
| ### `GET /api/v1/knowledge/jobs` | |
| Recent jobs for one `user_id`, newest first β lets a caller recover after a refresh without having stashed a `job_id`. | |
| ### `GET /api/v1/knowledge/documents` | |
| Parsed artifacts stored for the caller's company: page and chunk counts, parser, `schema_version`, `content_hash`. This is what a re-extraction is chosen from. | |
| ### `POST /api/v1/knowledge/documents/{knowledge_document_id}/extract` | |
| Re-run extraction over a **stored** artifact. Same job machinery and same polling route, but no parsing β prompt retuning is the common case and must not re-pay the parser for a byte-identical artifact. Takes `user_id` as a query parameter. | |
| ### `GET /api/v1/knowledge/entries` | |
| Candidate entries, filterable by `doc_id` and `kind` (`glossary` Β· `rule` Β· `formula` Β· `domain`), each annotated with the current expert decision if one exists. `payload` is the full typed entry and is the source of truth; the promoted fields beside it (`label`, `mention_count`, `diff_status`, β¦) exist so a list can be ordered without opening it. | |
| `diff_status` (`new` Β· `duplicate` Β· `conflicting`) is the **cross-document** relationship. Entries append and never merge: the same term in two documents keeps two ids, because deciding they are one concept is a judgement about the world rather than a string comparison. | |
| ### `GET /api/v1/knowledge/queue` | |
| The review queue β conflicts first, then descending mention count, each row carrying a plain-language `review_reason`. | |
| **Derived at read time, never stored:** persisting an ordering creates a second truth that drifts from the entries it came from. | |
| β οΈ Every row carries **`row_kind`** and a consumer **must switch on it**. A `key_parameter` row has a `surface` and no `term`, no definition and no mention count; a `term` row has all of them. Provenance carries both `page` (0-based, as the parser reports) and **`page_no` (1-based)** β **a UI must render `page_no`**. | |
| ### `POST /api/v1/knowledge/entries/{entity_id}/reviews` | |
| Record one expert decision: `approved` Β· `edited` Β· `rejected`. **Append-only** β a changed mind is a new row, never an update. | |
| Keyed on **`entity_id`**, not the entry's row id: `entity_id` is deterministic and content-derived, so it survives re-extraction while row ids do not. That is what lets approvals outlive a prompt retune. `content_hash` is **derived server-side** from the artifact the entry came from, never taken from the request, so a stale client cannot attribute a ruling to the wrong version. `edited` requires `edited_payload` (422 otherwise); an unknown `entity_id` is 404. | |
| **This closes the loop.** Approved and edited entries become the baseline the *next* extraction diffs against, so working the queue improves the following run rather than only annotating this one. | |
| > β οΈ **Injection path, recorded not fixed.** Because an `edited_payload` reaches a future prompt through that baseline, and this surface is unauthenticated, anyone who can reach it can influence later extractions. This is the strongest argument for the forwarded identity in DEV_PLAN #43, and it should be closed before an expert relies on the queue. | |