API_CONTRACT_BE_PYTHON.md

# Backend Agentic Service API Contract

✅ LIVE — CANONICAL: the contract of record for the FE-facing surface. This is the one that governs endpoint behaviour. The frozen 2026-06-30 snapshot sent to Harry is docs/archive/API_ENDPOINTS_RESTRUCTURE.md — do not implement from that one.

This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, and traceability data shown alongside chat answers.

Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.

Overview

The Python backend owns the generative AI interaction surface:

1. Stream chat answers from the AI agent.
2. Execute tool-style actions for help and report generation.
3. Return report versions and report details.
4. Return traceability/provenance for a completed assistant answer.

The frontend uses this service during the analysis conversation flow:

1. User sends a chat message.
2. Frontend calls POST /api/v2/chat/stream and renders the streamed answer.
3. When the stream emits done, frontend stores or reads the returned message_id.
4. Frontend calls GET /api/v1/traceability for planning, tool calls, and source provenance.
5. Frontend calls /api/v1/tools/help for guided help and /api/v1/tools/report for report generation.

Endpoint Summary

Method	Path	Purpose
POST	/api/v2/chat/stream	Stream an AI chat answer for one analysis conversation.
GET	/api/v1/tools/list	List available frontend tools.
POST	/api/v1/tools/help	Stream contextual help for the current analysis conversation.
POST	/api/v1/tools/report	Generate and persist a new report version.
GET	/api/v1/tools/report/{analysis_id}	List report versions for an analysis.
GET	/api/v1/tools/report/{analysis_id}/records	List analysis records for report curation (added 2026-07-09).
GET	/api/v1/tools/report/{analysis_id}/readiness	Report-readiness signal for the Generate-Report button (added 2026-07-09).
GET	/api/v1/tools/report/{analysis_id}/{version}	Retrieve one report version.
GET	/api/v1/traceability	Retrieve provenance for one assistant answer.
GET	/api/v1/charts	Retrieve chart(s) produced by render_chart for one assistant answer (added 2026-07-13).
POST	/api/v1/knowledge/ingest	Extract knowledge from a user's documents — async batch (added 2026-09-07).
GET	/api/v1/knowledge/jobs/{job_id}	Poll one ingestion job (added 2026-09-07).
GET	/api/v1/knowledge/jobs	Recent ingestion jobs for a user (added 2026-09-07).
GET	/api/v1/knowledge/documents	Parsed artifacts stored for the caller's company (added 2026-09-07).
POST	/api/v1/knowledge/documents/{id}/extract	Re-extract a stored artifact, no re-parsing (added 2026-09-07).
GET	/api/v1/knowledge/entries	Candidate knowledge entries (added 2026-09-07).
GET	/api/v1/knowledge/queue	The expert review queue (added 2026-09-07).
POST	/api/v1/knowledge/entries/{entity_id}/reviews	Record an expert decision (added 2026-09-07).

Authentication

None. The live surface is unauthenticated — do not expose it beyond the demo.

The service accepts user_id and analysis_id as ordinary request fields with no verification, and there is no caller authentication in front of any endpoint.

History (2026-07-23 → 2026-07-27). A shared service-secret gate (X-Dataeyond-Service-Secret, router-level, constant-time compare) shipped 2026-07-23, inert until dataeyond__service__secret was set. It was unwired on 2026-07-27 (DEV_PLAN #37): the only caller of this service is the browser SPA, which cannot be changed to send the header, so the gate could never be armed without a 401 outage. The gate code is parked in src/middlewares/service_auth.py (not deleted) and restores in one edit if the caller situation changes.

Updated 2026-09-07 — the /knowledge group raises the stakes. Those endpoints are writes that spend LLM budget, and one of them (POST …/reviews) accepts content that reaches a later prompt. They are bounded by a rate limit, one job per company, a pre-flight spend ceiling and content-addressed idempotency — none of which identifies the caller. The posture is recorded rather than inherited: see the Knowledge section.

The real fix (DEV_PLAN #43). Per-user authorization requires a verified identity forwarded by the Go service; once Go forwards a token, user_id comes from the verified claims rather than the request body. Until then the user_id predicates in the stores (F-1) are defensive-in-depth only, not an access control.

Common Concepts

Identifiers

• user_id: user identifier passed by the frontend.
• analysis_id: analysis conversation identifier.
• message_id: assistant answer identifier used to correlate chat streaming with traceability. Server-minted, never accepted from the client. Updated 2026-07-09: it is a UUID string (e.g. 77f06761-0fdf-4cc5-84f8-5f81bcbb6f84), matching the shape of Go's analyses_messages.id. The msg_… values in the examples below are illustrative placeholders only.

Server-Sent Events

Chat and help endpoints return text/event-stream.

Frontend should parse events by event name and data payload. Blank lines separate SSE events.

Common event types:

Event	Data	Meaning
sources	JSON array	Always [] — sources moved to GET /api/v1/traceability (KM-691). Event kept for backward-compat; read sources[] from the traceability call.
status	text	Optional progress update for slower paths.
chunk	text	Answer text fragment. Concatenate chunks in order.
done	JSON object	Terminal success event. Includes message_id.
error	text	Terminal error event. Stream stops after this.

The stream carries answer text only. Planning, tool call details, and full provenance are fetched from GET /api/v1/traceability after the stream is done.

Charts (added 2026-07-13, SPINE_V2_PLAN §4.5): the done event is unchanged by charts — no additive chart_count/chart_ids field yet (open question owned by Harry). Until that's resolved, the frontend fetches GET /api/v1/charts unconditionally on every done, the same fetch-on-done pattern already used for traceability. SSE order and every existing field stay exactly as documented above; sources stays [].

Chat

POST /api/v2/chat/stream

Streams an AI answer for one user message in an analysis conversation.

Request body:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}

Fields:

Field	Required	Description
user_id	Yes	User identifier.
analysis_id	Yes	Analysis conversation identifier.
message_id	—	Updated 2026-07-09: not a request field. Python always mints the id server-side and returns it on done; any caller-sent value is ignored (server-authoritative — open-Q #1).
message	Yes	User message text.

Response: text/event-stream.

Example structured answer:

event: sources
data: []

event: status
data: Planning analysis...

event: status
data: Running 3 steps...

event: chunk
data: Total sales by region last quarter: 

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}

Example simple chat answer:

event: sources
data: []

event: chunk
data: I'm your AI data analyst. Connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}

Behavior notes:

• Greeting and farewell messages may use a fast canned path.
• Stateless chat intent may use a 1-hour Redis response cache.
• The router may classify messages into intents such as chat, help, check, unstructured_flow, or structured_flow.
• sources in the stream is always [] (KM-691) — read the real sources[] from GET /api/v1/traceability after done.
• status events are optional and should be safe for the frontend to ignore.
• Fixed 2026-07-23 — sources is now genuinely always first, on every path. It previously did not match this document: the check and router-help turns emitted no sources event at all, and the structured_flow (slow) path emitted status before sources, inverting the documented order on exactly the turns that take longest. A frontend that initializes per-turn state on sources therefore never initialized on check/help turns and initialized late on slow turns. The change is purely additive — no event was removed or renamed, and sources is still [] — so no frontend change is required; behaviour simply now matches what this contract always said.
• analysis_id must be a UUID (2026-07-23). POST /api/v2/chat/stream and POST /api/v1/tools/help now return 422 for a non-UUID analysis_id instead of accepting it. It was never actually usable: analyses.id is uuid NOT NULL, so a non-UUID id silently matched no analysis and the turn ran with no state — help, readiness, the report write-back, and the traceability/chart rows all quietly no-op'd while the user still saw a normal answer. The frontend passes Go-issued analysis ids, which are UUIDs, so this should never fire in practice.

Tools

GET /api/v1/tools/list

Returns the deterministic list of tools available to the frontend.

Request: none.

Response 200:

{
  "count": 1,
  "tools": [
    {
      "command": "/help",
      "name": "help",
      "type": "skill",
      "description": "Show what the assistant can do and guide your next step."
    }
  ]
}

The catalog is /help only (KM-711). /report was removed as a slash command — report generation is a right-side Generate button, not a / command. The report HTTP endpoint (POST /api/v1/tools/report) still exists; the button calls it.

Tool item shape:

{
  "command": "/help",
  "name": "help",
  "type": "skill",
  "description": "Show what the assistant can do and guide your next step."
}

Frontend behavior:

• Surface /help in the slash menu.
• Surface report generation as a button or explicit UI action.

POST /api/v1/tools/help

Streams contextual guidance for the current analysis conversation.

Request body:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}

Response: text/event-stream using the same event shape as chat.

Help responses usually emit sources: [] and no status pings.

Example:

event: sources
data: []

event: chunk
data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}

Reports

POST /api/v1/tools/report

Generates, persists, and returns a new report version for an analysis.

Query params:

Query	Required	Description
analysis_id	Yes	Analysis identifier.
user_id	Yes	User identifier.
exclude_record_ids	No	Record ids to leave out of this version (repeat the param per id). Added 2026-07-09; get ids from GET /tools/report/{analysis_id}/records. Excluded runs are listed in the report's "Excluded Analyses" section. Excluding every substantive record returns 409.

Example:

POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c&exclude_record_ids=rec_a1&exclude_record_ids=rec_c3

Status codes:

Status	Meaning
201	New report version generated.
409	Report floor/precondition not met.
500	Generation or persistence failed.

Response 201:

{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
  "bq_answers": [
    {
      "question": "Which regions contribute most to total revenue?",
      "answer": "The Central region leads with 38% of total revenue.",
      "status": "answered",
      "record_ids": ["rec_a1"]
    },
    {
      "question": "Did any region decline quarter-over-quarter?",
      "answer": "Yes — the West region fell 12% QoQ.",
      "status": "answered",
      "record_ids": ["rec_b2"]
    }
  ],
  "findings": [
    {
      "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"],
      "supporting_data": null
    },
    {
      "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"],
      "supporting_data": null
    }
  ],
  "caveats": [
    {
      "text": "March data for the East region was partially missing, around 6% of rows.",
      "record_ids": ["rec_b2"]
    }
  ],
  "open_questions": [
    {
      "text": "What drove the West region's QoQ decline?",
      "record_ids": ["rec_b2"]
    }
  ],
  "unresolved": [
    {
      "text": "Correlate churn with tenure — churn column not found in the source.",
      "record_ids": ["rec_d4"]
    }
  ],
  "excluded": [],
  "evidence_tables": {
    "rec_a1": [
      {
        "title": "Aggregate revenue by region",
        "columns": ["region", "total_revenue"],
        "rows": [["Central", "18321"], ["West", "9954"]],
        "truncated": false
      }
    ]
  },
  "data_sources": [
    {
      "source_id": "src_sales_db",
      "name": "orders",
      "source_type": "postgres",
      "detail": {
        "tables": ["orders"],
        "row_count": 48213,
        "columns": ["region", "amount", "ordered_at"]
      }
    }
  ],
  "method_steps": [
    {
      "task_id": "t1",
      "stage": "data_understanding",
      "objective": "Inventory the sales source",
      "status": "success",
      "tools_used": ["check_data"]
    },
    {
      "task_id": "t2",
      "stage": "modeling",
      "objective": "Aggregate revenue by region",
      "status": "success",
      "tools_used": ["analyze_aggregate"]
    }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
}

Response 409:

{
  "detail": "Not ready to generate a report - still needs at least one completed analysis."
}

Report v2 fields (added 2026-07-09; all default-empty, so older stored reports read back unchanged):

• bq_answers — one entry per business question. status is answered | partial | unanswered; record_ids cite the backing analyses. Written in the analysis's language (Indonesian objective → Indonesian answers).

• unresolved — runs that were attempted but produced no usable evidence (every analyze_* step failed). Not part of the findings body.

• excluded — runs the caller excluded via exclude_record_ids.

• evidence_tables — record_id → small result tables copied from the run's stored outputs (max 3 tables per record, max 10 rows each; truncated: true when rows were capped). Rendered as markdown tables under the matching Key Findings group in rendered_markdown. PII redaction (added 2026-07-24): cells in a column the catalog flags as PII are replaced with "[redacted]"; column headers are kept. A report is permanent and versioned, so this is where an unmasked customer name would have lasted longest. The narrative (executive_summary, findings, bq_answers) is generated from the real values and is unaffected — only the raw evidence dump is redacted. Reports generated before this date are unchanged.

• charts (added 2026-07-14) — record_id → dataeyond.chart.v1 envelopes (see §Charts) copied verbatim from the run's stored outputs (max 3 per record). rendered_markdown gains an ## EDA section where each chart appears as a fenced block the frontend renders with plotly.js:

  ```plotly
  {
    "schema": "dataeyond.chart.v1",
    "chart_type": "bar",
    "title": "…",
    "plotly": { "data": [ … ], "layout": { … } }
  }
  ```
  

  The fence content is the full envelope (same shape as charts[].spec on GET /api/v1/charts) — the FE hook parses it and renders Plotly.newPlot(el, parsed.plotly.data, parsed.plotly.layout). A bold caption line (chart title) precedes each fence.

Precondition:

• Reports require at least one completed analysis record for the session (updated 2026-07-14: a run whose render_chart succeeded counts — a chart-only session can generate a report).
• If slow-path analysis recording is disabled, report generation can return 409 by design.

GET /api/v1/tools/report/{analysis_id}

Lists report versions for one analysis, oldest first.

Response 200:

[
  {
    "report_id": "1b2c3d4e",
    "version": 1,
    "generated_at": "2026-06-24T15:02:11Z",
    "record_count": 1
  },
  {
    "report_id": "8f3a2b1c",
    "version": 2,
    "generated_at": "2026-06-25T09:14:33Z",
    "record_count": 2
  }
]

If no reports exist, returns [].

GET /api/v1/tools/report/{analysis_id}/records (added 2026-07-09)

Lists the persisted analysis runs a report would be built from, oldest first. The frontend shows this before generating so the user can deselect runs; the chosen ids go to POST /tools/report as exclude_record_ids.

Response 200:

[
  {
    "record_id": "rec_a1",
    "goal_restated": "Rank regions by total revenue",
    "created_at": "2026-06-30T08:55:02Z",
    "substantive": true,
    "findings_count": 2
  },
  {
    "record_id": "rec_d4",
    "goal_restated": "Correlate churn with tenure",
    "created_at": "2026-06-30T09:01:47Z",
    "substantive": false,
    "findings_count": 1
  }
]

substantive: true means the run will appear in the report's findings body; substantive: false means it will be listed in the report's unresolved JSON field instead. (Since 2026-07-09 the rendered markdown is compact and no longer includes "Attempted, Unresolved" / "Notes & Limitations" / "How This Was Analyzed" sections; the JSON fields unresolved / caveats / open_questions / method_steps are unchanged.) If no runs exist, returns [].

Changed 2026-07-23 — behavioral, non-breaking (field name, type and position unchanged). substantive previously meant the narrower "a successful analyze_* step exists". Planner recipes R2/R2b legitimately answer a question with a single aggregate retrieve_data and no analyze_* step, so under the old rule such a run was reported substantive: false, was dropped from the report body, and its business question rendered "Unanswered" in bq_answers — while the chat had answered it correctly. substantive now tracks the report body exactly. FE impact: none required. A run that previously showed false and was silently excluded may now show true and be included; no field was added, removed or retyped.

GET /api/v1/tools/report/{analysis_id}/readiness (added 2026-07-09)

Deterministic report-readiness signal for the Generate-Report button — the same producer as Help's readiness signal, including the advisory delta-since-report check, so the button, Help, and this endpoint never disagree.

Response 200:

{
  "ready": false,
  "missing": ["a new analysis since the last report"]
}

Note: POST /tools/report itself only enforces the floor (at least one completed analysis) — a new version is always allowed. The delta gap in missing is a soft warning the frontend can surface ("nothing new since the last report") without blocking the button.

GET /api/v1/tools/report/{analysis_id}/{version}

Returns one report version. Shape is the same as the 201 response from POST /api/v1/tools/report.

Response 404:

{
  "detail": "No report v3 for analysis 'an_42'."
}

Traceability

Renamed from observability (team decision 2026-07-06) so it is never confused with the internal Langfuse observability stack (engineering telemetry, PII-masked). Traceability is user-facing provenance — real tool args, output previews, and the executed query — shown alongside the answer.

GET /api/v1/traceability

Returns user-facing provenance for one assistant answer.

The frontend should call this after the chat/help stream emits done, using the message_id from the done event. The row is written before done, so an immediate GET returns 200 (no polling race). A 404 means the id is unknown or the turn errored before completing (error turns never produce a row).

Query params:

Query	Required	Description
analysis_id	Yes	Analysis identifier.
message_id	Yes	Assistant answer identifier returned by the stream.

Example:

GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1

intent values the frontend may see: chat · help · check · unstructured_flow · structured_flow · out_of_scope · blocked (blocked = input-guard or Azure content-filter refusal; chat also covers the greeting fast-path and cache replays).

Field rules:

• planning: present only when the planner ran (structured_flow); otherwise null.
• thinking: always null in v1 — our agents are plain chat completions with no native reasoning output, and synthesizing it post-hoc would be unfaithful. The field stays in the payload so it can be populated later without a contract change.
• tool_calls: every invoked tool with summary (plain-English one-liner), input, output, status, task_id (nullable), and error (nullable); empty for chat / help / greeting / refusal paths. input/output are the raw tool I/O (opaque ids) — render them in a collapsible "technical details" section, not the headline; use summary for the headline.
• data_used: one entry per retrieve_data call, resolved to real names for display (present only when a structured pull ran; empty otherwise). Split into columns_read (columns read straight from the user's data, each tagged with its roles) and output_columns (kind: "column" = read from data, kind: "computed" = calculated, carrying a formula and no id). Also carries tables (all touched, incl. join targets), joins, filters (with a plain-language description), group_by, order_by, limit, rows_returned, and the executed query. Built by deterministic catalog lookup — no LLM.
• id fields are machine-only. Every id in data_used (source.id, tables[].id, columns_read[].id) is for linking/reconciliation/audit — the frontend must never render it. Show name (qualified as table.name). A computed output column has no id by design.
• sources: required for retrieval flows; empty for chat / help / refusal paths and for check. Database sources also carry source_name (the DB's real name) and tables (every table touched).
• thinking, filters[].description, tool_calls[].summary are built from fixed templates, never an LLM — traceability adds no latency or token cost and cannot hallucinate.
• The payload also carries an internal user_id (ownership); the frontend may ignore it.
• Truncation: preview ≤ 5 rows; any string inside input/output/preview/snippet ≤ 300 chars (executed query ≤ 2000); rows beyond the preview are dropped (row_count is preserved).
• PII redaction in preview (added 2026-07-24). Cells belonging to a column the catalog flags as PII are replaced with the literal string "[redacted]". When any cell was redacted the output also carries pii_masked: the list of column names affected — render that so the user understands the blanks are deliberate, not missing data. The column headers are never redacted, only the values. This affects the stored provenance record only: the chat answer itself is generated from the real values, so a question like "list our top customers" still answers normally. Records written before this date have no pii_masked key and are returned unchanged.

Response 200 for structured_flow:

{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "user_id": "user_7",
  "intent": "structured_flow",
  "generated_at": "2026-07-06T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": [],
    "steps": [
      {
        "step": 1,
        "stage": "data_understanding",
        "objective": "Inventory the sales source",
        "status": "success",
        "tools_used": ["check_data"]
      },
      {
        "step": 2,
        "stage": "modeling",
        "objective": "Aggregate revenue by region",
        "status": "success",
        "tools_used": ["retrieve_data", "analyze_aggregate"]
      }
    ]
  },
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "check_data",
      "summary": "Inspected your data source structure",
      "input": { "source_hint": "structured" },
      "output": {
        "kind": "table",
        "columns": ["source_id", "name", "source_type", "table_count"],
        "row_count": 1,
        "preview": [["src_sales_db", "orders", "schema", 1]]
      },
      "status": "success",
      "error": null
    },
    {
      "order": 2,
      "task_id": null,
      "name": "retrieve_data",
      "summary": "Retrieved 4 rows across 2 columns from orders",
      "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } },
      "output": {
        "kind": "table",
        "columns": ["region", "total"],
        "row_count": 4,
        "preview": [["Central", 1210000], ["East", 740000]]
      },
      "status": "success",
      "error": null
    }
  ],
  "data_used": [
    {
      "source": { "id": "src_sales_db", "name": "sales db", "type": "schema" },
      "tables": [ { "id": "orders", "name": "orders", "role": "base" } ],
      "joins": [],
      "columns_read": [
        { "id": "c_region", "name": "region", "table": "orders", "data_type": "string",  "pii": false, "roles": ["selected", "grouped"] },
        { "id": "c_amount", "name": "amount", "table": "orders", "data_type": "decimal", "pii": false, "roles": ["aggregated"] }
      ],
      "output_columns": [
        { "name": "region", "kind": "column",   "from": "orders.region" },
        { "name": "total",  "kind": "computed", "from": "orders.amount", "formula": "SUM(orders.amount)" }
      ],
      "filters": [],
      "group_by": ["orders.region"],
      "order_by": [],
      "limit": null,
      "rows_returned": 4,
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region"
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "source_name": "sales db",
      "name": "orders",
      "tables": ["orders"],
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": {
        "table": "orders",
        "row_count": 4
      }
    }
  ]
}

Note: tool_calls[].input is the raw compiled query IR (opaque column_id/table_id) — the technical layer. data_used is the user-facing layer: the same pull resolved to real names, with computed columns (e.g. total) flagged kind: "computed" so they are never shown as if they were columns in the user's database. Every id there is machine-only.

Response 200 for unstructured_flow:

{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "user_id": "user_7",
  "intent": "unstructured_flow",
  "generated_at": "2026-07-06T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project" },
      "output": { "kind": "documents", "row_count": 4 },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "document",
      "document_id": "doc_7",
      "filename": "tech_handbook.pdf",
      "page_label": "12",
      "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
      "score": 0.83
    }
  ]
}

Response 200 for chat / greeting / help / refusals (out_of_scope, blocked):

{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "user_id": "user_7",
  "intent": "chat",
  "generated_at": "2026-07-06T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}

Response 404:

{
  "detail": "No traceability for message 'msg_88f1' yet."
}

Frontend rendering guidance:

• Render traceability separately from the streamed answer.
• Default state can be collapsed.
• Show planning, tool calls, and sources as separate sections.
• Treat planning: null, tool_calls: [], and sources: [] as valid states.

Charts

Added 2026-07-13 (SPINE_V2_PLAN §4.4/§4.5, S2 visualization). A chart is a render_chart tool output, planner-selected only when the user explicitly asks to plot/visualize. Delivery mirrors traceability: a Python-owned store plus a dedicated GET endpoint; the streamed answer (SSE) stays text-only — charts are never embedded in chunk.

GET /api/v1/charts

Returns every chart produced by render_chart during one assistant answer.

The frontend should call this after the chat stream emits done, using the message_id from the done event — the same fetch-on-done pattern as traceability; the row(s) are written before done, so there is no polling race.

Updated 2026-07-13 (lead review): lookup is now by message_id alone (it is a server-minted UUID — globally unique; analysis_id was removed from the query), and every response is HTTP 200 with an explicit status marker instead of a bare list: success (≥1 chart), empty (the turn completed but produced no charts — the common case), not_found (no completed turn is known for this message_id: mistyped/stale id, or an error turn). (Note the asymmetry: GET /api/v1/traceability still takes analysis_id + message_id — aligning it is a separate change if wanted.)

Query params:

Query	Required	Description
message_id	Yes	Assistant answer identifier returned by the stream's done event.

Example:

GET /api/v1/charts?message_id=88f10c3a-6f03-4204-bf98-41ffc20388b2

The dataeyond.chart.v1 envelope (the shape of charts[].spec, verbatim from render_chart):

{
  "schema": "dataeyond.chart.v1",
  "chart_type": "bar",
  "title": "Revenue by region",
  "plotly": {
    "data":   [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
    "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
                "yaxis": {"title": {"text": "revenue"}} }
  }
}

Frontend renders it with Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout). v1 chart types: bar, line, pie, scatter.

Response 200 (status: success — ≥1 chart):

{
  "status": "success",
  "message": "1 chart(s) for this message.",
  "count": 1,
  "charts": [
    {
      "chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e",
      "chart_type": "bar",
      "title": "Revenue by region",
      "spec": {
        "schema": "dataeyond.chart.v1",
        "chart_type": "bar",
        "title": "Revenue by region",
        "plotly": {
          "data":   [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
          "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
                      "yaxis": {"title": {"text": "revenue"}} }
        }
      },
      "created_at": "2026-07-13T03:21:09.114Z"
    }
  ]
}

Response 200 (status: empty — chartless turn, the common case; not an error):

{
  "status": "empty",
  "message": "This message completed without producing charts.",
  "count": 0,
  "charts": []
}

Response 200 (status: not_found — no completed turn known for this id):

{
  "status": "not_found",
  "message": "No completed turn is known for this message_id.",
  "count": 0,
  "charts": []
}

Field rules:

• status is the outcome marker: success | empty | not_found (always HTTP 200 — the FE fetches unconditionally, so a missing turn is a data state, not a transport failure). empty vs not_found is decided against the turn's traceability row (written before done), so not_found reliably means "this id never completed a turn".
• message is a human-readable line for logs/debugging — do not parse it; branch on status.
• spec is the full dataeyond.chart.v1 envelope, unmodified — it is the source of truth, not a projection; render straight from it.
• chart_type / title are copied out of spec for convenience (list rendering without parsing spec); title may be null.
• A turn can produce more than one chart (multiple render_chart calls in the same plan); charts is ordered by creation time.
• The payload carries no user_id / analysis_id — charts are keyed by message_id alone.

DDL note (Harry / dedorch migration): the original manual index is (analysis_id, message_id), which does not serve a message_id-only lookup. Additive index for the migration (also safe to run manually now):

CREATE INDEX IF NOT EXISTS idx_message_charts_message ON message_charts (message_id);

Frontend rendering guidance:

• Fetch unconditionally on every done — see the SSE note above (no chart_count hint yet); an empty charts[] means render nothing extra.
• Render each chart under the assistant message it belongs to, via Plotly.newPlot.
• "Chart iteration" v1 = a follow-up chat turn (e.g. "make it a line chart") — the planner re-emits render_chart with patched args. There is no separate edit endpoint.

Knowledge

Added 2026-09-07 (KNOWLEDGE_SURFACE_PLAN.md). Turns a user's uploaded documents into reviewable knowledge artifacts: glossary terms, rules, formulas and one domain context per document. Admin-shaped, not part of the chat flow — no chat endpoint calls any of this.

Corrected 2026-09-14: the line that stood here ("the analytics path does not yet read what it produces") is out of date — the consumption path was armed 2026-09-08 (DEV_PLAN §0.8 K5/K9/K11). The planner reads the company's approved terminology on every slow-path turn. Approved entries only, so it stays inert until an expert works the queue.

Asynchronous by necessity. Parsing a nine-page document takes ~80 s and extracting from it ~170 s, far past any HTTP timeout, so ingestion admits a batch and returns immediately. Polling, not SSE (decided 2026-09-07): a job row survives a page refresh and a dropped connection, a stream does not, and a four-minute stream would hold a worker for its duration.

Scope. Knowledge belongs to the company, not the uploader: two employees' documents are one body of knowledge and must share a glossary. Every row carries scope_id (the tenant) beside user_id (who uploaded). scope_id is never accepted from the caller — it is derived server-side from the user's company. Accepting a tenant key on an unauthenticated route would be a cross-tenant read with extra steps. One click currently covers the caller's own documents; company-wide is a one-line change behind a single resolver when wanted.

Cost controls. This is the only endpoint group that spends LLM budget, so it carries four: a rate limit (5/minute), one running job per company, a pre-flight spend ceiling (the free stages build the exact prompts and quote a real number before anything is sent), and content-addressed idempotency — re-ingesting an unchanged document re-spends nothing. None of them identifies the caller; see Authentication.

POST /api/v1/knowledge/ingest

Extract knowledge from a user's documents. Returns 202 immediately; poll the job.

Body:

Field	Required	Description
user_id	Yes	The caller. Must be a UUID (422 otherwise). Scope is derived from it.
document_ids	No	Omit for the one-click case: every parseable document this user owns. Named ids are still filtered by owner and type.
reparse	No	Re-parse even when the content hash matches a stored artifact. Default false — on by default it silently re-spends.
review_mode	No	Added 2026-09-14. expert (default) or llm. Who rules on what this batch produces.

Only pdf and docx are parsed; spreadsheets and text files belong to the structured-data path and come back in skipped with a reason, never silently dropped.

review_mode (2026-09-14). expert leaves every entry waiting for a human. llm approves them on arrival with no reviewer step and no second opinion — the entries were written by a model and are accepted as-is; there is no second model call. What the FE must surface when offering it: choosing llm records authorised_by against the caller, so an entry approved this way can always answer who allowed this. Default is expert, so nobody gets LLM approval by accident. Entries are written through the ordinary append-only review path, so an expert can still disagree with any of them afterwards.

reused: true means a job for this company was already running and is being returned instead of a second one starting — a double click is not an error.

{
  "job_id": "8f3c…", "status": "queued", "stage": null,
  "n_requested": 1, "reused": false,
  "skipped": [{"document_id": "120a…", "filename": "pa.csv", "reason": "unsupported_type:csv"}],
  "message": "Ingesting 1 document(s)."
}

GET /api/v1/knowledge/jobs/{job_id}

Job state and per-document outcomes. 404 when the id is unknown — unlike GET /charts, whose absence is normal; a job_id was handed to the caller, so its absence is exceptional.

status is queued · running · succeeded · partial · failed · blocked. Two are worth reading carefully:

• partial — some documents landed and some did not. One unreadable PDF must not fail a batch of twenty, so results[] carries each document's outcome.
• blocked — not a failure. The pre-flight estimate exceeded the spend ceiling and the job declined to continue. It is the one outcome an operator can act on.

stage (parsing · filtering · estimating · extracting · persisting) is ordered by cost: everything up to and including estimating is free, and the ceiling is enforced between estimating and extracting.

Updated 2026-09-11 — the estimate is now returned, and blocked is now reachable.

estimated_cost was on the job model from the start and was never populated, so the field a caller would poll to answer "what will this cost, and why was that document refused" always came back null. It now carries the pre-flight shape of the batch, per document and in total. A wholly-refused batch also settles blocked rather than failed — previously every spend refusal was reported as a generic failure, which hid the one outcome with an action attached.

Spend is driven by CLUSTER COUNT — one LLM call per term cluster — not by page count, so the per-document breakdown is what tells an operator which document to split. There are now two ceilings, and blocked_by says which one bound:

blocked_by	Meaning	The operator's move
clusters	Too many term clusters (exact, known free before spending)	Split the document
prompt_tokens	Too many estimated prompt tokens	The document is simply long

{
  "status": "blocked",
  "estimated_cost": {
    "clusters": 2083, "total_calls": 2150, "estimated_prompt_tokens": 5100000,
    "per_document": {
      "d1": {"clusters": 83,   "total_calls": 106, "estimated_prompt_tokens": 201751},
      "d2": {"clusters": 2000, "total_calls": 2044, "estimated_prompt_tokens": 4898249}
    },
    "blocked": {"d2": "clusters"}
  },
  "results": [
    {"document_id": "d2", "status": "failed", "blocked_by": "clusters",
     "error": "estimated 2,000 term clusters exceeds the ceiling of 750 (267% of it) — nothing was spent; 2,044 LLM calls were declined"}
  ]
}

Each results[] entry also carries estimate — on success as well as refusal, because a number a caller only ever sees when it is already too late cannot be planned against.

estimate.genre is advisory document classification (standard · diagnostic · conceptual · unknown) with a well_served flag. Nothing is routed on it yet; it exists so a caller can be told before spending that a troubleshooting manual will yield almost no definitions.

GET /api/v1/knowledge/jobs

Recent jobs for one user_id, newest first — lets a caller recover after a refresh without having stashed a job_id.

GET /api/v1/knowledge/documents

Parsed artifacts stored for the caller's company: page and chunk counts, parser, schema_version, content_hash. This is what a re-extraction is chosen from.

POST /api/v1/knowledge/documents/{knowledge_document_id}/extract

Re-run extraction over a stored artifact. Same job machinery and same polling route, but no parsing — prompt retuning is the common case and must not re-pay the parser for a byte-identical artifact. Takes user_id as a query parameter.

GET /api/v1/knowledge/entries

Candidate entries, filterable by doc_id and kind (glossary · rule · formula · domain), each annotated with the current expert decision if one exists. payload is the full typed entry and is the source of truth; the promoted fields beside it (label, mention_count, diff_status, …) exist so a list can be ordered without opening it.

diff_status (new · duplicate · conflicting) is the cross-document relationship. Entries append and never merge: the same term in two documents keeps two ids, because deciding they are one concept is a judgement about the world rather than a string comparison.

GET /api/v1/knowledge/queue

The review queue — conflicts first, then descending mention count, each row carrying a plain-language review_reason.

Derived at read time, never stored: persisting an ordering creates a second truth that drifts from the entries it came from.

⚠️ Every row carries row_kind and a consumer must switch on it. A key_parameter row has a surface and no term, no definition and no mention count; a term row has all of them. Provenance carries both page (0-based, as the parser reports) and page_no (1-based) — a UI must render page_no.

POST /api/v1/knowledge/entries/{entity_id}/reviews

Record one expert decision: approved · edited · rejected. Append-only — a changed mind is a new row, never an update.

Keyed on entity_id, not the entry's row id: entity_id is deterministic and content-derived, so it survives re-extraction while row ids do not. That is what lets approvals outlive a prompt retune. content_hash is derived server-side from the artifact the entry came from, never taken from the request, so a stale client cannot attribute a ruling to the wrong version. edited requires edited_payload (422 otherwise); an unknown entity_id is 404.

This closes the loop. Approved and edited entries become the baseline the next extraction diffs against, so working the queue improves the following run rather than only annotating this one.

⚠️ Injection path, recorded not fixed. Because an edited_payload reaches a future prompt through that baseline, and this surface is unauthenticated, anyone who can reach it can influence later extractions. This is the strongest argument for the forwarded identity in DEV_PLAN #43, and it should be closed before an expert relies on the queue.