docs/API_CONTRACT_BE_PYTHON.md

Backend Agentic Service API Contract

Last updated: 2026-07-14

This document describes the Python agentic backend used by the frontend for AI chat, help/report tools, charts, and traceability data shown alongside chat answers.

Base path examples use relative URLs. Configure the frontend with the deployed Python service base URL.

Overview

The Python backend owns the generative AI interaction surface:

1. Stream chat answers from the AI agent.
2. Execute tool-style actions for help and report generation.
3. Return report versions, report details, and report-readiness signals.
4. Return traceability for a completed assistant answer.
5. Return chart specifications produced for a completed assistant answer.

The frontend uses this service during the analysis conversation flow:

1. User sends a chat message.
2. Frontend calls POST /api/v2/chat/stream and renders the streamed answer.
3. When the stream emits done, frontend uses the returned message_id as the assistant answer correlation id.
4. Frontend calls GET /api/v1/traceability for planning, tool calls, and source provenance.
5. Frontend calls GET /api/v1/charts with the same message_id and renders any returned charts under the answer.
6. Frontend calls /api/v1/tools/help for guided help and /api/v1/tools/report for report generation (the Generate-Report button; GET …/readiness drives the button state).

Endpoint Summary

Method	Path	Purpose
POST	/api/v2/chat/stream	Stream an AI chat answer for one analysis conversation.
GET	/api/v1/tools/list	List available frontend tools.
POST	/api/v1/tools/help	Stream contextual help for the current analysis conversation.
POST	/api/v1/tools/report	Generate and persist a new report version.
GET	/api/v1/tools/report/{analysis_id}	List report versions for an analysis.
GET	/api/v1/tools/report/{analysis_id}/records	List analysis records for report curation.
GET	/api/v1/tools/report/{analysis_id}/readiness	Report-readiness signal for the Generate-Report button.
GET	/api/v1/tools/report/{analysis_id}/{version}	Retrieve one report version.
GET	/api/v1/traceability	Retrieve provenance for one assistant answer.
GET	/api/v1/charts	Retrieve chart(s) produced for one assistant answer.

Common Concepts

Identifiers

• user_id: user identifier passed by the frontend.
• analysis_id: analysis conversation identifier.
• message_id: assistant answer identifier generated by Python and returned in the stream done event; used to correlate chat streaming, Golang message persistence, traceability, and charts. It is a UUID string (e.g. 77f06761-0fdf-4cc5-84f8-5f81bcbb6f84); the msg_… values in the examples below are illustrative placeholders only. Never generate or send it from the frontend.

Server-Sent Events

Chat and help endpoints return text/event-stream.

Frontend should parse events by event name and data payload. Blank lines separate SSE events.

Common event types:

Event	Data	Meaning
sources	JSON array	Always [] — real sources moved to GET /api/v1/traceability. Event kept for backward compatibility.
status	text	Optional progress update for slower paths.
chunk	text	Answer text fragment. Concatenate chunks in order.
done	JSON object	Terminal success event. Includes message_id.
error	text	Terminal error event. Stream stops after this.

The stream carries answer text only. Planning, tool call details, and full provenance are fetched from GET /api/v1/traceability after the stream is done; charts are fetched from GET /api/v1/charts. The done event carries no chart hint — fetch GET /api/v1/charts unconditionally on every done (the response tells you if there is nothing to render).

Chat

POST /api/v2/chat/stream

Streams an AI answer for one user message in an analysis conversation. The frontend must not generate or send message_id; Python generates it and returns it in the terminal done event.

Request body:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42",
  "message": "What were total sales by region last quarter?"
}

Fields:

Field	Required	Description
user_id	Yes	User identifier.
analysis_id	Yes	Analysis conversation identifier.
message	Yes	User message text.

Response: text/event-stream.

Example structured answer:

event: sources
data: []

event: status
data: Planning analysis...

event: status
data: Running 3 steps...

event: chunk
data: Total sales by region last quarter:

event: chunk
data: Central led at $1.21M (38%), East $0.74M, West $0.55M (down 12% QoQ).

event: done
data: {"message_id":"msg_88f1"}

Example simple chat answer:

event: sources
data: []

event: chunk
data: I'm your AI data analyst. Connect a source or ask a question to get started.

event: done
data: {"message_id":"msg_12"}

Behavior notes:

• Greeting and farewell messages may use a fast canned path.
• Stateless chat intent may use a 1-hour Redis response cache.
• The router may classify messages into intents such as chat, help, check, unstructured_flow, structured_flow, or out_of_scope.
• sources in the stream is always [] — read the real sources[] from GET /api/v1/traceability after done.
• status events are optional and should be safe for the frontend to ignore.
• When the user explicitly asks to plot/visualize ("show me a bar chart of…", "buatkan grafik…"), the answer text describes the result and the chart itself is delivered via GET /api/v1/charts — it is never embedded in chunk text.

Tools

GET /api/v1/tools/list

Returns the deterministic list of tools available to the frontend.

Request: none.

Response 200:

{
  "count": 1,
  "tools": [
    {
      "command": "/help",
      "name": "help",
      "type": "skill",
      "description": "Show what the assistant can do and guide your next step."
    }
  ]
}

The catalog is /help only. /report is not a slash command — report generation is a right-side Generate button; the button calls POST /api/v1/tools/report.

Tool item shape:

{
  "command": "/help",
  "name": "help",
  "type": "skill",
  "description": "Show what the assistant can do and guide your next step."
}

Frontend behavior:

• Surface /help in the slash menu.
• Surface report generation as a button or explicit UI action, driven by GET /tools/report/{analysis_id}/readiness.

POST /api/v1/tools/help

Streams contextual guidance for the current analysis conversation.

Request body:

{
  "user_id": "u_1a2b3c",
  "analysis_id": "an_42"
}

Response: text/event-stream using the same event shape as chat.

Help responses usually emit sources: [] and no status pings.

Example:

event: sources
data: []

event: chunk
data: Your goal is set. You can start exploring now. Try a question like "average order value by month", then I can generate a report.

event: done
data: {"message_id":"msg_h7"}

Reports

POST /api/v1/tools/report

Generates, persists, and returns a new report version for an analysis.

Query params:

Query	Required	Description
analysis_id	Yes	Analysis identifier.
user_id	Yes	User identifier.
exclude_record_ids	No	Record ids to leave out of this version (repeat the param per id). Get ids from GET /tools/report/{analysis_id}/records. Excluded runs are listed in the report's "Excluded Analyses" section. Excluding every substantive record returns 409.

Example:

POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c
POST /api/v1/tools/report?analysis_id=an_42&user_id=u_1a2b3c&exclude_record_ids=rec_a1&exclude_record_ids=rec_c3

Status codes:

Status	Meaning
201	New report version generated.
409	Report floor/precondition not met.
500	Generation or persistence failed.

Response 201:

{
  "report_id": "8f3a2b1c9d4e4f6a8b0c1d2e3f4a5b6c",
  "analysis_id": "an_42",
  "user_id": "u_1a2b3c",
  "version": 2,
  "generated_at": "2026-06-30T09:14:33.512Z",
  "problem_statement": {
    "objective": "Understand which regions drive revenue and why Q1 dipped.",
    "business_questions": [
      "Which regions contribute most to total revenue?",
      "Did any region decline quarter-over-quarter?"
    ]
  },
  "record_ids": ["rec_a1", "rec_b2"],
  "executive_summary": "Revenue is concentrated in the Central region (38% of total). The West was the only region to contract, down 12% QoQ, the main driver of the Q1 dip.",
  "bq_answers": [
    {
      "question": "Which regions contribute most to total revenue?",
      "answer": "The Central region leads with 38% of total revenue.",
      "status": "answered",
      "record_ids": ["rec_a1"]
    },
    {
      "question": "Did any region decline quarter-over-quarter?",
      "answer": "Yes — the West region fell 12% QoQ.",
      "status": "answered",
      "record_ids": ["rec_b2"]
    }
  ],
  "findings": [
    {
      "text": "Central region contributed 38% of total revenue, the largest share.",
      "record_ids": ["rec_a1"],
      "supporting_data": null
    },
    {
      "text": "West region revenue fell 12% quarter-over-quarter.",
      "record_ids": ["rec_b2"],
      "supporting_data": null
    }
  ],
  "caveats": [
    {
      "text": "March data for the East region was partially missing, around 6% of rows.",
      "record_ids": ["rec_b2"]
    }
  ],
  "open_questions": [
    {
      "text": "What drove the West region's QoQ decline?",
      "record_ids": ["rec_b2"]
    }
  ],
  "unresolved": [
    {
      "text": "Correlate churn with tenure — churn column not found in the source.",
      "record_ids": ["rec_d4"]
    }
  ],
  "excluded": [],
  "evidence_tables": {
    "rec_a1": [
      {
        "title": "Aggregate revenue by region",
        "columns": ["region", "total_revenue"],
        "rows": [["Central", "18321"], ["West", "9954"]],
        "truncated": false
      }
    ]
  },
  "charts": {
    "rec_a1": [
      {
        "schema": "dataeyond.chart.v1",
        "chart_type": "bar",
        "title": "Revenue by region",
        "plotly": {
          "data": [{ "type": "bar", "x": ["Central", "East", "West"], "y": [1210000, 740000, 550000], "name": "revenue" }],
          "layout": { "title": { "text": "Revenue by region" }, "xaxis": { "title": { "text": "region" } }, "yaxis": { "title": { "text": "revenue" } } }
        }
      }
    ]
  },
  "data_sources": [
    {
      "source_id": "src_sales_db",
      "name": "orders",
      "source_type": "postgres",
      "detail": {
        "tables": ["orders"],
        "row_count": 48213,
        "columns": ["region", "amount", "ordered_at"]
      }
    }
  ],
  "method_steps": [
    {
      "task_id": "t1",
      "stage": "data_understanding",
      "objective": "Inventory the sales source",
      "status": "success",
      "tools_used": ["check_data"]
    },
    {
      "task_id": "t2",
      "stage": "modeling",
      "objective": "Aggregate revenue by region",
      "status": "success",
      "tools_used": ["analyze_aggregate"]
    }
  ],
  "rendered_markdown": "# Analysis Report\n\n*Generated 2026-06-30 by u_1a2b3c*\n\n## Objective\nUnderstand which regions drive revenue..."
}

Field notes:

• bq_answers — one entry per business question. status is answered | partial | unanswered; record_ids cite the backing analyses. Written in the analysis's language (Indonesian objective → Indonesian answers).

• unresolved — runs that were attempted but produced no usable evidence. Not part of the findings body.

• excluded — runs the caller excluded via exclude_record_ids.

• evidence_tables — record_id → small result tables copied from the run's stored outputs (max 3 tables per record, max 10 rows each; truncated: true when rows were capped). Rendered as markdown tables under the matching Key Findings group in rendered_markdown.

• charts — record_id → chart envelopes (same dataeyond.chart.v1 shape as GET /api/v1/charts returns; max 3 per record) copied from the run's stored outputs. rendered_markdown contains an ## EDA section where each chart appears as a fenced block:

  ```plotly
  {
    "schema": "dataeyond.chart.v1",
    "chart_type": "bar",
    "title": "…",
    "plotly": { "data": [ … ], "layout": { … } }
  }
  ```
  

  The fence content is the full envelope — parse it and render Plotly.newPlot(el, parsed.plotly.data, parsed.plotly.layout). A bold caption line (the chart title) precedes each fence.

Precondition:

• Reports require at least one completed analysis record for the session. A run counts when an analysis step succeeded or a chart was produced — a chart-only session can generate a report.
• If slow-path analysis recording is disabled, report generation can return 409 by design.

Response 409:

{
  "detail": "Not ready to generate a report - still needs at least one completed analysis."
}

GET /api/v1/tools/report/{analysis_id}

Lists report versions for one analysis, oldest first.

Response 200:

[
  {
    "report_id": "1b2c3d4e",
    "version": 1,
    "generated_at": "2026-06-24T15:02:11Z",
    "record_count": 1
  },
  {
    "report_id": "8f3a2b1c",
    "version": 2,
    "generated_at": "2026-06-25T09:14:33Z",
    "record_count": 2
  }
]

If no reports exist, returns [].

GET /api/v1/tools/report/{analysis_id}/records

Lists the persisted analysis runs a report would be built from, oldest first. The frontend shows this before generating so the user can deselect runs; the chosen ids go to POST /tools/report as exclude_record_ids.

Response 200:

[
  {
    "record_id": "rec_a1",
    "goal_restated": "Rank regions by total revenue",
    "created_at": "2026-06-30T08:55:02Z",
    "substantive": true,
    "findings_count": 2
  },
  {
    "record_id": "rec_d4",
    "goal_restated": "Correlate churn with tenure",
    "created_at": "2026-06-30T09:01:47Z",
    "substantive": false,
    "findings_count": 1
  }
]

substantive: false means the run produced no usable result (no analysis step or chart succeeded) — that run is listed in the report's unresolved JSON field rather than the findings body. If no runs exist, returns [].

GET /api/v1/tools/report/{analysis_id}/readiness

Deterministic report-readiness signal for the Generate-Report button — the same producer as Help's readiness signal, so the button and Help never disagree.

Response 200:

{
  "ready": false,
  "missing": ["a new analysis since the last report"]
}

Note: POST /tools/report itself only enforces the floor (at least one completed analysis). The delta gap in missing is a soft warning the frontend can surface ("nothing new since the last report") without blocking the button.

GET /api/v1/tools/report/{analysis_id}/{version}

Returns one report version. Shape is the same as the 201 response from POST /api/v1/tools/report.

Response 404:

{
  "detail": "No report v3 for analysis 'an_42'."
}

Charts

A chart is produced when the user explicitly asks to plot/visualize something in chat. The chart is never embedded in the streamed text — it is fetched separately after the stream completes, then rendered with plotly.js under the assistant message.

GET /api/v1/charts

Returns every chart produced during one assistant answer.

Call this after the chat stream emits done, using the message_id from the done event (same fetch-on-done pattern as traceability). Chart rows are written before done, so there is no polling race. Every response is HTTP 200 — branch on the status field, not the HTTP code.

Query params:

Query	Required	Description
message_id	Yes	Assistant answer identifier returned by the stream's done event.

Example:

GET /api/v1/charts?message_id=88f10c3a-6f03-4204-bf98-41ffc20388b2

Response 200 — status: "success" (≥1 chart to render):

{
  "status": "success",
  "message": "1 chart(s) for this message.",
  "count": 1,
  "charts": [
    {
      "chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e",
      "chart_type": "bar",
      "title": "Revenue by region",
      "spec": {
        "schema": "dataeyond.chart.v1",
        "chart_type": "bar",
        "title": "Revenue by region",
        "plotly": {
          "data": [{ "type": "bar", "x": ["Central", "East", "West"], "y": [1210000, 740000, 550000], "name": "revenue" }],
          "layout": { "title": { "text": "Revenue by region" }, "xaxis": { "title": { "text": "region" } }, "yaxis": { "title": { "text": "revenue" } } }
        }
      },
      "created_at": "2026-07-13T03:21:09.114Z"
    }
  ]
}

Response 200 — status: "empty" (the turn completed but produced no charts; the common case, not an error):

{
  "status": "empty",
  "message": "This message completed without producing charts.",
  "count": 0,
  "charts": []
}

Response 200 — status: "not_found" (no completed turn is known for this message_id; usually a stale or mistyped id):

{
  "status": "not_found",
  "message": "No completed turn is known for this message_id.",
  "count": 0,
  "charts": []
}

The dataeyond.chart.v1 envelope (the shape of charts[].spec):

{
  "schema": "dataeyond.chart.v1",
  "chart_type": "bar",
  "title": "Revenue by region",
  "plotly": {
    "data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
    "layout": { "title": { "text": "Revenue by region" } }
  }
}

Field rules:

• status is the outcome marker: success | empty | not_found. Branch on it; do not parse message (human-readable, for logs only).
• spec is the full envelope, unmodified — render straight from it: Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout).
• chart_type / title are copied out of spec for convenience (list rendering without parsing spec); title may be null. Chart types: bar, line, pie, scatter.
• A turn can produce more than one chart; charts is ordered by creation time.

Frontend rendering guidance:

• Fetch unconditionally on every done; status: "empty" means render nothing extra.
• Render each chart under the assistant message it belongs to.
• Treat status: "not_found" as a signal worth logging (stale id or fetch bug) — not as a user-facing error.
• Chart iteration is a follow-up chat turn (e.g. "make it a line chart") — there is no separate edit endpoint.
• The same envelope shape appears inside report markdown as ```plotly fenced blocks (see Reports → charts), so one renderer can serve both surfaces.

Traceability

GET /api/v1/traceability

Returns user-facing provenance for one assistant answer.

The frontend should call this after the chat/help stream emits done, using the message_id from the done event. The row is written before done, so an immediate GET returns 200 (no polling race). A 404 means the id is unknown or the turn errored before completing (error turns never produce a row).

Query params:

Query	Required	Description
analysis_id	Yes	Analysis identifier.
message_id	Yes	Assistant answer identifier returned by the stream.

Example:

GET /api/v1/traceability?analysis_id=an_42&message_id=msg_88f1

intent values the frontend may see: chat · help · check · unstructured_flow · structured_flow · out_of_scope · blocked (blocked = input-guard or content-filter refusal; chat also covers the greeting fast-path and cache replays).

Field rules:

• planning: present only when the planner ran (structured_flow); otherwise null.
• thinking: always null in v1 — the field stays in the payload so it can be populated later without a contract change.
• tool_calls: every invoked tool with summary (plain-English one-liner), input, output, status, task_id (nullable), and error (nullable); empty for chat / help / greeting / refusal paths. input/output are the raw tool I/O (opaque ids) — render them in a collapsible "technical details" section, not the headline; use summary for the headline.
• A render_chart tool call reports a compact chart summary in output (chart_type, title, trace_count, point_count) — the full spec is served by GET /api/v1/charts, not here.
• data_used: one entry per structured data pull, resolved to real names for display (empty when no structured pull ran). Split into columns_read (columns read straight from the user's data, each tagged with its roles) and output_columns (kind: "column" = read from data, kind: "computed" = calculated, carrying a formula and no id). Also carries tables (all touched, including join targets), joins, filters (with a plain-language description), group_by, order_by, limit, rows_returned, and the executed query.
• id fields are machine-only. Every id in data_used (source.id, tables[].id, columns_read[].id) is for linking/audit — the frontend must never render it. Show name (qualified as table.name). A computed output column has no id by design.
• sources: required for retrieval flows; empty for chat / help / refusal paths and for check. Database sources also carry source_name (the DB's real name) and tables (every table touched).
• Summaries and filter descriptions are built from fixed templates, never an LLM — traceability adds no latency and cannot hallucinate.
• The payload also carries an internal user_id (ownership); the frontend may ignore it.
• Truncation: preview ≤ 5 rows; any string inside input/output/preview/snippet ≤ 300 chars (executed query ≤ 2000); rows beyond the preview are dropped (row_count is preserved).

Response 200 for structured_flow:

{
  "analysis_id": "an_42",
  "message_id": "msg_88f1",
  "user_id": "user_7",
  "intent": "structured_flow",
  "generated_at": "2026-07-06T03:21:09.114Z",
  "planning": {
    "goal_restated": "Find which regions drive revenue and why Q1 dipped.",
    "assumptions": [],
    "steps": [
      {
        "step": 1,
        "stage": "data_understanding",
        "objective": "Inventory the sales source",
        "status": "success",
        "tools_used": ["check_data"]
      },
      {
        "step": 2,
        "stage": "modeling",
        "objective": "Aggregate revenue by region",
        "status": "success",
        "tools_used": ["retrieve_data", "analyze_aggregate"]
      }
    ]
  },
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "check_data",
      "summary": "Inspected your data source structure",
      "input": { "source_hint": "structured" },
      "output": {
        "kind": "table",
        "columns": ["source_id", "name", "source_type", "table_count"],
        "row_count": 1,
        "preview": [["src_sales_db", "orders", "schema", 1]]
      },
      "status": "success",
      "error": null
    },
    {
      "order": 2,
      "task_id": null,
      "name": "retrieve_data",
      "summary": "Retrieved 4 rows across 2 columns from orders",
      "input": { "ir": { "source_id": "src_sales_db", "table_id": "orders", "select": ["region", "amount"], "group_by": ["region"] } },
      "output": {
        "kind": "table",
        "columns": ["region", "total"],
        "row_count": 4,
        "preview": [["Central", 1210000], ["East", 740000]]
      },
      "status": "success",
      "error": null
    }
  ],
  "data_used": [
    {
      "source": { "id": "src_sales_db", "name": "sales db", "type": "schema" },
      "tables": [ { "id": "orders", "name": "orders", "role": "base" } ],
      "joins": [],
      "columns_read": [
        { "id": "c_region", "name": "region", "table": "orders", "data_type": "string",  "pii": false, "roles": ["selected", "grouped"] },
        { "id": "c_amount", "name": "amount", "table": "orders", "data_type": "decimal", "pii": false, "roles": ["aggregated"] }
      ],
      "output_columns": [
        { "name": "region", "kind": "column",   "from": "orders.region" },
        { "name": "total",  "kind": "computed", "from": "orders.amount", "formula": "SUM(orders.amount)" }
      ],
      "filters": [],
      "group_by": ["orders.region"],
      "order_by": [],
      "limit": null,
      "rows_returned": 4,
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region"
    }
  ],
  "sources": [
    {
      "type": "database",
      "source_id": "src_sales_db",
      "source_name": "sales db",
      "name": "orders",
      "tables": ["orders"],
      "query": "SELECT region, SUM(amount) AS total FROM orders GROUP BY region",
      "detail": {
        "table": "orders",
        "row_count": 4
      }
    }
  ]
}

Note: retrieve_data's real input is the compiled query IR under an ir key (the planner builds an IR, never raw SQL). The executed SQL/rendered query appears on the corresponding sources[].query.

Response 200 for unstructured_flow:

{
  "analysis_id": "an_42",
  "message_id": "msg_55",
  "user_id": "user_7",
  "intent": "unstructured_flow",
  "generated_at": "2026-07-06T03:40:02.001Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [
    {
      "order": 1,
      "task_id": null,
      "name": "retrieve_knowledge",
      "input": { "query": "technology stack used in this project" },
      "output": { "kind": "documents", "row_count": 4 },
      "status": "success",
      "error": null
    }
  ],
  "sources": [
    {
      "type": "document",
      "document_id": "doc_7",
      "filename": "tech_handbook.pdf",
      "page_label": "12",
      "query": "technology stack used in this project",
      "snippet": "The backend is built on FastAPI with async SQLAlchemy...",
      "score": 0.83
    }
  ]
}

Response 200 for chat / greeting / help / refusals (out_of_scope, blocked):

{
  "analysis_id": "an_42",
  "message_id": "msg_12",
  "user_id": "user_7",
  "intent": "chat",
  "generated_at": "2026-07-06T03:05:00.000Z",
  "planning": null,
  "thinking": null,
  "tool_calls": [],
  "sources": []
}

Response 404:

{
  "detail": "No traceability for message 'msg_88f1' yet."
}

Frontend rendering guidance:

• Render traceability separately from the streamed answer.
• Default state can be collapsed.
• Show planning, tool calls, and sources as separate sections.
• Treat planning: null, tool_calls: [], and sources: [] as valid states.